E43: Development, Validation, and Comparison of Machine Learning Models for Predicting Pediatric Surgical Site Infection Using the NSQIP-P Database

0:03You are listening to The Operative Word, a podcast brought to you by the Journal of the American College of Surgeons. I'm Dr Tom Varghese, and throughout the series, Dr Lillian Erdahl and I will speak with recently published authors about the motivation behind their latest research and the clinical implications it has for the practicing surgeon. The opinions expressed in this podcast are those of the participants, and not necessarily that of the American College of Surgeons.

0:33Hello, loyal listeners, welcome to another episode of The Operative Word, the official podcast of the Journal of the American College of Surgeons. My name is Tom Varghese. I'm the host today. I'm also the Editor-in-Chief for JACS. Today, we're discussing a new study that addresses an important challenge in pediatric surgery: how to better predict pediatric surgical site infections using a large-scale clinical data and modern machine learning approaches.

1:02The article published in JACS, which is available right now, is titled, “Development, Validation, and Comparison of Machine Learning Models for Predicting Pediatric Surgical Site Infection Using the NSQIP-P Database.” Joining us today are the lead authors of the study. Carrie T Chan is a researcher in epidemiology and biostatistics at the University of California, San Francisco, as well as an advanced practice provider at Stanford Medicine Children's Health, whose work focuses on surgical outcomes and predictive modeling.

1:39Balakrishnan, a professor of otolaryngology head and neck surgery at Stanford University and a pediatric surgeon-scientist with expertise in surgical quality and outcomes research. Carrie and Karthik, welcome to the podcast. We'll start first to see if you have any disclosures. Carrie, any disclosures relevant for today's session? No disclosures other than the fact that the work was funded by the NICHD through an F31. Perfect. And then, Karthik, any disclosures for today's episode?

2:11No disclosures. Tom. Perfect. Well, let's start with the actual clinical problem. Surgical site infections remain a significant source of morbidity in children undergoing surgery. What actually motivated your team to take on this question of predicting SSI risk in pediatric patients? Carrie, let's start with you. So, really, this all started a few years ago. I was actually part of a lot of conversations at the time around how we could improve our surgical prevention practices at Stanford Medicine, Children's Health and actually some very focused conversations in my home department of orthopaedic surgery, because we had just seen a couple postoperative spine infections come up.

2:54And we all know that infections aren't super common in our day to day. But when they do happen, you know, we're talking about taking them back to surgery, prolonged antibiotic treatment, rehospitalization and just a bumpier road to recovery. So while we were looking at how we can make improvements for like let's look at the evidence, what does the evidence say? And we found that a large majority of it was based off of adult literature. And I think those of us who work in pediatrics always try to hammer home, like kids are not just little adults.

3:22And we should make sure that what we're doing is specific to what they need. And so that really drove us to think about, like, with all this amazing real-world data out there through registries like NSQIP, electronic health records, now we should be able to do better and be able to predict an individualized risk a little bit better. That makes amazing sense. Carrie. Karthik from your perspective, I mean, all of us surgeons have heard about prediction tools forever.

3:55I mean, it's it that's that itself is not new, but do you think there's anything else like politically or policy wise that why specifically in the pediatric surgical population, the predictive tools are lagging behind this compared to the adults? That's a really good question, Tom. I think there's a few reasons. One is just data collection for pediatric purposes has, pediatric populations has not been going on as long. You know, the pediatric NSQIP is a newer institution, for instance, than the larger adult NSQIP, for example.

4:26Part of it, too, is funding for pediatric research is often generally less available or less robust. And then I think another reason is just if we look at the, the business case in a way for developing some of these models, you know, adult populations of patients are generally larger and more easily available. The data sources are larger and more easily available, and it's easier to imagine the populations that they're applicable to. And so I think all of those things together have combined to put the pediatric aspect perhaps a little bit behind the adult aspect, behind the adult, that makes sense.

5:04Well, thanks for that. So let's, deep dive into the article. So as Carrie alluded to, the study, aim to develop, validate and compare predictive models that estimate patient-specific, surgical soft tissue infection risk in children using preoperative data. Carrie, the selection of the database. Obviously you chose the NSQIP-P database, which has more than 1.1 million cases. It's just it's the largest database out there for this particular population.

5:40Yeah. That is actually really why we chose that database. It's because it's the largest available, you know, nationally representative database and validated for surgical outcomes in pediatrics. I think when we're looking at developing prediction models, you just need a lot of data. And if you try to use institution-specific data, a lot of times you don't have the volume that you need. And so, kind of step 1 was seeing if we could build a prediction model given the amount of data that is available to, you know, NSQIP.

6:07And you know, we're again reading directly from the, your paper. multi-institutional registry, collects standardized surgical outcomes from more than 150 hospitals across the United States. For this particular study, it was a 10-year, data point, that is from 2012 to 2022. And the focus was on 30-day postoperative SSIs. Now, interestingly, you used a composite metric that is all the three categories, right? And that again, because the goal was really to do a predictive modeling, for this, I guess the the next question for the methodology is you you evaluated five different modeling approaches that from the traditional kind of logistic regression to neural networks.

6:59Was that, like evidence based in terms of like this is the standard way to do this? These are the five different models or what was the rationale for those, the choice of those five different models? Largely, I think different approaches to modeling have strengths and then weaknesses. Depending on the type of data, what you're trying to predict. So really, this was almost a way for us to say, like, we had this amazing data and we have all these different approaches we can explore.

7:27Let's just kind of throw the book at it and see what how they compare. And like you mentioned, all the way from kind of the most approachable regularized logistic regression approach, is really great when it comes to structured data interpretability, being able to perform feature selection and computational efficiency. So in one way, if we're looking at a model that we can actually deploy real-time and has a lot of strengths and not studies vs all the way to the more complex neural networks, kind of these black box methods.

8:01And if so, better is better. But knowing that it had some downsides when it came to real-time use, in clinical care, I'll ask this question to Karthik. Is this, for surgeons, I mean, I know that, you know, you know, you see, AI is this massive word and we're trying to figure all these things out and everything. as they're dealing with this, like, hey, when you're trying to test this, you've got a, like, a learning aspect and then a validation. Like what?

8:33What's the easy way to, for surgeons to get and get their minds wrapped around this? Yeah. You know, that's a really good question, Tom. Because Carrie is brilliant at this. I mean, Carrie's a genius. I mean, there's a question about that. I mean, as the sort of the clinical context person. But I learned, you know, along the way a little bit. And the way I came to think about it is that you can develop a model using a set of data, and regardless of the type of model you develop, and the methods used to get there, it is cheating to test the model on the same set of data, correct?

9:08So you have to have another set of similar data where you can test the model. And if it looks good there, then what you've proven is that it can work under sort of ideal circumstances. Then you have to take it and test it on real-world data, and then you have to see does it work on real-world data from different sources are different institutions. that's how I kind of came to think of it. And largely that's because that's the brilliant way that Carrie designed the work.

9:35No, and I and I think that, what we're hoping, you know, obviously selfishly for the Journal, we're, we're hoping is as we're seeing more and more, you know, unbelievably high-quality manuscripts in this space, you know, just getting surgeons and our readership used to that. Oh, I get it. There's a gazillion data points. We have to create the variables to kind of the learning. Now we have to test it for validity. And then now the external validity is tested on a different data set as well.

10:05No, I think that that makes the sense, all in the world. And so that's where again, reading from the manuscript, it says the full data split data set was then split into training and test sets to facilitate model training and internal validation. Before that, as well, I guess, this may be a question for, for Carrie. I think that, interestingly, of those five models, I personally was surprised that the regularized logistic regression model ultimately performed just as well as some of the really more complex approaches.

10:41Were you surprised by that finding or, I mean, or is just my naivete in saying that, oh, the more complex it is, maybe it should perform better. and I don't know why that was kind of I just thought with the type of data we had, it was a nice balance between, you know, for me, it looked like oh, a good old logistic regression model. You know, we'll throw that in there. And for the listeners, just so explain, clarify a little bit about Random Forest or some of these other neural networks, a couple of those terms.

11:11If you could just take pause for a moment and explain that. Oh yeah. Sure. So the the random forest is essentially like, a beefed up decision tree where you can think, they look at all the different predictors that we have, and then it determines based on each node, like what predictor is potentially most important or is helpful in predicting this infection risk. And then a random forest is basically a bunch of different decision trees put together.

11:35So it's smarter than just a single decision tree. And less likely to overfit to just the one try that the decision tree would do. So we call it an ensemble method. And so it just adds a little bit more complexity. It can it can read a little bit more into like interactions between the different variables that a logistic regression might not be able to capture. So I like I said, my money like you was just kind of your money was on that. Yeah. Okay.

12:01My money was up on one of the slightly more complex methods. But really, when you look at it, the logistic regression performed pretty much just as well, and like I mentioned previously, it has the wonderful qualities where a regularized logistic regression model can do things like feature selection, which is where it can pick out what is actually an important predictor and what is not. And so that makes the model, I think, more approachable. A logistic regression model is more transparent.

12:29And so it has all these wonderful pros with it that I think when we looked at the type of data that was NSQIP, which is largely structured data from the EHR like predictors that we all know, like, you know, ICD-10 codes, CPT codes, and then certain comorbidities that it just did really well with that type of structured data. That that's amazing. and obviously I encourage our listeners to, to read this paper. It may seem very complex. It is. There's some computations and the analysis is very complex, but I think that if you think about the framework of the rationale why they did these researchers did this study and like how it applies to clinical practice.

13:18So the logistic, the regularized logistic regression model was the one that favored the most favorable balance, that is, of predictive performance. You know, the computational, you know, efficiency, interpretability and practicality for potential implementation. And really, that was the goal or trying to figure out the model that's going to be for real-world implementation. And that, as you said, Carrie, that unlike more complex machine learning models, the beauty of the regularized logistic regression is its direct, interpretable coefficients that map to known, clinical factors.

13:56I thought it was fascinating, though, in the discussion section. And this is probably a question for Karthik, in the sense that you took a moment to compare your model to the, traditional American College of Surgeons NSQIP-Pediatric surgical risk calculator. Can you just take a moment and just kind of tell the audience, like, why you went about that approach? Like, there is a risk calculator out there, why you took that moment to kind of compare the two, two models?

14:23Yeah. You know, we thought that would be a useful step for a few reasons. One is it's kind of I guess, the best available thing out there right now. It uses the same data. So then it makes things a little more comparable. But the other is really pragmatically, the ACS has developed this beautiful NSQIP risk calculator, but not many people use it because it's kind of it's in its own little space. Not everyone is familiar with NSQIP data. It's not built into the EHR or easily accessible.

14:53It takes extra steps. And so, one of the questions we had, which kind of is, is a further step from the work in this particular manuscript, is if this performs as well or better than the existing risk calculator, is it also something that we can embed in the EHR and build into the normal clinical workflow, so that surgeons don't have to take that extra step? They can, just as they work up their patient in their normal way, have the risk calculator built in and, account for that in their clinical decisions.

15:27And so that was the rationale. That was that was amazing. So, well, let's talk about next steps. You know, you've you've tested the model, you know, you've got the internal validation. Is it fair to say the next step is now external validation, like using this particular model and real-world examples. And potentially you're going to be looking for a lot of different types of real-world examples. Is that correct, Carrie, in terms of framing, like what the next steps are going to be?

15:57I think that's the perfect framing. I think as we've all seen, even the greatest predictive models in the world, in and of itself, don't really change care. Right? It needs to actually work in the population that we want to use them in, and then be presented and integrated in a way that providers and care teams will actually use it and find it useful. So with that context in mind. Exactly right. I think external validation is truly an essential step to any predictive model.

16:25And the clinical translation, because we see that almost across the board, predictive models do degrade outside of their derivation settings. When you try to move them to a setting where they weren't derived. So we need to make sure that this model actually works and is transportable to our patient population. So we're actually performing external validation now first retrospectively using like real-world, messy EHR data, which is not as beautiful as NSQIP data to see how it works with that kind of messiness.

16:55And then what also what to pursue prospective validation of this model as well. Now, the potential limitation, though is going to be those really small samples, rare disease categories, correct? I mean, I mean, I think Karthik alluded to this has always been kind of the struggle in the pediatric surgical world, right? Like neonates with a congenital abnormality that only affects like a couple hundred people, it's really going to be hard to come up with a predictive model.

17:24Is that correct? Karthik? From that perspective? Yeah, I think that's true, Tom. And part of it is the vulnerability of just children tend to have rare diseases in some cases, but the other vulnerability is the structure of NSQIP itself. Not every surgical procedure is included. Not every specialty or surgical subgroup is equally represented, and that introduces some vulnerability or bias as well. Yeah. It's amazing. It's also amazing that I just realized, like how much time has already flown by like trying to wind things down.

17:54But let me ask you a couple questions about the future. Carrie, I love the way that you framed this in the discussion. So the salvific, you know, I'll read directly from the manuscript, so it says, integration of this model into EHR systems as part of a clinical decision tool could help the clinicians distinguish higher-risk patients from the broader low-risk population before surgery, informing decisions such as tailoring preoperative optimization plans.

18:24Obviously, music to my ears. Adjusting surgical timing, modifying prophylactic antibiotic approaches, targeting postoperative surveillance or engaging support services. I mean, that's a you're talking about all the holy grails that surgery and surgeon scientists have been chasing after for the last two decades. Are you anticipating with the use of these AI tools that now a lot of these efforts and successes are in sight? Carrie, is that what you're thinking?

18:55I wish I had a crystal ball to be able to answer that question. I do think that is the direction that we're moving in, and that the future of predictive analytics is this push towards more individualized, data-informed decision-making across the spectrum. Like that study focused on prediction of SSI, on complication prediction. can inform more than just predicting complications. Like I mentioned in the manuscript, it can be about resource planning even from an operation side like anticipating bed needs ICU needs, and resources from that standpoint.

19:30So I do think that that is in the future. And I, I feel like we can't have any discussion about predictive analytics without also bringing in this idea of where generative AI can then also be pulled in, where it's a complementary layer of predictive models. where predictive models can identify risks or identify needs. But then generative AI can help summarize the relevant clinical context and package it in a way that is approachable and usable in the clinical workflow.

20:01So I think those two big buckets are definitely rapidly developing and will be part of that. big buckets that you just outlined, the human is still in the loop. You're not saying 100% humans are being replaced. It's like hopefully trying to get teams and decision-making and clinical care much more efficient and standardized and better for all. Correct. That is 100% correct. And I think as we're framing the future of this work that we're actively working on, it is actually the human is at the center of kind of all of the design, design and the decisions that we're making.

20:38That's amazing. Karthik putting you on the spot. Surgeons in general are very cynical group. I mean, in terms of, it's like, you know, it's fascinating when you just look at the history of surgery. Right? It's like we have cutting edge techniques, but then there's this old guard that saying, wait a minute, you have to prove it over and over again. groups are going to be approaching these integrations of these new AI tools? And it just seems like every other day there's a there's a new cure coming out.

21:11I mean, what are your thoughts about that? Oh my gosh. That is that could be a whole podcast by itself. But it's, That we need to balance the introduction of these tools into clinical practice with understanding their drawbacks and their benefits. And I think that's how we're really going to get surgeons to understand, right, as this, you know, if we, let's say, build something like this into an EHR and, you know, you're doing your workup, you see a patient and you're putting in your pre-op orders in your EHR, and a little box pops up and says, this patient is at elevated SSI risk.

21:54your experience with those populations of patients and weigh it and contextualize it using your own experience and knowledge. But in order to do that usefully, you have to understand what the model is doing, how it was built, what data it's using. There has to be real transparency there. and APPs that that we should really advocate for is that transparency, not just to have these models handed to us or shoved down our throats, but to help build them and also to help spread understanding for each specific model of all of these issues.

22:31Perfect. Well, before we close, I want to offer each of you, some final words. I mean, I, you know, I'm biased. I love this work. I mean, I think it's exactly the type of work that's needed. It's based on real-world evidence and, you know, the prediction and, you know, like, patients versus like and getting teams involved with this, what keeps you optimistic, about the near future. Carrie we’ll start with you. I'm very optimistic that this can help improve the care that we deliver, but I do want to, I do want to hammer home that this work is really just step 1 in a much larger effort, that building the models matter.

23:15But the real test is whether we can validate it in a real world setting and translate it to tools that are actually useful at the point of care. So what keeps me optimistic is actually I've been working with surgeons, surgical care team members. I'm like, is this actually useful? Would you actually want to use this? And the answer resoundingly has been yes. So how can we get it to you guys? So I think there's a lot of excitement around this, which keeps me optimistic about that.

23:41We can move this work forward. Perfect. Karthik, the same question to you. What what keeps you optimistic about the path forward? Yeah. You know, I'm optimistic about surgeons and surgical APPs. And our colleagues are actively involved in this work. So we have a voice in shaping it, how it's built, how it's used. And we also are developing the skill sets and knowledge to be cautious and raise concerns. And I think that that's that onus is on us to do that and we should take ownership of it.

24:12And I'm optimistic that we are. Perfect. You know, Carrie and Karthik, you know, thank you for joining us today and sharing insights from this incredibly important work. You know, your study really, you know, highlights how large surgical data sets and predictive modeling can help us move closer to more individualized risk assessment and infection prevention strategies in pediatric surgery. And for our listeners, again, the reminder the full article is available in the Journal of the American College of Surgeons right now.

24:43I encourage everybody to go and read this and, want to thank everybody for this opportunity to, to host this incredible episode for the JACS Operative Word. Carrie and Karthik, thank you for all that you do and looking forward to your future work. Thank you so much. Thank you Tom, thank you for having us. Thank you for listening to the journal of the American College of Surgeons Operative Word Podcast. If you enjoyed today's episode, spread the word on social media by using the hashtag JACS Operative Word.

25:20Subscribe to The Operative Word wherever podcasts are available, or listen on the American College of Surgeons website at facs.org/podcast.