Widely used AI tool for early detection of sepsis could allay doctors’ suspicions

Proprietary artificial intelligence software designed to be an early warning system for sepsis cannot differentiate between high- and low-risk patients before receiving treatments, according to a new study from the University of Michigan.

The article is published in the journal NEJM AI.

The tool, named Epic Sepsis Model, is part of Epic’s electronic medical record software, which serves 54% of patients in the United States and 2.5% of patients internationally, according to a statement from the CEO of the company reported by the Wisconsin State Journal. It automatically generates sepsis risk estimates in hospital patient records every 20 minutes, which clinicians hope will allow them to detect when a patient might contract sepsis before things get worse.

“Sepsis has all these vague symptoms, so when a patient presents with an infection, it can be very difficult to know who can be sent home with antibiotics and who may need to stay in the intensive care unit “We’re still missing a lot of patients with sepsis,” said Tom Valley, associate professor of pulmonary and critical care medicine, critical care clinician and co-author of the study.

Sepsis is responsible for one-third of all hospital deaths in the United States, and early treatment is key to patient survival. The hope is that AI predictions could help achieve this, but at present they do not appear to benefit more from patient data than clinicians.

“We suspect that some of the health data the Epic Sepsis model relies on encodes, perhaps unintentionally, clinicians’ suspicions that the patient has sepsis,” said Jenna Wiens, associate professor of computer science and science. engineering and corresponding author of the study.

Patients will not receive blood cultures or antibiotic treatments until they begin to have symptoms of sepsis, for example. While such data could help an AI very accurately identify sepsis risks, it could also enter medical records too late to help clinicians move forward with treatments.

This lag in timing between when information becomes available to AI and when it is most relevant to clinicians was evident in the researchers’ assessment of the Epic Sepsis model’s performance for 77,000 adults hospitalized at University of Michigan Health, the clinical arm of Michigan Medicine. .

The AI ​​had already estimated the sepsis risk for each patient as part of the medical center’s standard operations, so the researchers only needed to extract the data and perform their analysis. Nearly 5% of patients suffered from sepsis.

To measure the AI’s performance, the team calculated the probability that the AI ​​would assign higher risk scores to patients diagnosed with sepsis, compared to patients never diagnosed with sepsis.

By including predictions made by the AI ​​at all stages of the patient’s hospital stay, the AI ​​was able to correctly identify a high-risk patient 87% of the time. However, the AI ​​was only correct 62% of the time when using patient data recorded before the patient met criteria for sepsis. Perhaps most telling is that the model only assigned higher risk scores to 53% of sepsis patients when predictions were limited to before a blood culture was ordered.

The results suggest that the model indicated whether patients received diagnostic tests or treatments when making its predictions. At this point, clinicians already suspect their patients have sepsis, so AI predictions are unlikely to make a difference.

“We need to consider when the model is evaluated in the clinical workflow to decide whether it is useful to clinicians,” said Donna Tjandra, a computer science and engineering doctoral student and co-author of the study. “Evaluating the model with data collected after the clinician has already suspected the onset of sepsis may give the impression that the model’s performance is robust, but this is not what would help clinicians in the practical.”