Commercial AI tool moderately successful at predicting hospitalization-related kidney injury

Hospital-acquired acute kidney injury (HA-AKI) is a common complication in hospitalized patients that can lead to chronic kidney disease and is associated with longer hospital stays, higher health care costs and increased mortality. Given these negative consequences, preventing HA-AKI can improve hospitalized patient outcomes. However, anticipating HA-AKI onset is difficult due to a large number of contributing factors involved.

Researchers from Mass General Brigham Digital tested a commercial machine learning tool, the Epic Risk of HA-AKI predictive model, and found it was moderately successful at predicting risk of HA-AKI in recorded patient data. The study found a lower performance than those recorded by Epic Systems Corporation’s internal validation, highlighting the importance of validating AI models before clinical implementation.

The Epic model works by assessing adult inpatient encounters for the risk of HA-AKI, marked by predefined increases in serum creatinine levels. After training the model using data from MGB hospitals, the researchers tested it on data from nearly 40,000 inpatient hospital stays for a five-month period between August 2022 and January 2023. The dataset was extensive with many points collected on patient encounters, including information such as patient demographics, comorbidities, principal diagnoses, serum creatinine levels and length of hospital stay. Two analyses were completed looking at encounter-level and prediction-level model performance.

The investigators observed that the tool was more reliable when assessing patients with lower risk of HA-AKI. Although the model could confidently identify which low-risk patients would not develop HA-AKI, it struggled to predict when higher-risk patients might develop HA-AKI. Results also varied depending on the stage of HA-AKI being evaluated —predictions were more successful for Stage 1 HA-AKI compared to more severe cases.

The authors concluded overall that implementation may result in high false-positive rates and called for further study of the tool’s clinical impact.

“We found that the Epic predictive model was better at ruling out low-risk patients than identifying high-risk patients,” said lead study author Sayon Dutta, MD, MPH, of Mass General Brigham Digital’s Clinical Informatics team, and an emergency medicine physician at Massachusetts General Hospital. “Identifying HA-AKI risk with predictive models could help support clinical decisions such as by warning providers against ordering nephrotoxic medications, but further study is needed before clinical implementation.”

The study is published in the journal NEJM AI.

More information:
Sayon Dutta et al, External Validation of a Commercial Acute Kidney Injury Predictive Model, NEJM AI (2024). DOI: 10.1056/AIoa2300099