Assessment of Machine Learning–Based Medical Directives to Expedite Care in Pediatric Emergency Medicine

Key Points

Question

Can machine learning–based medical directives (MLMDs) be used to autonomously order testing at triage for common pediatric presentations in the emergency department?

Findings

This decision analytical model analyzing 77 219 presentations of children to an emergency department noted that the best-performing MLMD models obtained high area-under-receiver-operator curve and positive predictive values across 6 pediatric emergency department use cases. The implementation of MLMD using these thresholds may help streamline care for 22.3% of all patient visits.

Meaning

The findings of this study suggest MLMDs can autonomously order diagnostic testing for pediatric patients at triage with high positive predictive values and minimal overtesting; model explainability can be provided to clinicians and patients regarding why a test is ordered, allowing for transparency and trust to be built with artificial intelligence systems.

Abstract

Importance

Increased wait times and long lengths of stay in emergency departments (EDs) are associated with poor patient outcomes. Systems to improve ED efficiency would be useful. Specifically, minimizing the time to diagnosis by developing novel workflows that expedite test ordering can help accelerate clinical decision-making.

Objective

To explore the use of machine learning–based medical directives (MLMDs) to automate diagnostic testing at triage for patients with common pediatric ED diagnoses.

Design, Setting, and Participants

Machine learning models trained on retrospective electronic health record data were evaluated in a decision analytical model study conducted at the ED of the Hospital for Sick Children Toronto, Canada. Data were collected on all patients aged 0 to 18 years presenting to the ED from July 1, 2018, to June 30, 2019 (77 219 total patient visits).

Exposure

Machine learning models were trained to predict the need for urinary dipstick testing, electrocardiogram, abdominal ultrasonography, testicular ultrasonography, bilirubin level testing, and forearm radiographs.

Main Outcomes and Measures

Models were evaluated using area under the receiver operator curve, true-positive rate, false-positive rate, and positive predictive values. Model decision thresholds were determined to limit the total number of false-positive results and achieve high positive predictive values. The time difference between patient triage completion and test ordering was assessed for each use of MLMD. Error rates were analyzed to assess model bias. In addition, model explainability was determined using Shapley Additive Explanations values.

Results

There was a total of 42 238 boys (54.7%) included in model development; mean (SD) age of the children was 5.4 (4.8) years. Models obtained high area under the receiver operator curve (0.89-0.99) and positive predictive values (0.77-0.94) across each of the use cases. The proposed implementation of MLMDs would streamline care for 22.3% of all patient visits and make test results available earlier by 165 minutes (weighted mean) per affected patient. Model explainability for each MLMD demonstrated clinically relevant features having the most influence on model predictions. Models also performed with minimal to no sex bias.

Conclusions and Relevance

The findings of this study suggest the potential for clinical automation using MLMDs. When integrated into clinical workflows, MLMDs may have the potential to autonomously order common ED tests early in a patient’s visit with explainability provided to patients and clinicians.

Abstract

This decision analytical modeling study assesses the use of machine learning–based medical directives to automate diagnostic testing at triage for patients in a pediatric emergency department.