A model utilizing natural language processing and machine learning can accurately detect radiology reports that demand follow-up imaging, reported researchers of a new study published in the Journal of Digital Imaging.
“While radiologists regularly issue follow-up recommendations, our preliminary research has shown that anywhere from 35 to 50% of patients who receive follow-up recommendations for findings of possible cancer on abdominopelvic imaging do not return for follow-up,” wrote lead author Robert Lou and colleagues. “As such, they remain at risk for adverse outcomes related to missed or delayed cancer diagnosis.”
For their study, Lou, with the Perelman School of Medicine at the University of Pennsylvania and colleagues developed an algorithm that automatically detects free-text radiology reports containing follow-up guidance using both NLP and ML techniques.
The team trained three machine learning algorithms on 6,000 free text abdominal MRI, CT and ultrasound reports from one of three hospitals within the researcher’s health system. Lou et al. used an 80:20 training test ratio, dividing 1,200 radiology reports for the test set and 4,800 for training. The NLP techniques were engineered to recognize 1,500 features.
Of the three models—naïve Bayes, decision tree and maximum entropy model—the decision tree outperformed both others. It registered an F1 score of 0.458 and an accuracy of 0.862, compared to F1 scores of 0.381 and 0.387 for naïve Bayes and maximum entropy, respectively.
“While perhaps not robust enough yet for clinical usage, this study demonstrates proof of concept and underlines the strength of the ML decision tree algorithm,” noted Lou and colleagues.
The results also underscore that the accuracy of follow-up recommendation detection relies on the choice of NLP processing techniques, feature selection and ML algorithm, among other variables, the authors wrote.
“While it is a complex task that is dependent on factors such as the institution, imaging modality, and individual radiologist’s language, careful feature engineering and appropriate selection of machine learning algorithms are important strategies to improve the performance of follow-up detection classifiers,” the researchers concluded.
Matt joined Chicago’s TriMed team in 2018 covering all areas of health imaging after two years reporting on the hospital field. He holds a bachelor’s in English from UIC, and enjoys a good cup of coffee and an interesting documentary.