Investigators have developed and tested a hybrid reading strategy for screening mammography images with artificial intelligence (AI) that includes reads from radiologists and a stand-alone AI interpretation of mammograms with an uncertainty quantification. According to study findings published in Radiology, the hybrid model led to a reduction in radiologist workload by 38% and the results of the hybrid strategy matched that of standard double radiologist reads in terms of recall rate and cancer detection rate.
“Although the overall performance of state-of-the-art AI models is very high, AI sometimes makes mistakes,” said lead study author Sarah D. Verboom, MSc, a doctoral candidate in the Department of Medical Imaging at Radboud University Medical Center in the Netherlands. “Identifying exams in which AI interpretation is unreliable is crucial to allow for and optimize use of AI models in breast cancer screening programs.”
“We can achieve this performance level if the AI model provides not only an assessment of the probability of malignancy for a case but also a rating of its certainty of that assessment,” Ms. Verboom added. “The key component of our study isn’t necessarily that this is the best way to split the workload, but that it’s helpful to have uncertainty quantification built into AI models.”
Study Methods
The investigators developed an AI model that would output a probability of malignancy based on a reading of the mammogram as well as a measure of its certainty or uncertainty regarding the result of its read. They established a hybrid reading approach where recall decisions from the AI model alone would be sufficient if the model was confident in its reading, and a radiologist double reading would be required if the model was not confident in its reading.
This hybrid strategy was tested on a data set of 41,469 mammogram exams from 15,522 women that was previously unseen by the AI model. The scans were all collected between July 2003 and August 2018 in Utrecht, Netherlands, as part of the Dutch National Breast Cancer Screening Program. The data set included 332 screen-detected cancers and 34 interval cancers. They split the data set into two groups to establish optimal thresholds for the hybrid reading strategy and evaluate the strategy in the second group.
Key Study Findings
Of the eight uncertainty metrics evaluated by the researchers, the best-performing uncertainty metric was the uncertainty of the mean probability of malignancy score of one suspicious region. A total of 61.9% of scans were read by radiologists.
Hybrid reading demonstrated a recall rate of 23.6% (95% confidence interval [CI] = 21.6%–25.5%) and a cancer detection rate of 6.6% (95% CI = 5.5%–7.7%). These rates were similar to that of a standard double reading at a 23.9% recall rate (95% CI = 21.9%–25.8%; P = .27) and a cancer detection rate of 6.6% (95% CI = 5.5%–7.7%; P = .14).
The model’s area under the curve was lower for mammogram scans with uncertain reads than for scans with certain reads (0.87 vs 0.96; P = .02). The sensitivity of the AI model’s reads was 85.4% compared with 88.9% from the double radiologist readings.
Younger women with dense breast tissue were more likely to have uncertain readings from the AI model.
“The use of AI with uncertainty quantification can be a possible solution for workforce shortages and could help build trust in the implementation of AI,” Ms. Verboom said, adding that further research is needed to determine how the hybrid reading strategy could decrease radiologist reading time.
“I think in the future, we could get to a point where a portion of women are sent home without ever having a radiologist look at their mammogram because AI will determine that their exam is normal,” she concluded. “We’re not there yet, but I think we could get there with this uncertainty metric and quality control.”
Disclosure: The study was funded by the Dutch Research Council, Dutch Cancer Society, and Health Holland. For full disclosures of the study authors, visit pubs.rsna.org.
