Generalisation challenges in deep learning models for medical imagery: insights from external validation of COVID-19 classifiers.

Haynes, Sophie Crawford; Johnston, Pamela; Elyan, Eyad

Abstract

The generalisability of deep neural network classifiers is emerging as one of the most important challenges of our time. The recent COVID-19 pandemic led to a surge of deep learning publications that proposed novel models for the detection of COVID-19 from chest x-rays (CXRs). However, despite the many outstanding metrics reported, such models have failed to achieve widespread adoption into clinical settings. The significant risk of real-world generalisation failure has repeatedly been cited as one of the most critical concerns, and is a concern that extends into general medical image modelling. In this study, we propose a new dataset protocol and, using this, perform a thorough cross-dataset evaluation of deep neural networks when trained on a small COVID-19 dataset, comparable to those used extensively in recent literature. This allows us to quantify the degree to which these models can generalise when trained on challenging, limited medical datasets. We also introduce a novel occlusion evaluation to quantify model reliance on shortcut features. Our results indicate that models initialised with ImageNet weights then fine-tuned on small COVID-19 datasets, a standard approach in the literature, facilitate the learning of shortcut features, resulting in unreliable, poorly generalising models. In contrast, pre-training on related CXR imagery can stabilise cross-dataset performance. The CXR pre-trained models demonstrated a significantly smaller generalisation drop and reduced feature dependence outwith the lung region, as indicated by our occlusion test. This paper demonstrates the challenging problem of model generalisation, and the need for further research on developing techniques that will produce reliable, generalisable models when learning with limited datasets.

Citation

HAYNES, S.C., JOHNSTON, P. and ELYAN, E. 2024. Generalisation challenges in deep learning models for medical imagery: insights from external validation of COVID-19 classifiers. Multimedia tools and applications [online], 83(31), pages 76753-76772. Available from: https://doi.org/10.1007/s11042-024-18543-y

• Journal Article Type: Article
• Acceptance Date: Jan 29, 2024
• Online Publication Date: Feb 21, 2024
• Publication Date: Sep 30, 2024
• Deposit Date: Feb 2, 2024
• Publicly Available Date: Mar 1, 2024
• Journal: Multimedia tools and applications
• Print ISSN: 1380-7501
• Electronic ISSN: 1573-7721
• Publisher: Springer
• Peer Reviewed: Peer Reviewed
• Volume: 83
• Issue: 31
• Pages: 76753-76772
• DOI: https://doi.org/10.1007/s11042-024-18543-y
• Keywords: COVID-19; Deep learning; Generalization; X-ray imaging; Medical imagery
• Public URL: https://rgu-repository.worktribe.com/output/2226023