Context in Medical Imaging: The Case of Focal Liver Lesion Classification

Abstract

There are many different sources of information beyond the actual images that serve to inform radiologists when they are making diagnoses. Therefore, it is important to include such information, referred to as context, in machine learning algorithms designed for automated classification. To better utilize and to explore the power of context in medical imaging we developed a classification algorithm based on convolutional neural networks (CNNs) with various contexts to classify liver lesions in multi-phase computed tomography data. We designed an algorithm, referred to as sliced 3D, to efficiently handle 3D context in the image data. We further included clinical context summarizing the patient’s medical status to improve performance, and we also exploited the presence or absence of other lesions in the volume to inform classification of a lesion under consideration. The effect of co-occurrence is learned from the data during the training process. The algorithm was developed, validated, and tested on 2205 multi-vendor multi-institution studies. We found that the sliced 3D algorithm performed better than equivalent 2D and 3D CNN baselines (average F1=0.629 vs F1=0.589). Using the clinical and co-occurrence contexts further improved the algorithm’s performance (average F1=0.734). Our evaluation demonstrates that this novel CNN architecture, in conjunction with additional information about medical status and lesion co-occurrence, substantially improves classification results.