Assessing an AI-based smart imagery framing and truthing (SIFT) system to assist radiologists annotating lung abnormalities on chest x-ray images for development of deep learning models

Lin Guo; Kunlei Hong; Ziqi Zhang; Bin Zheng; Stefan Jaeger; Jordan Fuhrman; Hui Li; Maryellen Giger; Andrei Gabrielian; Alex Rosenthal; Darrell E. Hurt; Ziv Yaniv; Y. M. Fleming Lure

doi:10.1117/12.2653826

7 April 2023 Assessing an AI-based smart imagery framing and truthing (SIFT) system to assist radiologists annotating lung abnormalities on chest x-ray images for development of deep learning models

Lin Guo, Kunlei Hong, Ziqi Zhang, Bin Zheng, Stefan Jaeger, Jordan Fuhrman, Hui Li, Maryellen Giger, Andrei Gabrielian, Alex Rosenthal, Darrell E. Hurt, Ziv Yaniv, Y. M. Fleming Lure

Author Affiliations +

Proceedings Volume 12465, Medical Imaging 2023: Computer-Aided Diagnosis; 124650R (2023) https://doi.org/10.1117/12.2653826
Event: SPIE Medical Imaging, 2023, San Diego, California, United States

Abstract

To assess a Smart Imagery Framing and Truthing (SIFT) system in automatically labeling and annotating chest X-ray (CXR) images with multiple diseases as an assist to radiologists on multi-disease CXRs. SIFT system was developed by integrating a convolutional neural network based-augmented MaskR-CNN and a multi-layer perceptron neural network. It is trained with images containing 307,415 ROIs representing 69 different abnormalities and 67,071 normal CXRs. SIFT automatically labels ROIs with a specific type of abnormality, annotates fine-grained boundary, gives confidence score, and recommends other possible types of abnormality. An independent set of 178 CXRs containing 272 ROIs depicting five different abnormalities including pulmonary tuberculosis, pulmonary nodule, pneumonia, COVID-19, and fibrogenesis was used to evaluate radiologists’ performance based on three radiologists in a double-blinded study. The radiologist first manually annotated each ROI without SIFT. Two weeks later, the radiologist annotated the same ROIs with SIFT aid to generate final results. Evaluation of consistency, efficiency and accuracy for radiologists with and without SIFT was conducted. After using SIFT, radiologists accept 93% SIFT annotated area, and variation across annotated area reduce by 28.23%. Inter-observer variation improves by 25.27% on averaged IOU. The consensus true positive rate increases by 5.00% (p=0.16), and false positive rate decreases by 27.70% (p<0.001). The radiologist’s time to annotate these cases decreases by 42.30%. Performance in labelling abnormalities statistically remains the same. Independent observer study showed that SIFT is a promising step toward improving the consistency and efficiency of annotation, which is important for improving clinical X-ray diagnostic and monitoring efficiency.

Conference Presentation

Citation Download Citation

Lin Guo, Kunlei Hong, Ziqi Zhang, Bin Zheng, Stefan Jaeger, Jordan Fuhrman, Hui Li, Maryellen Giger, Andrei Gabrielian, Alex Rosenthal, Darrell E. Hurt, Ziv Yaniv, and Y. M. Fleming Lure "Assessing an AI-based smart imagery framing and truthing (SIFT) system to assist radiologists annotating lung abnormalities on chest x-ray images for development of deep learning models", Proc. SPIE 12465, Medical Imaging 2023: Computer-Aided Diagnosis, 124650R (7 April 2023); https://doi.org/10.1117/12.2653826

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available