An analysis of two ground truth estimation methods

A. M. Biancardi; A. C. Jirapatnakul; S. Fotin; T. Apanasovich; A. P. Reeves

doi:10.1117/12.813440

27 February 2009 An analysis of two ground truth estimation methods

A. M. Biancardi, A. C. Jirapatnakul, S. Fotin, T. Apanasovich, A. P. Reeves

Proceedings Volume 7260, Medical Imaging 2009: Computer-Aided Diagnosis; 72600E (2009) https://doi.org/10.1117/12.813440
Event: SPIE Medical Imaging, 2009, Lake Buena Vista (Orlando Area), Florida, United States

Abstract

An estimation of the so called Ground Truth (GT), i.e. the actual lesion region, can minimize readers' subjectivity if multiple readers' markings are combined. Two methods perform this estimate by considering the spatial location of voxels: Thresholded Probability-Map (TPM) and Simultaneous Truth and Performance Level Estimation (STAPLE). An analysis of these two methods has already been performed. The purpose of this study, however, is gaining a new insight into the method outcomes by comparing the estimated regions. A subset of the publicly available Lung Image Database Consortium archive was used, selecting pulmonary nodules documented by all four radiologists. The TPM estimator was computed by assigning to each voxel a value equal to average number of readers that included such voxel in their markings and then applying a threshold of 0.5. Our STAPLE implementation is loosely based on a version from ITK, to which we added the graph cut post-processing. The pair-wise similarities between the estimated ground truths were analyzed by computing the respective Jaccard coefficients. Then, the sign test of the differences between the volumes of TPM and STAPLE was performed. A total of 35 nodules documented on 26 scans by all four radiologists were available. The spatial agreement had a one-sided 90% Confidence Interval of [0.92, 1.00]. The sign test of the differences had a p-value less than 0.001. We found that (a) the differences in their volume estimates are statistically significant, (b) the spatial disagreement between the two estimators is almost completely due to the exclusion of voxels marked by exactly two readers, (c) STAPLE tends to weight more, in its GT estimate, readers marking broader regions.

Citation Download Citation

A. M. Biancardi, A. C. Jirapatnakul, S. Fotin, T. Apanasovich, and A. P. Reeves "An analysis of two ground truth estimation methods", Proc. SPIE 7260, Medical Imaging 2009: Computer-Aided Diagnosis, 72600E (27 February 2009); https://doi.org/10.1117/12.813440

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available