2.1.

Imaging System

The HRME is described in detail elsewhere.²⁰ Briefly, the system couples a compact fluorescence microscope with a coherent optical fiber bundle. The 1-mm diameter HRME probe is a coherent fiber bundle consisting of 30,000 individual fibers with a core-to-core spacing of $\sim 4\mu \mathrm{m}$ and a circular field-of-view (FoV) of $720\mu \mathrm{m}$ (IGN-08/30, Sumitomo Electric Industries). An LED centered at 455 nm (M455L2, Thorlabs, Newton, New Jersey) is used to provide illumination and a scientific CCD camera (Grasshopper 2, FLIR Integrated Imaging Solutions Inc, Richmond, Canada) is used for fluorescence imaging. Real-time videos are recorded and displayed using a laptop computer at a rate of 10 frames per second.

2.2.

Patient Enrollment and Imaging Procedure

An in vivo clinical trial was conducted in patients undergoing routine screening or surveillance colonoscopy at Mount Sinai Medical Center. The study information was provided to eligible patients and written informed consent was obtained. This study was approved by the IRBs at Mount Sinai Medical Center and Rice University. The clinical trial was registered on Clinicaltrials.gov (registration number NCT01384240).

Colonoscopy was performed by an expert endoscopist (S.A.) using a high-definition white-light endoscope. Polyps identified during white-light examination were further interrogated with the HRME in real time. Prior to HRME imaging, 1 to 4 ml of proflavine (0.01%) was topically applied through a spray catheter (Olympus America, Center Valley, Pennsylvania). The imaged polyps were then removed and submitted for histopathological examination. All biopsies were diagnosed by an expert gastrointestinal pathologist (A.D.P.) using standard criteria.

2.3.

Quantitative Image Analysis

The original HRME videos and images were reviewed for quality control by reviewers blinded to clinical impressions and histologic diagnosis. First, one reviewer selected frames of high quality from recorded videos. Typically, one or two frames were extracted from each 1-s video clip. Images were discarded if 50% or more of the FoV was not clearly visible (out-of-focus, debris, motion artifact or significant saturation). Second, two independent researchers reviewed the image library containing 200 images from 58 sites to select a single representative image per site with the best image quality. When heterogeneous candidate images of one site passed the initial quality control, previously established qualitative criteria by Chang et al.¹⁷ were applied to select a single image presenting the most severe imaging features.

A variety of quantitative features were calculated to evaluate their potential diagnostic ability (Table 1). Luminal morphology (e.g., lumen caliber, shape, and size) has been shown to be critical in the qualitative differentiation of neoplastic from benign polyps.¹⁷ To quantitatively evaluate these metrics, we developed an automated algorithm to segment lumens in HRME images, as illustrated in Fig. 1. First, images were preprocessed to remove the fiber bundle patterns with a low-pass filter and enhance the overall contrast. An initial segmentation was performed using a histogram-based threshold. Segmentation noise was then removed through morphologic processing including image opening and closing. In the next step, we identified segmented regions that fell on the borders of the images and outlined them in red. Since these outlined regions may either arise from noise along the borders or represent only a small fraction of imaged lumens, regions smaller than 1% of the image size were marked in gray and excluded from the analysis. Using the final segmentation results, we calculated the eccentricity, diameter (as the equivalent diameter of a circle with the same area), and perimeter for each of the segmented lumens. Their mean values, coefficient of variation (CV), skewness, and kurtosis on each image were used as separate features.

Table 1

Description of features calculated for each colon HRME image.

Fig. 1

Lumen segmentation algorithm in two representative images (one diagnosed as normal and one as tubular adenoma). Before segmentation, the fiber patterns were removed, and image contrast was enhanced. The images were then converted into a binary image and small structures arising from noise were removed. The next step consisted of identifying lumens on the image border (outlined in red) and removing these partial lumens if they contained only a small fraction of the total area (gray lumens outlined in red). Partial lumens on the border containing a significant luminal area (white lumens outlined in red) were included in the final segmentation image. TA: tubular adenoma.

It has also been shown that texture features can be used to extract important spatial information about glandular architecture.^21–25 Since the colon mucosa images show quasiperiodic 2-D luminal patterns and texture features, frequency analysis was performance in the Fourier frequency domain. A power spectrum was calculated after a 2-D Fourier transform and evenly divided into 10 partitions using a previously reported method.²¹ Specifically, a maximum square region within the FoV is used for frequency content analysis. Its discrete Fourier transform is divided into 10 partitions, with the first partition representing the lowest spatial frequency range and the 10th representing the highest. The frequency contents are summed up in each partition and their relative contributions to the total power spectral density are calculated, resulting in 10 distinct features that represent low to high spatial frequency ranges.

2.4.

Algorithm Development

The entire dataset was divided into a training/validation set and a test set. In the training/validation set that contains the first 40 images, we performed 10-fold cross-validation and selected the number of features to build a predictive model. The model was then evaluated using a separate test set that contains the remaining 18 images.

In the 40-image training/validation set, calculated features were assessed in the subsequent discriminant analysis to classify neoplastic (tubular adenoma, tubulovillous adenoma and sessile serrated adenoma, cancer) from non-neoplastic (normal mucosa, inflammatory polyp, hyperplastic polyp) polyps. First, the classification potential of each feature was evaluated with a Student’s $t$-test in a univariate analysis. The mean values of a single input feature in two groups were compared and its significance level was determined.

We also performed a multivariate analysis to build a two-class classifier based on a subset of up to five features. The number of features to construct the final predictive model was determined through a 10-fold cross-validation in the 40-image set. In each fold of the cross-validation, a 90%/10% split was used to separate the training set (36 images) and the validation set (4 images). First, a best performing k-feature model was selected to maximize the classification area under the curve (AUC) in the training set. Specifically, k features were used to compute the posterior probability through a linear discriminant analysis. A receiver operating characteristic (ROC) curve was constructed with histopathology as the gold standard and AUC was maximized to select a best performing model. Second, the model was validated in the validation set to calculate the classification error. For models that contain k features, the overall performance was evaluated using the cross-validation error, defined as the average classification error in 10 folds. The number of features in the final predictive model was determined to limit the cross-validation error with a small set of features. As the number of features was decided, the final predictive model was built in the full training/validation set and its performance in the separate test set was reported.

3.1.

Patient Enrollment: Sites and Images

We extracted images from original videos for 70 sites in 52 patients with corresponding histopathology. Images for 12 sites failed to pass the quality control and the reviewers reached consensus to select a single image per site for the remaining 58 sites in 46 patients. In this 58-image set, 36 sites are diagnosed as non-neoplastic (normal mucosa, inflammatory polyp, hyperplastic polyp) and 22 are diagnosed as neoplastic (tubular adenoma, tubulovillous adenoma, sessile serrated adenoma and cancer). The detailed histopathologic diagnosis is shown in Table 2.

Table 2

Site information based on histopathology.

Figure 2 shows representative images for sites diagnosed as normal mucosa (a), hyperplastic polyp (b, c), tubular adenoma (d, e), and tubulovillous adenoma (f) with the corresponding histopathology. Normal colon mucosa in Fig. 2(a) is characterized by the uniform glandular distribution across the entire image; round or oval openings are present with similar shapes and sizes. The lumens become slightly distorted in the hyperplastic polyp in Fig. 2(b); while some lumens appear elongated, they are confined by intact glandular borders. Both tubular adenoma and tubulovillous adenoma in Figs. 2(e) and 2(f) show larger and more linear lumens. In addition to images showing distinct benign or neoplastic features, images that reveal mixed morphology are also shown in Figs. 2(c) and 2(d). In Fig. 2(c), the lumens are more widened than Fig. 2(b) and can appear connected. In the tubular adenoma in Fig. 2(d), wide and linear openings, as well as small round lumens are observed, suggesting structural transitions between benign and neoplastic lesions. All these structural features are confirmed in the corresponding histopathology images.

Fig. 2

Representative in vivo images and the corresponding pathology. (a) The normal colorectal mucosa reveals uniform glandular distribution across the image. (b) The luminal size and shape are slightly varied with occasional widening in the hyperplastic polyp. (e, f) In contrast, both TA and TVA images show linear crypts; villous structures are observed in TVA. HRME images showing mixed morphology are also shown. (c) image is obtained from a hyperplastic site, which shows widened and sometimes connected lumens. (d) small round lumens, as well as wide and linear openings, are observed, and the site is diagnosed as TA. HP, hyperplastic polyp; TA, tubular adenoma; TVA, tubulovillous adenoma.

3.2.

Algorithm Development: Model Selection and Performance

Table 3 lists features that show statistically significant differences between the neoplastic and non-neoplastic groups in the training/validation set ($p<0.05$). The features are shown in descending order based on the $p$-value. Differences in the mean values are statistically significant for 6 of the 22 features.

Table 3

Features that show statistically significant differences between neoplastic and non-neoplastic groups (p<0.05).

In the multivariant analysis, we monitored the performance of k-feature models using cross-validation error as added features were selected from the 22-feature set. As shown in Fig. 3, the cross-validation error began to plateau when more than 2 features were used. As a result, two features were used to build the final predictive model.

Fig. 3

Cross-validation error for k-feature models. The linear classifiers evaluated in the plot comprise one to five features. As the number of features increased, the cross-validation error began to plateau when more than two features were used. As a result, a two-feature model was selected to build the final predictive model.

The final predictive model was constructed to maximize the AUC with two features in the training/validation set, and then evaluated in the test set. Selected features were frequency content 7 and mean of luminal perimeter. The ROC curve in the training/validation set shows an AUC of 0.93 for classification of neoplastic from non-neoplastic polyps; a posterior probability threshold was chosen at the Q-point, resulting in a sensitivity and specificity of 100% and 85%. The two-feature linear discriminant model was then tested in the test set, achieving a sensitivity and specificity of 100% and 80%, respectively. Scatter plots of the posterior probability in the two datasets are also shown in Fig. 4.

Fig. 4

Performance of the linear discriminant model based on two features. (a)–(c) The final predictive model was trained in the training/validation set, resulting in an AUC of 0.93, and a sensitivity and specificity of 100% and 85% at the Q-point. The model with a fixed posterior probability threshold was then evaluated in the test set, achieving a sensitivity and specificity of 100% and 80%, respectively.