1.

Introduction

Optical coherence tomography (OCT) provides high-resolution, cross-sectional tomographic images of the human retina and permits direct evaluation of retinal thickness.¹ Recent technological developments in spectral-domain OCT (SDOCT) have greatly increased imaging capabilities compared to earlier time-domain technology. SDOCT provides estimates of retinal layer thicknesses across the macula to aid in clinical diagnosis and treatment decisions for a variety of ocular diseases.^2–6 Interpretation of data has been complicated by the variety of platforms designed by commercial SDOCT instrument manufacturers, each with different proprietary software technologies. Previous studies have identified OCT-derived retinal thickness measurement variability due to differences in their segmentation algorithms, their reported axial resolutions in tissue, their scan density options, and their ability to correct for subject fixation.^7–13 Additional anatomic factors vary between individual patients, including axial length, refractive focal length, and macular curvature.¹⁴ These anatomic variations may affect the accuracy of comparing lateral and axial measurements between SDOCT instruments in clinical studies.¹⁴ Other studies have addressed measurement differences inherent to individual instruments with the same time-domain OCT (TDOCT) platform.^15–17 These TDOCT studies have used large sample sizes and built-in retinal segmentation software to show retinal thickness measurements with widespread variation between instruments, but differences reported in each study were not consistent.^15–17

A model eye eliminates variability caused by anatomic differences between human patients and by potential morphologic changes between imaging sessions due to diurnal fluctuations, vascular changes, head tilt, or subject fixation. In a recent study, a customized model eye with a retinal nerve fiber layer phantom has been used to assess thickness differences between SDOCT platforms and individual instruments.¹⁸ However, this study used automated retinal segmentation software from each SDOCT platform, which causes reproducible thickness differences between platforms by using different anatomic definitions to identify retinal layer boundaries.^7–10 Furthermore, previous studies have not addressed SDOCT measurements of lateral width, which are important for novel SDOCT methods of disease analysis, such as drusen diameter and geographic atrophy in age-related macular degeneration.⁶

Accurate interpretation of retinal measurements for the treatment of macular diseases and for clinical research requires consistency and reproducibility between different SDOCT platforms and between instruments from the same platform. Significant differences in the quantitative measurements obtained manually from different SDOCT platforms may support the use of a conversion scale to compare data obtained from different systems. The purpose of this study is to determine the variability of lateral and axial retinal measurements among SDOCT instruments from the same commercial platform and across different systems.

2.

Methods

2.3.

SDOCT Measurements and Statistical Analysis

Two graders viewed all SDOCT scans and agreed upon the one B-scan with the minimum central thickness that best approximated the foveal center of the retinal tissue phantom. Images were viewed in each platform’s standard display screen, and image processing was not allowed (i.e., magnification, brightness, contrast, summation, or Gaussian smoothing). Each grader performed measurements on the central B-scan of 10 raster scans obtained with each SDOCT instrument in masked and independent fashion. We selected anatomic landmarks on the tissue phantom that could be readily identified and measured in the lateral or axial planes of the central B-scan image. The lateral measurement was performed on the lateral width (LW) of the tissue phantom. Axial measurements were performed on the central foveal thickness (CFT), parafoveal thickness (PFT) at 1 mm to the left of center, and PFT at 1 mm to the right of center. These measurements included the largest dimensions of the tissue phantom in the lateral and axial planes in order to capture as much range of error as possible across SDOCT platforms. Figure 1 shows the borders defined for each manual measurement on different SDOCT platforms. Instruments from the same SDOCT platform had the same version of software and built-in screen calipers to take manual measurements. On all platforms, measurement accuracy was limited by pixel resolution and automatically converted to microns or millimeters by built-in software.

Fig. 1

Model eye measurements were obtained for the fixed lateral width, center foveal thickness, and parafoveal thickness 1 mm to the left and right of center. The B-scan with the widest width and minimum central thickness of the circular tissue phantom was selected for measurements. Lateral width was defined as the horizontal distance at the base. Axial thickness was defined as the vertical distance from the inner border of the hyperreflective inner retinal surface to the inner border of the hyperreflective base substrate beneath the tissue phantom. Values shown here are mean values obtained with each spectral-domain optical coherence tomography platform.

Intergrader reproducibility of retinal measurements was assessed with intraclass correlation coefficients (ICC) and 95% confidence intervals (CI). Due to high intergrader agreement, data from both graders were combined to assess intraplatform variability between instruments and interplatform variability between SDOCT systems. Coefficients of variance (COV) were calculated for each instrument and measurement, and instruments were compared with two-tailed $t$-tests. Intra- and interplatform differences for each measurement were assessed with analysis of variance models and Tukey-Kramer tests. All statistical analysis was performed with SAS statistical modeling software (SAS JMP 10, SAS Institute, Cary, North Carolina), and $p$ values $<0.05$ were considered statistically significant.

3.

Results

Qualitative image differences were observed between SDOCT platforms (Fig. 1). Spectralis instruments suppressed the most reflections, but signal suppression also complicated layer identification and observer measurements. Cirrus scan images appeared to be more saturated, illustrated by broadening of the hyperreflective bands created by laminations within the tissue phantom. Images from the Bioptigen systems (Envisu and SDOIS) had intermediate signal strength and were similar in appearance to each other.

4.

Discussion

This study examined the variability in lateral and axial manual measurements between several commercial SDOCT platforms. Dimensions were measured by hand with each instrument’s caliper tool, rather than by the manufacturer’s segmentation program. A single model eye was used to test for variability and to serve as a standardized solid-state target for SDOCT imaging. Under consistent imaging conditions, we found statistically significant differences in all lateral and axial manual measurements between instruments from the same manufacturer and different manufacturers, but intraplatform differences between instruments were significantly smaller than interplatform differences. From these results, we generated conversion factors to facilitate the comparison of manual measurements between different SDOCT platforms in future clinical trials and in daily treatment of macular diseases.

Before the appearance of numerous commercial SDOCT systems, several studies looked at errors and variability between instruments with the same platform.^15–17 Barkana et al. evaluated several TDOCT instruments and they found substantial differences between devices, few being statistically significant.¹⁶ Interestingly, they found that the differences observed were significantly correlated with signal strength. Our findings differ from Barkana et al. and others, who reported no statistically significant difference between instruments.^15–17 However, these reports only evaluated TDOCT instruments and had higher standard deviation of thickness measurements than recent SDOCT studies, in part due to the inferior pixel resolution of TDOCT systems.^7–10

This study is the first to rigorously compare quantitative manual measurements from several commercial platforms utilizing a commercially available model eye. We decided to evaluate two commercial platforms that are commonly used in human adult imaging, clinical research, and randomized clinical trials.^2–4 We chose a commercial hand-held portable platform approved for retinal imaging in pediatric human subjects^5,14,20–22 and in basic animal research.^23–25 Furthermore, the largest ongoing randomized trial for age-related macular degeneration (AMD), the NEI-sponsored Age-Related Eye Disease Study 2, exclusively allows the Bioptigen SDOIS platform for its longitudinal, observational ancillary SDOCT study (AREDS2 Ancillary SDOCT Study).^6,26 The baseline dataset and measurements for both control and AMD eyes in this study has been made publicly available.⁶

Several studies have concluded that comparing retinal thickness with instruments from different manufacturers is not advised for clinical studies.^7–10 Determining the true variability in these measurements with a cohort of patients would be biased by errors in lateral and axial scaling. For example, Spectralis machines are programmed to offer scan parameters based on degrees of visual angle; however, it provides caliper measurements in millimeter distance. The same visual angle would span a shorter diameter in an eye with shorter axial length, but the distance would be converted to the same millimeter distance as a scan distance on a longer eye. Axial measurement differences may be caused by variability in the default algorithms for automated segmentation line placement, refractive index correction, or dispersion compensation across different SDOCT platforms. Since these calculations are proprietary components of each platform’s software, it is difficult for third party investigators to test their separate contributions to measurement variability.

We have also demonstrated statistically significant variability in manual measurements of a single retinal tissue phantom between two different instruments with the same SDOCT platform. Variability between these instruments may result from inherent variability in the optical path length measured at two different time points, variability in the degree of decalibration between instruments that occurs over time with regular use, or measurement variability caused by speckle noise. We attempted to control for decalibration by selecting same-platform instruments with similar frequency of use in daily clinical care. In SDOCT, speckle noise results from interference between densely packed reflectors, reducing contrast between highly scattering structures in tissue.²⁷ However, the averaging methods commonly used by commercial SDOCT platforms were not applicable to the motionless imaging protocol of this study, where speckle noise was highly correlated across images and instruments. Figure 1 showed acceptably low image noise, and even state-of-the-art denoising algorithms produce some level of image blur,²⁷ permitting us to perform measurements on the unprocessed images shown. Based on the small differences between graders (Table 1) and between same-platform instruments (Table 2), we concluded there was negligible effect of speckle noise on measurement variability.

Measurement differences between platforms were statistically significant; however, the clinical significance of this difference is less clear. With the exception of the Bioptigen SDOIS, the SDOCT systems evaluated in this study had low variability from a clinical standpoint, albeit statistically significant. Lateral scaling variability was 0.3 to 2% between platforms, which represents a range of 15 to 106 μm in width difference between images (based on nominal 6-mm scans divided by sampling density of 512 A-scans). Axial measurements performed in this study suggest that variability across all platforms was 1.1 to 17% between platforms, equivalent to a difference of 5 to 45 μm based on the nominal axial resolution of these SDOCT platforms. Excluding the axial measurements from the Bioptigen SDOIS, which were consistently smaller than all other platforms, the mean difference decreased to $<3.7\%$ ($\sim 1.3\mathrm{pixels}$ or 8 μm) across the other three systems. Low variability between Cirrus, Spectralis, and the portable Envisu system suggested that hand motion or instability of a human operator does not introduce additional error while holding the hand-held probe over the target. These differences may not affect disease management with uniform scanning protocols and manual measurements based on the small number of pixels required for the observed differences and the larger errors associated with automated segmentation, sampling density, and fixation variability.^7,10–13 However, clinical studies gathering repeated measurements over time to evaluate disease modification may obtain statistically significant differences that remain within the range of instrument variability.

In conclusion, we have shown significantly greater variability across different platforms than between instruments from the same platform, while controlling for the influence of anatomic variations in human imaging and differences created by automated segmentation programs. This report suggests that clinical investigators may need to account for inherent variances in quantitative SDOCT data collected for clinical trials and routine patient follow-up. Standardized conversion factors may improve the accuracy of data collected from different SDOCT platforms. These conversion tools require further validation with larger samples and human imaging studies. We note that optical imaging instruments may perform differently with eyes of different axial length, refraction, and optical scattering. Accurate quantification of such parameters is part of our ongoing research. Robust, precise, and reproducible conversion factors between commercial SDOCT platforms may allow for the use of a greater range of SDOCT systems in clinical studies and can improve the clinical interpretation of statistically significant differences obtained from study results.