1.

Introduction

Surgeons rely on medical imaging to plan surgery and to accurately locate the lesions. Currently, many traditional modalities, such as x-ray radiography, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, positron emission tomography, and single-photon emission computed tomography, are widely used in daily clinical practice.^1,2 These preoperative images greatly assist in disease diagnosis and surgical planning. However, preoperative imaging modalities, such as CT and MRI, suffer from several limitations, including difficulties in adapting these systems to the operating room (OR) environment as well as slow scan time. For abdominal surgeries, where organs typically deform, the practicality of using preoperative images for surgical planning presents unique challenges that require the development of new intraoperative imaging modalities.

Traditionally, fluoroscopy and intraoperative ultrasound (IUS) are deployed in the OR for guiding surgery.^3,4 These imaging modalities provide real-time feedback to complement the preoperative imaging information available to the surgeon. Yet, fluoroscopy provides poor contrast for soft tissues, limiting its utility in abdominal surgery. IUS is a contact structural imaging method that suffers from low contrast and small field of view. Additionally, both fluoroscopy and IUS do not provide molecular and functional information that enhances the accuracy of oncologic imaging. Recent studies have shown that fluorescence imaging fills this void.^5–8

The development of diverse fluorescent exogenous molecular probes has enabled the reporting of intricate molecular and cellular processes at an unparalleled sensitivity.^5,7,9,10 Recent efforts to adapt fluorescence microscopy to organ imaging have expanded fluorescence molecular imaging to the clinical arena.^11,12 With near-infrared (NIR) probes, fluorescence imaging can penetrate deeper into tissues than visible light imaging because of the reduced scattering and absorption of NIR light.^11,12

Leveraging the strengths of NIR fluorescence imaging, we have previously developed a prototype fluorescence goggle system to guide surgery.¹³ The prototype goggle system was a compact wearable fluorescence imaging and display system based on the CCD imaging technology and a monocular eyepiece display.¹³ The line-of-view of the surgeon was synchronized with the system, and hands free imaging was enabled.

In the current study, we report the development of a newer version of the goggle system using an NIR-sensitive fast complementary metal–oxide–semiconductor (CMOS) sensor and a video-optical, dual-modal see-through head-mounted display (HMD). We characterized the new system and tested it in surgical studies in rodents.

2.

Materials and Methods

3.

Results

The new goggle system is a compact see-through wearable system that enables real-time fluorescence imaging capability for up to 60 frames per second (fps) (Fig. 1). Compared to the CCD sensor used in the previous prototype (approximately 20% quantum efficiency at 830 nm), the CMOS sensor used in the current system has a high quantum efficiency at 830 nm ($>30\%$). The combination of high frame rate and high NIR quantum efficiency makes the sensor ideal for intraoperative NIR fluorescence imaging. The wearer has the option to choose either automatic or manual gain as well as either automatic or manual exposure time. The automatic gain and exposure time can present the optimized contrast to the surgeons without manually adjusting it during surgical procedures.

Fig. 1

The fluorescence goggle system. (a) A color picture and (b) setup of the fluorescence goggle system.

We demonstrated the first application of a video-optical dual-modal see-through display in NIR fluorescence imaging. When the video mode was turned off, the wearer could visualize the surroundings with natural vision. When the video mode was turned on, the real time NIR fluorescence video was presented to the wearer with high contrast. The flexibility of a switchable see-through display simplified the surgical workflow and allowed the surgeon to visualize the surgical bed with natural vision or enhanced vision as needed.

We first evaluated the sensitivity of the goggle system by measuring the SNR of different ICG concentrations ranging from 100 pM to 700 nM [Fig. 2(a)]. For this study, the system was operated with an exposure time of 15.25 ms and a frame rate of 28 fps. We demonstrated that the goggle could detect fluorescence signal from as low as 300 pM ICG. Noise analysis model of the CMOS imaging sensor^19,20 indicated that photon shot noise dominated reset and readout noise at an SNR of 30, which corresponded to fluorescence signal from approximately 300 nM of ICG detected at 28 fps.

Fig. 2

Near-infrared (NIR) fluorescence detection sensitivity of the fluorescence goggle system. (a) Signal-to-noise ratio (SNR) of fluorescence signal versus indocyanine green (ICG) concentration is plotted in linear scale. (b) SNR of fluorescence signal versus ICG concentration is plotted in log scale. Measurement results for binning and temporal averaging methods are also plotted. (c) SNR of fluorescence signal versus exposure time is plotted in linear scale.

We also measured the SNR of the fluorescence signal using optional features including pixel binning and temporal averaging methods [Fig. 2(b)]. In this study, the system was operated with an exposure time of 45.72 ms and a frame rate of 10 fps. The SNR improved approximately by a factor of 2 when either binning or temporal averaging methods are implemented independently and approximately by a factor of 4 when both methods are implemented together. Pixel binning involved combining signals from a group of pixels in the spatial domain, which was analogous to increasing the number of photons that contribute to the detected signal. Binning improved the SNR by the square root of the number of pixels binned. Thus binning a neighborhood of $2\times 2\text{pixels}$ improved the SNR approximately by a factor of 2. Temporal averaging involved combining signals from a group of pixels in time domain, which was also analogous to increasing the number of photons that contribute to the detected signal. Similarly, temporal averaging increased the SNR by the square root of the number of averaged pixels in time domain. Hence, temporal averaging of four consecutive image frames increased the SNR approximately by a factor of 2. Both temporal averaging and pixel binning were combined together to further improve the SNR of the sensor. Averaging four frames coupled with averaging a pixel neighborhood of $2\times 2\text{pixels}$ improved the SNR approximately by a factor of 4, as expected.

In addition, we also evaluated sensitivity of the goggle system by measuring the SNR versus exposure time for 3 and 10 nM ICG samples. SNR increased linearly with exposure time at a rate that depends on ICG concentration [Fig. 2(c)]. This was due to the fact that for a longer exposure time, the photodiodes accumulated more photons and, hence, the contrast between the target and background increased linearly. As the exposure time increased, the SNR of the florescence signal increased linearly at the cost of a slower frame rate. For example, if the exposure time is set to 20 ms, the SNR is 4, and the frame rate is 50 fps. If the exposure time is 100 ms, the SNR is 20, and the frame rate is 10 fps.

After evaluating the sensitivity of the new goggle imaging system, we deployed it in surgical oncology using rodents. We first investigated the use of new system in SLNs mapping. PEGylated CdTeSe/ZnS QDs with emission at 800 nm (PEG_QD800) were used for this purpose because they do not extravasate rapidly from the lymphatic vessel. Upon injection, the goggle system rapidly detected the QD fluorescence signal at the axillary region in the rats within 30 min (Fig. 3). Under the goggle guidance, the lymph nodes were rapidly located and excised. The putative SLNs were also compared with the control lymph nodes ex vivo [Fig. 4(a)–4(c)]. The SLN-to-control fluorescence ratio is $4.78\pm 2.87$ ($\text{mean}\pm \mathrm{std}$). The SLNs were examined by histology, which confirmed that all of the six putative SLNs removed were actual SLNs [Fig. 4(d)].

Fig. 3

Sentinel lymph node (SLN) mapping using new fluorescence goggle system and quantum dots (QDs). (a) NIR fluorescence image, (b) reflectance image, and (c) merged image of (a) and (b) of a rat at 5 min postinjection. NIR fluorescence is pseudocolored in green in (c). Fluorescence goggle readily detects the SLNs at the axillary positions (blue arrows).

Fig. 4

Ex vivo examination and histology of resected lymph nodes. (a) NIR fluorescence image, (b) reflectance image, and (c) merged image of (a) and (b). Blue arrows indicate the putative SLNs, while purple arrows indicate an ischiatic lymph node taken from the hindquarter as a nonfluorescent control. (d) A representative image of SLNs with hematoxylin and eosin (H&E) staining.

Furthermore, we demonstrated the utility of new goggle system in intraoperative molecular imaging of cancer. For this purpose, a fluorescent molecular probe LS301 was used to detect breast cancer liver metastases.¹³ At 24 h postinjection, the fluorescence goggle successfully detected the metastases intraoperatively [Fig. 5(a)–5(c)]. In addition, small tumors of approximately 1 mm in diameter were also detected. These tumors were not evident by visual inspection with the naked eye or by palpation. Under the goggle guidance, the metastatic lesions were successfully located and removed. At the time of surgery, the liver also showed relatively higher fluorescence signal than other organs, because it is the major excretion organ for the molecular probes. After resection, the local fluorescence in the surgical bed decreased [Fig. 5(d)]. Ex vivo examination confirmed the in vivo findings [Fig. 6(a)–6(c)]. The histology further confirmed the cancerous nature of tissues excised, as evidenced by their epithelial inclusion [Fig. 6(d)].

Fig. 5

Intraoperative imaging of breast cancer metastases in the liver using new goggle system and LS301. (a) NIR fluorescence image, (b) reflectance image, and (c) merged image of (a) and (b) of a mice at 24 h postinjection. (d) Fluorescence image after surgical resection of metastases. Blue arrows indicate the metastasis, red arrow indicates the surgical bed after resection, and purple arrows indicate the liver. The goggle clearly detected nonobvious small metastases in the liver. After resection of metastases, the fluorescence level in the tumor bed decreased. Healthy liver also showed moderate fluorescence signal due to the clearance of the contrast agents by the hepatobiliary pathway.

Fig. 6

Ex vivo examination and histology of excised metastases. (a) NIR fluorescence image, (b) reflectance image, and (c) merged image of (a) and (b). In (a)–(c), blue arrows indicate the putative metastasis, while purple arrows indicate the healthy liver. (d) Representative image of liver metastases with H&E staining. Dashed blue lines outline the metastasis in (d).

4.

Discussion

In this study, we developed a new fluorescence goggle system with fast temporal resolution and dual-modal see-through display. The current system uses a NIR-sensitive CMOS sensor, unlike the CCD sensor reported in the previous goggle system.¹³ This CMOS sensor has a quantum efficiency of over 30% at 830 nm spectral window, which is ideal for NIR imaging. The picomolar level fluorescence detection sensitivity provides a powerful platform for imaging fluorescent molecular information intraoperatively. Furthermore, the CMOS technology facilitated imaging at a faster frame rate without the need of binning using the new goggle system. A frame rate of up to 60 fps was realized for in vivo imaging. Compared to the previous generation goggle system, which can operate at up to 30 fps, the frame rate of the new system has further improved to beyond video rate. This is peculiarly desirable for intraoperative imaging, particularly when organs of interests are in constant motion due to many physiological processes such as respiration. For liver surgery, high-temporal resolution is crucial to overcome the large range of liver motion.

The new goggle system features novel display technology. To the best of our knowledge, this is for the first time a dual-modal see-through display technology is used for intraoperative fluorescence imaging. As a wearable imaging and display system, an integral part of fluorescence goggle system is the display module. Leveraging on the existing technology platforms of HMDs, the goggle system is empowered to present the imaging data to the surgeon in a convenient way. HMDs have been extensively used for military purposes in the past.²¹ They have also been adapted for personal entertainment purposes in recent years. Currently, two types of HMDs have been developed including video see-through HMDs and optical see-through HMDs. In the current system, a switchable optical-video HMD is used. The new goggle system incorporates the capability to switch between the two display modes, a capability which was previously unavailable. This simplifies the surgical workflow for open surgery and enables the surgeon to use their natural vision as needed without taking off the goggle.

From the characterization study, we observed a trend that is in accordance with theoretical understanding of CMOS technologies. The intensity signal measured by a CMOS detector is given by Eq. (2), where $P$ is optical power of the incident light, $\lambda$ is wavelength of the incident light, $A$ is photosensitive area of the pixel, $t$ is exposure time, QE is quantum efficiency of detector, $h$ is Planck’s constant, and $c$ is speed of light. Equation (2) reveals the linear dependence relationship between the intensity signal and quantum efficiency. Quantum efficiency of the current detector greater than 30% at 830 nm is highly desirable for NIR fluorescence imaging purposes. However, the temporal noise limits performance of the imaging system, which is generated during reset, integration, and readout operations. The total noise is governed by Eq. (3), where $n_{\mathrm{rst}}$ is reset noise, $n_{\text{read}}$ is readout noise, and $n_{\text{photon}}$ is photon noise. Analysis of temporal noise in CMOS image sensors indicates that reset and readout noises dominate at low illumination, while photon noise dominates at high illumination.^19,20 The ratio of signal to total noise is SNR, which describes the sensitivity performance of the detector. The number of photons incident on a pixel follows Poisson distribution. Therefore, photon noise has a square root relationship with the number of incident photons. Consequently, SNR of an image sensor that is limited by photon noise has a square root relationship with the number of incident photons. An increase in ICG concentration or exposure time increases the number of incident photons, and thus an improvement in SNR. In addition, pixel binning and temporal averaging are implemented as optional features in this goggle system. These features improve the SNR with trade-off in spatial and temporal resolutions. They may be used in extremely low light imaging conditions, as needed.

In conjunction with QDs, we demonstrated that the new goggle system could be used to map the SLN. QDs have several advantages over ICG that was used in the previous SLN study.¹³ The higher quantum yield and photostability render the QDs better imaging agent than ICG.¹⁰ The new goggle system is versatile, as demonstrated by its compatibility with diverse NIR tracers including small organic molecules and inorganic nanoparticles. Compared to the mouse models used in the previous study,¹³ the rat model used in this study reflects the scalability of the system and potential translation to human lymph biopsy.

Furthermore, we demonstrated the potential of combining the goggle system with a targeted molecular agent LS301 to guide resection of breast cancer metastases in the liver. Small lesions which could have been otherwise missed were readily detected. The intraoperative fluorescence image guidance complemented the naked eye observation and palpation, which were not sensitive to small lesions. In future study, the fluorescence imaging strategy could be coupled with IUS to further facilitate cancer detection. The high resolution of fluorescence imaging at the liver surface would complement IUS, which has difficulties in resolving surface-located, isoechoic, and small lesions.

Although our current findings are exciting, there is room for improvement. For instance, the current system cannot fully store all imaging data when operating beyond 30 fps, due to the limited bandwidth for data transfer imposed by USB 2.0. In the future, other data transfer standard with larger bandwidth will be utilized to solve this problem. Another way to overcome this challenge is to compress the data with data-compression chip. Another area for improvement is the control module, which currently consists of an FPGA board and a laptop. We plan to develop an embedded system to fully circumvent the need for a PC. Furthermore, the current system uses off-the-shelf dual-modal see-through HMDs primarily designed for entertainment purposes. Although it is useful for demonstrating the concept and feasibility, we will also develop a customized and ergonomical display unit optimized for medical imaging.

5.

Conclusion

We have developed a new fluorescence goggle system based on CMOS sensor and see-through display technologies. The new prototype features picomolar fluorescence detection sensitivity and a fast temporal resolution of up to 60 fps. To the best of our knowledge, this is the first dual-modal see-through display used in intraoperative fluorescence imaging. We showed the feasibility of using this system to guide SLN mapping with NIR QDs. Furthermore, we demonstrated the potential value of the goggle system and tumor-targeted NIR fluorescence molecular probes in detecting and guiding resection of breast cancer metastases in the liver. Further optimization of the new goggle system holds promise to improve surgical planning, in serving various clinical needs in the future.