Hyperspectral imaging combines the characteristics of computer vision and point spectroscopy by obtaining an image with both spatial and spectral information. Therefore, in combination with microscopy, it can increase material discrimination possibilities with respect to regular microscopy imaging. We explore this increased discrimination potential to assess exposure to particle contamination, since workplace exposure to specific particle materials poses well-known health hazards. In this respect, we are focusing on discriminating more health relevant particles such as silica in the respirable size fraction. For this purpose, a particle sampling protocol has been proposed and hyperspectral imaging in combination with transmission microscopy is used for particle material identification. We use a Snapscan visual near-infrared (VNIR) camera providing high spectral and spatial resolution in the 460-900 nm range, 150 spectral bands and up to 7 Mpixels of spatial resolution and high acquisition speed. The hyperspectral microscopy system has been tested for discrimination of fifteen different particle materials, such as silica, coal, dolomite, barite, or rutile, among others. The combined analysis of spatial and spectral information shows potential to accurately discriminate the 15 tested particle materials so far by means of a random forest classifier. In addition, a band relevance analysis is performed showing that only a few specific bands are needed to provide accurate discrimination of the tested materials. The hyperspectral hardware and method presented could lead to a faster exposure assessment than traditional techniques used for occupational exposure estimation.
KEYWORDS: Deep learning, Tumors, Surgery, Neural networks, Hyperspectral imaging, RGB color model, Tissues, Cameras, Brain, Real time optical diagnostics
Surgery for gliomas (intrinsic brain tumors), especially when low-grade, is challenging due to the infiltrative nature of the lesion. Currently, no real-time, intra-operative, label-free and wide-field tool is available to assist and guide the surgeon to find the relevant demarcations for these tumors. While marker-based methods exist for the high-grade glioma case, there is no convenient solution available for the low-grade case; thus, marker-free optical techniques represent an attractive option. Although RGB imaging is a standard tool in surgical microscopes, it does not contain sufficient information for tissue differentiation. We leverage the richer information from hyperspectral imaging (HSI), acquired with a snapscan camera in the 468 − 787 nm range, coupled to a surgical microscope, to build a deep-learning-based diagnostic tool for cancer resection with potential for intra-operative guidance. However, the main limitation of the HSI snapscan camera is the image acquisition time, limiting its widespread deployment in the operation theater. Here, we investigate the effect of HSI channel reduction and pre-selection to scope the design space for the development of cheaper and faster sensors. Neural networks are used to identify the most important spectral channels for tumor tissue differentiation, optimizing the trade-off between the number of channels and precision to enable real-time intra-surgical application. We evaluate the performance of our method on a clinical dataset that was acquired during surgery on five patients. By demonstrating the possibility to efficiently detect low-grade glioma, these results can lead to better cancer resection demarcations, potentially improving treatment effectiveness and patient outcome.
Label-free tissue identification is the new frontier of image guided surgery. One of the most promising modalities is hyperspectral imaging (HSI). Until now, the use of HSI has, however, been limited due to the challenges of integration into the existing clinical workflow. Research to reduce the implementation effort and simplifying the clinical approval procedure is ongoing, especially for the acquisition of feasibility datasets to evaluate HSI methods for specific clinical applications. Here, we successfully demonstrate how an HSI system can interface with a clinically approved surgical microscope making use of the microscope’s existing optics. We outline the HSI system adaptations, the data pre-processing methods, perform a spectral and functional system level validation and integration into the clinical workflow. Data were acquired using an imec snapscan VNIR 150 camera enabling hyperspectral measurement in 150 channels in the 470-900 nm range, assembled on a ZEISS OPMI Pentero 900 surgical microscope. The spectral range of the camera was adapted to match the intrinsic illumination of the microscope resulting in 104 channels in the range of 470-787 nm. The system’s spectral performance was validated using reflectance wavelength calibration standards. We integrated the HSI system into the clinical workflow of a brain surgery, specifically for resections of low-grade gliomas (LGG). During the study, but out of scope of this paper, the acquired dataset was used to train an AI algorithm to successfully detect LGG in unseen data. Furthermore, dominant spectral channels were identified enabling the future development of a real-time surgical guidance system.
This paper presents the latest advances on Imec snapshot multispectral imagers based on either 3x3, 4x4 and 5x5 mosaic filter patterning on industry ready VIS/NIR and SWIR detectors. The mosaic patterns are implemented by means of high-transmission Fabry-Pérot interferometers processed using thin-film technology. Our snapshot multispectral imagers offer a spatial resolution of 640x480 pixels (SWIR) and 2048x1088 (VIS/NIR) down sampled according to the mosaic pattern to acquire data in nine (3x3), 16 (4x4) of 25 (5x5) spectral bands respectively. To achieve imaging at the native spatial resolution of the sensor, super resolution methods are available post-acquisition. Moreover, our compact USB-3 cameras of 260 gr (SWIR) and 27 gr (VNIR), without lens, reach an acquisition speed of up to 120 multispectral cubes/second and are therefore suitable for dynamic applications, high-speed inspection such as a conveyor belt or UAV inspection. The potential for snapshot cameras in a wide range of applications is showcased in this paper. We first show how applications on industrial quality inspection (chocolate gloss estimation) and precision agriculture (plant disease detection) achieve good discrimination potential in the VNIR range. Specifically, UAV inspection benefits from our compact camera size, low weight, and video capabilities. We then demonstrate the potential for plastic and textile recycling in the SWIR range and the benefit brought by both VNIR and SWIR ranges for people tracking under low visibility conditions. Finally, an application involving the joint use of a microscope and a multispectral camera system is presented for particle contamination exposure assessment. The suitable range is in this case application dependent.
KEYWORDS: Cameras, Imaging systems, Sensors, Unmanned aerial vehicles, Control systems, System integration, Data processing, Visualization, Data acquisition, UAV imaging systems
Multispectral imaging technology analyzes for each pixel a wide spectrum of light and provides more spectral
information compared to traditional RGB images. Most current Unmanned Aerial Vehicles (UAV) camera systems are
limited by the number of spectral bands (≤10 bands) and are usually not fully integrated with the ground controller to
provide a live view of the spectral data.
We have developed a compact multispectral camera system which has two CMV2K 4x4 snapshot mosaic sensors
internally, providing 31 bands in total covering the visible and near-infrared spectral range (460-860nm). It is compatible
with (but not limited to) the DJI M600 and can be easily mounted to the drone. Our system is fully integrated with the
drone, providing stable and consistent communication between the flight controller, the drone/UAV, and our camera
payload. With our camera control application on an Android tablet connected to the flight controller, users can easily
control the camera system with a live view of the data and many useful information including histogram, sensor
temperature, etc. The system acquires images at a maximum framerate of 2x20 fps and saves them on an internal storage
of 1T Byte. The GPS data from the drone is logged with our system automatically. After the flight, data can be easily
transferred to an external hard disk. Then the data can be visualized and processed using our software into single
multispectral cubes and one stitched multispectral cube with a data quality report and a stitching report.
We report a fast and effective lens-free imaging platform for optical diffraction tomography (ODT). Using single wavelength illumination from only 4 angular directions and a lensless inline holographic imaging setup to directly capture the resulting diffraction patterns, our method can reconstruct high quality 3D images of biological samples at micron-scale resolution across a cubic-millimeter-level volume with a compact, scalable and inexpensive system. To achieve this, we developed a compressive tomographic reconstruction algorithm to solve the inverse problem of lens-free ODT by combining Wirtinger derivatives and primal-dual splitting. This fast and inexpensive lens-free tomographic microscopy system provides a promising 3D imaging approach for high throughput biomedical applications.
Lens-free imaging (LFI) has become an important microscopy tool in many life science and industrial applications. Due to the absence of optical lenses (such as objectives) and accompanying lens aberrations (such as chromatic aberrations), the LFI modality is well suited for optical inspection of microscopic objects in a wide spectral range. However, the relatively restricted spectral sensitivity of CMOS imagers, i.e. from visible (~400 nm) up to near-infrared range (~900 nm), limits the wide spectral use of the technique. Many microscopic samples contain valuable information both in the visible and in the short wave infra-red (SWIR), sometimes in addition to visible (VIS) and near-infrared (NIR). With the recent emergence of cost-effective image sensor technologies such as quantum-dot and graphene-based image sensors with high quantum efficiency in SWIR, new lens-free imaging opportunities are emerging for wideband and high throughput microscopy. We demonstrate for the first time an LFI system based on a quantum-dot image sensor, capable of operating in both the visible and short-wave infrared range. The holograms of the samples are obtained through multiple partially coherent illumination sources in both visible and short-wave infrared (ranging from 405 nm to 1550 nm). The captured holograms are reconstructed to obtain images of the sample in focus. We demonstrate an optical resolution of 3.48 micron in a field of view of 9.6 mm2 over the whole spectral range. Our technique mitigates the need for bulky and expensive achromatic imaging optics and offers significant improvements in cost, field-of-view, scalability, and optical resolution to achieve microscopic imaging in both the visible and short-wave infrared spectral range with a simple imaging system. We present in this paper a performance analysis of the system and several potential applications and use cases.
Lens-free holographic microscopy (LHM) is a promising imaging technique for life science and industrial applications, yet system miniaturization and cost reduction without compromising imaging performance remain challenging for field applications in low-resource settings. We demonstrate a cost-effective LHM system without needs for precision optical and mechanical parts (such as lenses, beam-splitters, or kinematic stages) and relies solely on robust optoelectronic hardware and software co-design for high performance imaging. The compact and lightweight form-factor is achieved through integration of light sources, an image sensor and all control electronics with automated calibration and multiwavelength reconstruction algorithms. Amplitude and phase images of a sample can be reconstructed in a few seconds with a micron level optical resolution in a field-of-view of 16.5 mm2. The method offers a portable and scalable solution for microscopic imaging applications.
This paper presents system-level analysis of a sensor capable of simultaneously acquiring both standard absorption based RGB color channels (400-700nm, ~75nm FWHM), as well as an additional NIR channel (central wavelength: ~808 nm, FWHM: ~30nm collimated light). Parallel acquisition of RGB and NIR info on the same CMOS image sensor is enabled by monolithic pixel-level integration of both a NIR pass thin film filter and NIR blocking filters for the RGB channels. This overcomes the need for a standard camera-level NIR blocking filter to remove the NIR leakage present in standard RGB absorption filters from ~700-1000nm. Such a camera-level NIR blocking filter would inhibit the acquisition of the NIR channel on the same sensor. Thin film filters do not operate in isolation. Rather, their performance is influenced by the system context in which they operate. The spectral distribution of light arriving at the photo diode is shaped a.o. by the illumination spectral profile, optical component transmission characteristics and sensor quantum efficiency. For example, knowledge of a low quantum efficiency (QE) of the CMOS image sensor above 800nm may reduce the filter’s blocking requirements and simplify the filter structure. Similarly, knowledge of the incoming light angularity as set by the objective lens’ F/# and exit pupil location may be taken into account during the thin film’s optimization. This paper demonstrates how knowledge of the application context can facilitate filter design and relax design trade-offs and presents experimental results.
This paper presents multispectral active gated imaging in relation to the transportation and security fields. Active gated imaging is based on a fast gated camera and pulsed illuminator, synchronized in the time domain to provide range based images. We have developed a multispectral pattern deposited on a gated CMOS Image Sensor (CIS) with a pulsed Near Infrared VCSEL module. This paper will cover the component-level description of the multispectral gated CIS including the camera and illuminator units. Furthermore, the design considerations and characterization results of the spectral filters are presented together with a newly developed image processing method.
Traditional spectral imaging cameras typically operate as pushbroom cameras by scanning a scene. This approach makes
such cameras well-suited for high spatial and spectral resolution scanning applications, such as remote sensing and
machine vision, but ill-suited for 2D scenes with free movement. This limitation can be overcome by single frame,
multispectral (here called snapshot) acquisition, where an entire three-dimensional multispectral data cube is sensed at
one discrete point in time and multiplexed on a 2D sensor.
Our snapshot multispectral imager is based on optical filters monolithically integrated on CMOS image sensors with
large layout flexibility. Using this flexibility, the filters are positioned on the sensor in a tiled layout, allowing trade-offs
between spatial and spectral resolution. At system-level, the filter layout is complemented by an optical sub-system
which duplicates the scene onto each filter tile. This optical sub-system and the tiled filter layout lead to a simple
mapping of 3D spectral cube data on the sensor, facilitating simple cube assembly. Therefore, the required image
processing consists of simple and highly parallelizable algorithms for reflectance and cube assembly, enabling real-time
acquisition of dynamic 2D scenes at low latencies. Moreover, through the use of monolithically integrated optical filters
the multispectral imager achieves the qualities of compactness, low cost and high acquisition speed, further
differentiating it from other snapshot spectral cameras. Our prototype camera can acquire multispectral image cubes of
256x256 pixels over 32 bands in the spectral range of 600-1000nm at 340 cubes per second for normal illumination
levels.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.