KEYWORDS: Video, 3D image processing, Cameras, Feature extraction, 3D metrology, RGB color model, Near infrared, Oxygen, Convolutional neural networks, 3D modeling
SignificanceMonitoring oxygen saturation (SpO2) is important in healthcare, especially for diagnosing and managing pulmonary diseases. Non-contact approaches broaden the potential applications of SpO2 measurement by better hygiene, comfort, and capability for long-term monitoring. However, existing studies often encounter challenges such as lower signal-to-noise ratios and stringent environmental conditions.AimWe aim to develop and validate a contactless SpO2 measurement approach using 3D convolutional neural networks (3D CNN) and 3D visible-near-infrared (VIS-NIR) multimodal imaging, to offer a convenient, accurate, and robust alternative for SpO2 monitoring.ApproachWe propose an approach that utilizes a 3D VIS-NIR multimodal camera system to capture facial videos, in which SpO2 is estimated through 3D CNN by simultaneously extracting spatial and temporal features. Our approach includes registration of multimodal images, tracking of the 3D region of interest, spatial and temporal preprocessing, and 3D CNN-based feature extraction and SpO2 regression.ResultsIn a breath-holding experiment involving 23 healthy participants, we obtained multimodal video data with reference SpO2 values ranging from 80% to 99% measured by pulse oximeter on the fingertip. The approach achieved a mean absolute error (MAE) of 2.31% and a Pearson correlation coefficient of 0.64 in the experiment, demonstrating good agreement with traditional pulse oximetry. The discrepancy of estimated SpO2 values was within 3% of the reference SpO2 for ∼80% of all 1-s time points. Besides, in clinical trials involving patients with sleep apnea syndrome, our approach demonstrated robust performance, with an MAE of less than 2% in SpO2 estimations compared to gold-standard polysomnography.ConclusionsThe proposed approach offers a promising alternative for non-contact oxygen saturation measurement with good sensitivity to desaturation, showing potential for applications in clinical settings.
Non-contact methods can expand the application scenarios of blood oxygen measurement with better hygiene and comfort, but the traditional non-contact methods are usually less accurate. In this study a novel non-contact approach for measuring peripheral oxygen saturation (SpO2) using deep learning and near-infrared multispectral videos is proposed. After a series of data processing including shading correction, global detrending and spectral channel normalization to reduce the influences from illumination non-uniformity, ambient light, and skin tone, the preprocessed video data are split into half-second clips (30 frames) as input of the 3D convolutional residual network. In the experiment, multispectral videos in 25 channels of hand palms from 7 participants were captured. The experimental results show that the proposed approach can accurately estimate SpO2 from near-infrared multispectral videos, which demonstrates the agreement with commercial pulse oximeter. The study also evaluated the performance of the approach with different combinations of near-infrared channels.
The contactless estimation of vital signs based on conventional color cameras and ambient light can be affected by motions of patients and changes in ambient light. In this work, a multimodal 3D imaging system with an irritation-free controlled illumination was developed in order to mitigate these both problems. In the developed system the real-time high-precision 3D imaging is combined with VIS-NIR multispectral imaging and thermal imaging. Based on 3D data and color images, an approach was proposed for the efficient compensation of head motions, and novel approaches based on 3D regions of interest were developed for the estimation of heart rate, oxygen saturation, respiration rate, and body temperature from NIR multispectral video data and thermal video data. A proof-of-concept for the developed imaging system and algorithms can be delivered with first experimental results.
In this work, we developed a multimodal imaging system for real-time applications by integrating 2D image sensors in different spectral ranges as well as a polarization camera into a high-speed optical 3D sensor. For the generation of the multimodal image data, a pixel-level alignment of 2D images in different modalities to 3D data is realized by applying projection matrices to each point in the 3D point cloud. For the calculation of projection matrices for each 2D image sensor, a calibration procedure is proposed for the extrinsic calibration of arbitrarily positioned image sensors. The final imaging system delivers multimodal video data with one mega-pixel resolution at a frame rate of 30 Hz. As application examples, we demonstrate the estimation of vital signs and the detection of human body parts with this imaging system.
With the advent of industry 4.0, the introduction of smart manufacturing and integrated production systems, the interest in 3D image-based supervision methods is growing. The aim of this work is to develop a scalable multi-camera-system suitable for the acquisition of a dense point cloud representing the interior volume of a production machine for general supervision tasks as well as for navigation purposes without a priori information regarding the composition of processing stations. Therefore, multiple low-cost industrial cameras are mounted on the machine housing observing the interior volume. In order to obtain a dense point cloud, this paper reviews aspects of metric stereo calibration and 3D reconstruction with attention being focused on target-based calibration methods and block matching algorithms.
This paper presents the design and simulation of a single-shot optical 3D sensor based on multispectral pattern projection and a stereo-vision setup of two multispectral snapshot cameras. The performances of various combinations of available multispectral cameras, spatial light patterns, and 3D reconstruction algorithms as well as the geometric arrangements of the stereo-vision camera setup are simulated and analyzed. This simulation-based investigation delivers two optimized combinations of sensor components in terms of hardware and algorithm as orientations for practical sensor development in future, and an appropriate arrangement of the stereo-vision setup is determined. Moreover, the influences of sensor noise on 3D reconstruction are also estimated.
Currently, Deep Learning (DL) shows us powerful capabilities for image processing. But it cannot output the exact photometric process parameters and shows non-interpretable results. Considering such limitations, this paper presents a robot vision system based on Convolutional Neural Networks (CNN) and Monte Carlo algorithms. As an example to discuss about how to apply DL in industry. In the approach, CNN is used for preprocessing and offline tasks. Then the 6- DoF object position are estimated using a particle filter approach. Experiments will show that our approach is efficient and accurate. In future it could show potential solutions for human-machine collaboration systems.
The 3D multispectral imaging system proposed in this contribution realizes simultaneous detection of contaminants and 3D localization. The imaging system is composed of a digital pattern projector, a stereo-vision setup of two multispectral filter wheel cameras, and an external light source. A calibration procedure is developed to estimate simultaneously stereo camera parameters and light source parameters. For an acceleration of image acquisition, the entire spectral range is split into two parts which are captured from two different camera views and merged using structured light. The usefulness of the proposed system is demonstrated with an example of cutting oil detection. For this the fluorescence effect is utilized, and specular reflections are filtered out based on the estimation of illumination geometry. Experimental results show that the surface areas with cutting oil could be reliably distinguished from workpieces using the proposed algorithms.
This paper presents the latest developments on filter-wheel based multispectral imaging systems as well as their extension to making 3D images. The system, capable of producing high spatial resolution images on a spectrum spanning from 400nm to 1050nm (in 12 steps of 50nm (configurable) with 50nm or less bandwidth) can be used, without hardware change. To produce 3D image stacks where the height resolution is given by the numerical aperture of the optics used and the reproducibility of the image plane moving motor is also possible. This paper introduces the reader to spectral imaging and to 3D measurement techniques. The main parameters and relevant publications of/about the industrial monocular multispectral 3D-Imager are then presented. Correction of chromatic aberration on filter wheel system, a key idea for 3D image reconstruction, is revised. 3D imaging capabilities of the system as well as proper calibration are introduced. Selected applications and algorithms are presented towards to the end of the paper.
Usually, a large number of patterns are needed in the computational ghost imaging (CGI). In this work, the possibilities to reduce the pattern number by integrating compressive sensing (CS) algorithms into the CGI process are systematically investigated. Based on the different combinations of sampling patterns and image priors for the L1-norm regularization, different CS-based CGI approaches are proposed and implemented with the iterative shrinkage thresholding algorithm. These CS-CGI approaches are evaluated with various test scenes. According to the quality of the reconstructed images and the robustness to measurement noise, a comparison between these approaches is drawn for different sampling ratios, noise levels, and image sizes.
This paper presents an approach for single-shot 3D shape reconstruction using a multi-wavelength array projector and a stereo-vision setup of two multispectral snapshot cameras. Thus, a sequence of six to eight aperiodic fringe patterns can be simultaneously projected at different wavelengths by the developed array projector and captured by the multispectral snapshot cameras. For the calculation of 3D point clouds, a computational procedure for pattern extraction from single multispectral images, denoising of multispectral image data, and stereo matching is developed. In addition, a proof-of-concept is provided with experimental measurement results, showing the validity and potential of the proposed approach.
Mosaic filter-on-chip CMOS sensors enable the parallel acquisition of spatial and spectral information. These mosaic sensors are characterized by spectral filters which are applied directly on the sensor pixel in a matrix which is multiplied in the x- and y-direction over the entire sensor surface. Current mosaic sensors for the visible wavelength area using 9 or 16 different spectral filters in 3 × 3 or 4 × 4 matrices. Methods for the reconstruction of spectral reflectance from multispectral resolving sensors have been developed. It is known that the spectral reflectance of natural objects can be approximated with a limited number of spectral base functions. Therefore, continuous spectral distributions can be reconstructed from multispectral data of a limited number of channels. This paper shows how continuous spectral distributions can be reconstructed using spectral reconstruction methods like Moore-Penrose pseudo-inverse, Wiener estimation, Polynomial reconstruction and Reverse principal component analysis. These methods will be evaluated with monolithic mosaic sensors. The Goodness of Fit Coefficient and the CIE color difference are used to evaluate the reconstruction results. The reconstruction methods and the spectral base functions applied for the mosaic sensors are juxtaposed and practical conclusions are drawn for their application.
We present an approach for single-frame three-dimensional (3-D) imaging using multiwavelength array projection and a stereo vision setup of two multispectral snapshot cameras. Thus a sequence of aperiodic fringe patterns at different wavelengths can be projected and detected simultaneously. For the 3-D reconstruction, a computational procedure for pattern extraction from multispectral images, denoising of multispectral image data, and stereo matching is developed. In addition, a proof-of-concept is provided with experimental measurement results, showing the validity and potential of the proposed approach.
Recently, a number of hyperspectral cameras with novel image sensors that are equipped with a monolithic pixel-by-pixel filter array have been introduced onto the market. In this contribution, we describe the design and implementation of a structured light 3D sensor consisting of two such snapshot mosaic sensors, each with 25 different spectral bands between 600 and 975 nm, and a broadband GOBO projector that images varying, aperiodic sinusoidal patterns into the measurement volume. We characterize the system with regard to accuracy, and we present measurements of different objects illustrating the benefits of hyperspectral 3D sensors.
3D - Inline - Process - Control is getting more attention in any fields of manufacturing processes to increase productivity
and quality. Sensor systems are necessary to capture the currently process status and are basement for Inline-Process-
Control. The presented work is a possibility to get inline information’s about the additive manufacturing process Fused
Filament Fabrication. The requirement is the ability to manipulate the machine code to get free field of view to the
topside of the object after every manufactured layer. The adaptable platform layout makes possible to create different
approaches for inline process control. One approach is the single camera layout from bird view to get 2,5D information’s
about the manufactured object and the other one is the active stereoscopic camera layout with pattern projection. Both
approaches are showing a possibility to get information’s of the manufactured object in process. Additional this cases
allow a view inside the manufactured object and defects can be located. Deviations in the manufacturing process can be
corrected and relevant parameters can be adapted during slicing process to increase the manufacturing quality.
Inline three-dimensional measurements are a growing part of optical inspection. Considering increasing production capacities and economic aspects, dynamic measurements under motion are inescapable. Using a sequence of different pattern, like it is generally done in fringe projection systems, relative movements of the measurement object with respect to the 3d sensor between the images of one pattern sequence have to be compensated.
Based on the application of fully automated optical inspection of circuit boards at an assembly line, the knowledge of the relative speed of movement between the measurement object and the 3d sensor system should be used inside the algorithms of motion compensation. Optimally, this relative speed is constant over the whole measurement process and consists of only one motion direction to avoid sensor vibrations. The quantified evaluation of this two assumptions and the error impact on the 3d accuracy are content of the research project described by this paper.
For our experiments we use a glass etalon with non-transparent circles and transmitted light. Focused on the circle borders, this is one of the most reliable methods to determine subpixel positions using a couple of searching rays. The intersection point of all rays characterize the center of each circle. Based on these circle centers determined with a precision of approximately 1=50 pixel, the motion vector between two images could be calculated and compared with the input motion vector. Overall, the results are used to optimize the weight distribution of the 3d sensor head and reduce non-uniformly vibrations. Finally, there exists a dynamic 3d measurement system with an error of motion vectors about 4 micrometer. Based on this outcome, simulations result in a 3d standard deviation at planar object regions of 6 micrometers. The same system yields a 3d standard deviation of 9 µm without the optimization of weight distribution.
The requirement for a non-transparent Lambertian like surface in optical 3D measurements with fringe pattern projection cannot be satisfied at translucent objects. The translucency causes artifacts and systematic errors in the pattern decoding, which could lead to measurement errors and a decrease of measurement stability. In this work, the influence of light wavelength on 3D measurements was investigated at a stereoscopic system consisting of two filter wheel cameras with narrowband bandpass filters and a projector with a wide-band light source. The experimental results show a significant wavelength dependency of the systematic measurement deviation and the measurement stability.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.