This PDF file contains the front matter associated with SPIE Proceedings Volume 11787, including the Title Page, Copyright information and Table of Contents

The use of deep features coming from pre-trained neural networks for unsupervised anomaly detection purposes has recently gathered momentum in the computer vision field. In particular, industrial inspection applications can take advantage of such features, as demonstrated by the multiple successes of related methods on the MVTec Anomaly Detection (MVTec AD) dataset. These methods make use of neural networks pre-trained on auxiliary classification tasks such as ImageNet. However, to our knowledge, no comparative study of robustness to the low data regimes between these approaches has been conducted yet. For quality inspection applications, the handling of limited sample sizes may be crucial as large quantities of images are not available for small series. In this work, we aim to compare three approaches based on deep pre-trained features when varying the quantity of available data in MVTec AD: KNN, Mahalanobis, and PaDiM. We show that although these methods are mostly robust to small sample sizes, they still can benefit greatly from using data augmentation in the original image space, which allows to deal with very small production runs.

Besides its importance for greenhouse emission reduction, the remote detection, localization and quantification of gas leaks in industrial facilities remains a challenging problem in industry and research. In that sense, the development of new data processing techniques that allow deriving new and/or more accurate information about the gas leaks from made measurements has gained more attention in the recent years. This becomes apparent from the increased use of optical gas imaging (OGI) cameras (specialised mid-wave infrared cameras e.g. for methane and carbon dioxide) along with image processing and computer vision techniques, to tackle these challenges. In this work, deep-learning-based optical flow methods are evaluated for determining gas velocities from gas images of an OGI camera. For this, a dataset of simulated and real gas images under controlled and real conditions is used for supervised training and validation of two different state of the art CNNs for optical flow computation: FlowNetC, FlowNet2 and PWC-Net. Classical optical flow methods based on variational methods are also considered and the differences in performance and accuracy between classical and deep-learning-based methods are shown. In addition, FlowNet2 is further improved for working with gas images by fine tuning the network weights. This approach has demonstrated to make FlowNet2 more reliable and less sensitive to image noise and jitter in the experiments. For further validation, a set of real gas images acquired in a wind channel and one from a biogas plant with reference mean gas velocities from a 3D anemometer are being used. The results show that the fine-tuned version of FlowNet2 (FNet2-G) allow computing larger optical flow magnitudes than classical optical flow methods while being less sensitive to image noise under field conditions. The obtained results also show the potential of deep-learning-based approaches for image processing tasks such as gas segmentation, disparity computation and scene flow in stereo gas images.

Automated fibre layup techniques are commonly used composite manufacturing processes in the aviation sector and require a manual visual inspection. Neural Network classification of defects has the potential to automate this visual inspection, however, the machine decision-making processes are hard to verify. Thus, we present an approach for visualising Convolutional Neural Network (CNN) based classifications of manufacturing defects and quantifying its robustness suitably. Our investigations have shown that especially Smoothed Integrated Gradients and DeepSHAP are particularly well suited for the visualisation of CNN classifications. The Smoothed Integrated Gradients technique also reveals advantages in robustness when evaluating degraded input images.

The stability, durability, and safety of concrete bridges are not only endangered by visible damages on the surface such as cracks or spalling, but also internal damages such as cavities, delamination, or material changes due to carbonation or moisture penetration. In Germany, the inspection is regulated by the norm DIN 1076. Accordingly, the inspectors with an experienced hearing still interpret the sound of a hammer on the entire underside of the bridge. Therefore, to inspect the underside of a bridge with its underlying infrastructures like traffic, water, or railroads and its usually complex and difficult to reach structures, including girders, a fast and reliable remote sensing approach is required to increase both efficiency and level of automation. This paper summarizes recent work using both passive and active thermography for bridge inspection. We compare three different sensor systems including the Parrot Anafi Thermal, the FLIR Vue Pro R, and the FLIR SC640 which could be used in a UAV, UGV, or human based inspection. To test and verify our concepts, we apply passive thermography on real concrete bridges in Freiburg (Germany) and derive 2D and 3D data using a photogrammetric approach. We analyse spatial and thermal resolution dependent on camera selection, working distance, weather conditions, and the bridges' surface properties. For damage detection, we use high-pass or lowpass filtering dependent on the ambient temperature gradient. Moreover, we discuss further damage detection methods that could be integrated into the proposed processing workflow.

Algorithms that interpret images to locate surface defects, such as cracks, play a key role in an automated inspection system. That is the reason the success of convolutional neural networks (CNNs) in image object detection persuaded researchers to apply deep CNNs for visual surface crack detection. Among various deep learning architectures, encoder decoder architectures with fully convolutional networks (FCNs) are powerful tools for automatically segmenting inspection images and detecting crack maps. In this study the U-Net architecture, as a particular FCN, is trained using the available concrete crack datasets. The trained network is then employed to detect crack maps in a sequence of images taken from a concrete beam-column specimen under a cyclic load test. To enhance performance of the crack segmentation, instead of treating each image in the sequence independently, the detection results of the next stages of the experiment are used to determine the crack map at the current stage. By leveraging the fact that cracks propagate sequentially, a data fusion technique is proposed that updates crack maps by considering the outcome of the next steps. To realize this method, reference points on images are utilized to estimate the deformation of the structural members. The deformation information is then used to project the previously detected crack maps onto the current image. This makes it possible to aggregate current and future detections and achieve higher accuracy. The framework laid out in this study provides tools to filter out false positives and recover missed detections.

Phase-measuring deflectometry (PMD) is an optical inspection technique for full-field topography measurements of reflective sample surfaces. The measurement principle relies on the analysis of specific patterns, reflected at the sample surface. Evaluation algorithms often model the respective pattern screen as a planar light source. However, the 3200 pattern screen in our inspection setup exhibits a central bulge of its surface of about 2–3mm. This paper presents a simulation framework for PMD to evaluate the effects of a deformed screen surface. The idea is to simulate image data acquired with screen surface deformations and to examine the effects on the PMD evaluation results. The simulated setup consists of a 3200 pattern screen with an adjustable central bulge height of 0–3mm and two cameras with a field of view (FOV) of approximately 225mm by 172mm on the sample surface. A first experiment examines the reconstruction errors for a planar sample surface if the reconstruction algorithm uses perfect calibration data (i.e. the same parameters used for the simulated image acquisition). The reconstructed surfaces exhibit a tilt with a maximum height difference of 174 μm across the FOV. A second experiment repeats the reconstruction process of the same sample surface, using camera parameters determined in a simulated calibration process. The resulting surfaces possess irregular, wave-like errors with amplitudes of up to 9 μm in the FOV. The presented simulation results reveal the accuracy limits if a deformation model of the pattern screen is not explicitly included in the reconstruction process.

Identification and documentation of the state of road infrastructure are rapidly gaining in importance due to increased traffic. This requires regular inspection in order to optimize maintenance efforts. Additionally, construction projects are getting more complex and demand digital planning and building processes. The data, which forms the basis for these processes, must be rapidly acquired and it must be digital. In this paper, a measurement vehicle is presented that efficiently collects all desired data. The vehicle is based on a modular system platform from the Fraunhofer-Institute for Physical Measurement Techniques IPM. It enables measurements for road analysis while driving with up to 80 km/h. Four lasers quantify the longitudinal evenness of the road with submillimetre precision and kHz temporal resolution. A pavement profile scanner built by Fraunhofer IPM is used to measure the transverse evenness of the road with millimetre precision and a 2 MHz sampling rate. Two cameras and an illumination system provide continuous high-resolution and high-contrast images of the road surface for the quantification of the road condition. Furthermore, additional instruments are mounted for the surveying of road surroundings. A clearance profile scanner built by Fraunhofer IPM scans a 345° area perpendicular to the driving direction and delivers 3D data with millimetre precision. Four cameras acquire highresolution images of the surrounding in all directions. In combination with a sophisticated post-processing pipeline, which extracts semantically segmented information from the images, this yields a digital, semantically interpreted representation of the mapped urban area.

The measurement of breathing biomechanics, such as tidal volume, can be used to assess both the breathing performance and the respiratory health of individuals. State-of-the-art methods like spirometry or body plethysmography require a mouthpiece or facemask., which can be uncomfortable to the test person. As an alternative, we propose to use the change of the geometric shape of the subject’s torso while breathing. By acquiring 3D point clouds of the person with a real-time near-infrared (NIR) 3D scanner, we measure those changes in a comfortable, irritation-free, and contact-free manner. Accordingly, two continuously measuring structured light 3D sensors, using a GOBO-based aperiodic sinusoidal pattern projector at a wavelength of 850 nm, simultaneously capture the upper front and side torso of the subject at a frame rate of 200 Hz. Both 3D scanners are calibrated and operated in a sensor network fashion, yielding a unified data stream within a global coordinate system. This results in increased coverage and reduced occlusion of the patient’s body shape, enabling robust measurements even in the presence of loose clothing and varying body figure. We collected data from 16 healthy participants in an upright sitting position, wearing everyday clothing during the measurements. For reference, we simultaneously recorded spirometry readings. An algorithm (“OpTidal”) tracks the volume of the subject’s torso from the 3D data. Comparison whith the reference data shows high correlation and low mean error for the absolute tidal volume readings. As such, our method is a viable, safe, and accurate alternative to spirometry and plethysmography.

For the automated optical inspection of manufactured components with complex geometries or highly reflective surfaces, a suitable selection of measurement poses and the associated planning of the measurement trajectory is crucial. This is especially important for active triangulation measurement methods like fringe projection. Due to complex measurement object geometries or poor alignment of the measuring system the influence of multiple reflections can potentially lead to incorrect or incomplete 3-D reconstruction of the specimen surface. This paper introduces a simulative GPU-based inverse ray tracing approach to identify low-reflection measurement poses for active optical measurement systems. Starting from the virtual camera origin, rays are emitted from each camera pixel and the reflection at the measurement objects surface is calculated using the Torrence- Sparrow BRDF. With an additional approach based on Whitted raytracing, the influence of multiple reflections and the reflection depth on the rendered camera image is taken into account. By calculating the summed reflection depth of each rendered measurement sequence, a height map of the reflection frequency distribution is created. By sampling a predefined surface point on the path of a limited sphere, the comparability of possible measurement poses is achieved. Thus, local minima can be identified and the poses with the lowest reflection influence can be selected to perform a suitable trajectory planning. This a priori knowledge can also be transferred into application and used for the estimation of image areas, which captured multiple reflections. Thus for these areas specific masks are generated and can be applied in real measurements to reconstruct multiple reflection free surfaces.

Studying radiative properties of molecules, for instance to identify species during fast phenomena, is not always simple because fast detection systems are required. With the experimental setup presented, high-resolution spectra in the visible (VIS) and the near infrared (NIR) range can be recorded at high frame rates for gaseous deflagrations. For such phenomenon, the flame front spreads in a few tens of meters per second. In the setup, the deflagrations are performed in a stainless steel cylindrical combustion chamber, which includes a sapphire window and incorporates a high-speed piezoelectric sensor to measure pressure variations. The emitted radiation is focused into the slit of a monochromator, and at its exit slit a camera records spectra in real time.

Minimization of the environmental impact of the incineration process and to produce energy efficiently are the most important considerations in obtaining efficient operation of waste-to-energy (WtE) plants. WtE operation can obtain significant improvements by predicting combustion properties of municipal solid waste (MSW) prior to incineration. Combustion properties of MSW can be assessed by estimating the weighted waste fractions such as paper and cardboard, plastic or inert and fines. Waste materials and fractions can be recognized using imaging techniques and image classification methods based on deep convolution neural networks (CNN). We have tested a new sensor system for image classification based on using multispectral (MS) images and deep CNN pretrained on the ImageNet database to recognize MSW categories. MS camera was used for sampling images above the walking floor of a WtE plant (StatKraft Varme Tiller, Trondheim). The waste load was automatically registered as industrial or household waste at the time of delivery. The MS images from 49 waste loads were used to perform transfer-learning on efficientNetB0 model weighted with ImageNet-NoisyStudent parameters. Using the predefined classes, a test and training set were generated from the 49 waste loads delivered between June and September (2020). The training set consisted of 35 waste loads while the remaining 14 waste loads were used as test. The weights for the image feature extraction was constant during training while the fully connected layer (top-layer) was updated for each epoch. The model performance on the test set was assessed by making predictions on the household or industrial waste images. With a fixed threshold value at 0.5, the model showed 85% accuracy, 92% precision, 89% recall and 90% F-measure for industrial class, while for household class the model showed 80% accuracy, 94% precision, 81% recall and 87% F-measure. For all threshold values, the area under curve estimated from the receiver operating characteristic plot showed that the model has 87% confidence in distinguishing household waste images from industrial waste images and 90% confidence in distinguishing industrial waste images from household waste images.

Vision Ray Calibration provides a description of imaging properties of cameras by identification of an independent vision ray for each sensor pixel. Due to this approach, no model parameters of any cameras are determined in the non-linear optimization procedure of the calibration. Therefore, a setup of multiple cameras can be considered as one imaging system. This enables simultaneous and holistic calibration of an arbitrary number of cameras. Vision Ray Calibration utilizes Liquid Crystal Displays as calibration targets since these can provide the required continuous spatial coding of their surfaces by means of Phase Shifting Technique. However, displays employed as calibration targets exhibit some unfavorable properties such as flatness deviations of the surface. It is known that extending the Vision Ray Calibration by a polynomial parameterization of the display surface increases the calibration accuracy. This work investigates the influence of the order of polynomial terms employed for display surface parameterization. The stereoscopic Fringe Projection enables the evaluation of the calibration accuracy since Vision Ray Calibration of a stereo camera setup provides full system calibration for this technique. Our results confirm the significant improvement of calibration accuracy due to parameterization of flatness deviations of the display. Best calibration results were obtained by limiting the maximum order of polynomial terms to three. Our results indicate that terms of higher orders do not contribute true surface shape features.

Optical metrology techniques such as fringe projection technology and phase-measuring deflectometry, which use cameras as image sensors for 3D coordinate measurement and inspection, are becoming increasingly demanding. The necessary precise and at the same time robust calibration of image sensors of these systems is offered by so called vision ray calibration. This generic approach is a model-free description of a vision ray and are determined for each imaging sensor pixel. Numerous images of the sinusoidal fringe images displayed on the active calibration target (LCD) are captured in different poses. This generates a large amount of data and consequently leads to time-consuming evaluation for the camera calibration. In this paper, we propose a new optimization method that uses “vision threads" instead of vision rays to enhance the computation efficiency associated with conventional vision ray calibration. We present MATLAB simulation results in to validate our novel approach.

In this paper, we propose a preprocessing method of exploiting noise and blur for effective noise elimination in data. At present, there are many kinds of research to improve the performance of object classification, detection, and image segmentation based on deep learning. For instance, adding noise to data, multiple in-depth convolution layers, and data augmentation have been studied in many ways. An in-depth convolution network results in long processing time and data augmentation gives a burden to memory usage. However, adding noise and blur data preprocessing method gives less burden to hardware, which helps improve algorithm performance. The proposed method is applied to TFT-LCD (Thinfilm Transistor Liquid Crystal Display) PAD defect detection for improved performance. To verify the accuracy and repeatability, 691 actual defect images are used in experiments. These images are composed of complex patterns and defects in the images having barely 2 pixels with little intensity difference. To confirm which filters are better, Gaussian blur, Salt & Pepper noise, and Gaussian noise filters are used for comparison. According to the result, the experiments with Salt & Pepper and Gaussian noise detect all defects. However, the repeatability of the Gaussian noise filter seems better than that with Salt & pepper. Furthermore, applying noise and blur to train data shows more than twice higher detection accuracy than those without such applications. We verified that using Gaussian noise and blur indicates excellent accuracy and repeatability when inspecting the TFT-LCD PAD area in AOI machines.

We propose a novel multispectral imaging technique employing complementary notch filters instead of bandpass filters which are conventionally used in filter-based multispectral cameras. Therefore, only little power of the incoming photon signal is lost and thus the SNR of the multispectral data can be significantly improved. To validate the proposed approach, simulations of conventional bandpass filters as well as complementary notch filters are presented. To compare the resulting SNRs, the EMVA 1288 standard is adopted in such a way that it is applicable to notch filter-based multispectral cameras. It is found that the SNR can be significantly improved by using complementary filters instead of the conventional bandpass filters, especially at high spectral resolution.

Scanning Electron Microscopes (SEM) and Dual Beam Focused Ion Beam Microscopes (FIB-SEM) are essential tools used in the semiconductor industry and in relation to this work, for wafer inspection in the production of hard drives at Seagate. These microscopes provide essential metrology during the build and help determine process bias and control. However, these microscopes will naturally drift out of focus over time, and if not immediately detected the consequences of this include: incorrect measurements, scrap, wasted resources, tool down time and ultimately delays in production. This paper presents an automated solution that uses deep learning to remove anomalous images and determine the degree of blurriness for SEM and FIB-SEM images. Since its first deployment, the first of its kind at Seagate, it has replaced the need for manual inspection on the covered processes and mitigated delays in production, realizing return on investment in the order of millions of US dollars annually in both cost savings and cost avoidance. The proposed solution can be broken into two deep learning steps. First, we train a deep convolutional neural network, a RetinaNet object detector, to detect and locate a Region Of Interest (ROI) containing the main feature of the image. For the second step, we train another deep convolutional neural network using the ROI, to determine the sharpness of the image. The second model identifies focus level based on a training dataset consisting of synthetically degraded infocus images, based on work by Google Research, achieving up to 99.3% test set accuracy.

In actual industrial sites, the ability of the deep learning model to detect defects at a high speed and reducing the time required to train the model is also a very important issue. In this paper, we propose a fast and accurate deep learning model and training method that can be applied to inspect the TFT-LCD(The Film Transistor - Liquid Crystal Display) PAD area image. The deep learning model we propose is a lightweight model based on U-net. By training only about 250,000 parameters, it was possible to confirm excellent performance in defect segmentation. In addition, a study on train data was also conducted so that the model can learn more effectively. We studied a method of training both normal images (images without defects) and abnormal images (images with defects), and it was confirmed that this performance showed better performance than when only data with defects were learned. It was shown that the method of learning both normal and abnormal results in a 50% or more reduction in the incidence of false judgment images than the method of learning only simple abnormal data.

Recent advances in infrared sensor technologies made improvements in spatial resolution and frame rates. How- ever, these progresses have been done at the cost of other performances, such as the conversion of charge to voltage. This is especially problematic for fast emission spectroscopy, where an infrared camera measures the radiation that has been previously dispersed by a grating. In this situation, the measured radiation level is low even at high temperature, because of the spectral width (here, a 40 nm spectrum is dispersed over 488 pixels). Because of the size reduction of pixels, each one collects less energy. Calibration becomes challenging because it is carried out in conditions where the response of the detector is not linear. The work presented suggests the correction of the nonlinear response of an InSb sensor of a FLIR camera. Measurements with a former camera are used to correct the FLIR sensor's response with a linear and a logarithmic function. An application is presented with the temperature determination of a H₂, O₂, N₂, CO₂ and Al particles deflagration by spectroscopy.

As a popular topic in automation, fabric defect detection is a necessary and essential step of quality control in the textile manufacturing industry. The main challenge for automatically detecting fabric damage, in most cases, is the complex structure of the textile. This article presents a two-stage approach, combining novel and traditional algorithms to enhance image enhancement and defect detection. The first stage is a new combined local and global transform domain-based image enhancement algorithm using block-based alpha-rooting. In the second stage, we construct a neural network based on the modern architecture to detect fabric damage accurately. This solution allows localizing defects with higher accuracy than traditional methods of machine learning and modern methods of deep learning. All experiments were carried out using a public database with examples of damage to the TILDA fabric dataset.

Currently, VR and AR headsets are becoming widespread. Such technologies make it possible to significantly expand the possibilities of using services for remote maintenance and repair complex technical systems by highly qualified specialists. During the COVID-19 pandemic, the need for service representatives of manufacturing enterprises from foreign countries using such funds has increased significantly. Restrictive measures on movement between states limit the possibility of interaction between manufacturers' representatives (without losing the quality of work performed), so AR technology has become virtually uncontested. The article describes augmented reality glasses based on a mobile phone (system "DAR"), which combines the functions of VR and AR technologies and a low cost of the final product. The proposed solution combines a helmet with a smartphone, which is used to transmit information about the surrounding space and connect the augmented reality elements built on this image. Information about the surrounding space comes to the smartphone screen from stereo cameras equipped with autofocus. It allows the user to transmit the picture with a minimum delay and high quality. The low cost of the final device is ensured by stereo cameras, a module with sensors, and housing for attaching to the user's head. Processing of information about movement, sound transmission, and superposition of augmented reality elements is done using a smartphone. This solution makes it possible to expand AR technology scope for remote maintenance of complex technical systems by highly qualified specialists at remote sites since using a smartphone, and a DAR headset will be sufficient. The device's proposed technical solutions allow providing a high IP class, which is necessary for industrial use.

Carbon fiber reinforced plastic (CFRPs) is a composite material that has substituted metal alloys in many industrial fields. Non-destructive testing techniques are interesting inspection methods for the integrity assessment of composite materials and Optical Lock-in Thermography (OLT) is a particularly convenient alternative to inspection because setting different loading frequencies will result in different scanning depths. Regarding the segmentation task, the problem to be solved is to develop a tool that can correctly identify defective areas with several geometric shapes and features even if there is noise, and without using any manual input or creating artifacts in the image. This work describes the application of Unet and Mask R-CNN in the segmentation of defects in OLT phase images of CFRP plates. The output images from the evaluation were compared using the IoU and ANOVA test as a significance evaluator. The results show that Mask R-CNN performed better-segmenting OLT images.