JEI thanks the reviewers who served the journal in 2023.

Polarization information of the light can provide rich cues for computer vision and scene understanding tasks, such as the type of material, pose, and shape of the objects. With the advent of new and cheap polarimetric sensors, this imaging modality is becoming accessible to a wider public for solving problems, such as pose estimation, 3D reconstruction, underwater navigation, and depth estimation. However, we observe several limitations regarding the usage of this sensorial modality, as well as a lack of standards and publicly available tools to analyze polarization images. Furthermore, although polarization camera manufacturers usually provide acquisition tools to interface with their cameras, they rarely include processing algorithms that make use of the polarization information. In this work, we review recent advances in applications that involve polarization imaging, including a comprehensive survey of recent advances on polarization for vision and robotics perception tasks. We also introduce a complete software toolkit that provides common standards to communicate with and process information from most of the existing micro-grid polarization cameras on the market. The toolkit also implements several image processing algorithms for this modality, and it is publicly available on GitHub.

Guest Editors Shi Dong, Stelvio Cimato, Joarder Kamruzzaman, and Yongsheng Hao introduce the Special Section on Real-Time Image Steganography and Steganalysis.

Personalized news suggestions are an important technology to enhance people’s online news reading experiences. How to better understand users and news representation is a major issue in news recommendation. The majority of cutting-edge news recommendation techniques mostly neglect the link between title and content, explicitly and implicitly. They neglect to take into account the effects of many prospective news preferences on people’s behavior when they click on various news items. We first build a user-news interaction graph and then present the weight learning and preference decomposition (WLPD) news recommendation model for graph neural networks, which is based on WLPD. This model not only takes into account the impact of the relationship between news titles and content, explicit and implicit, on the likelihood that users will click on the news, but also takes into account the various potential preferences between users and news interaction. Finally, using actual news databases, we run a number of experiments. We discover that our model significantly improved in terms of accuracy and performance compared with other cutting-edge news recommendation techniques.

Due to the availability of various photo editing tools, intruders can tamper with an image very easily. So, various watermarking and tamper detection approaches have been proposed by researchers. Basically, tamper detection techniques focus on embedding the watermark, extracting the water mark, and identifying the tampered regions. But it is very important that the tampered pixels should also be corrected. We bring forward an image watermarking technique for tamper detection and correction using remainder value differencing (RVD) and extended Hamming code (EHC). It operates on a pixel group of size 2 × 2. Watermark bits (WBs) are generated from four most significant bits of the pixels in a pixel group by EHC and concealed in four lower bit planes by the principle of RVD. The WBs are extracted at the receiver along with the identification of tampered pixels. The tampered pixels are corrected by the developed correction logic. As the principle of RVD is used, precautions are taken to avoid the fall-off boundary problem. The efficacy of this technique is accessed through various quality metrics. It is noted that it performs better than the existing techniques. The recorded peak signal-to-noise ratio value is 45.49 dB with structural similarity value 0.9889. The tampered pixels are identified and corrected.

We live in an interconnected world, and digital data transmissions have reached significant importance as they are the most used means of communication. Connected objects are widely used in our digital life, yet they may pose vulnerabilities and risks such as hacking or personal data theft due to their often inadequate security. In response to the size and capability restrictions of many existing embedded devices, lightweight cryptography has been considered. This article describes a hardware implementation of the lightweight cryptographic method [very high speed integrated circuit hardware description language (VHDL) and field-programmable gate arrays (FPGAs)]. This work presents a hardware solution for securing digital information, more specifically images, and to ensure a high level of security and favorable performance, a robust algorithm with respect to Shannon’s theory, Chaos, and its hardware implementation. The algorithm is a block cipher scheme that treats a block of 128-bit length. The paper discusses the method’s two main parts: the key generator-based chaos and the block cipher algorithm that consists of a whitening operation, random permutation, and key-dependent S-box. Both the key generator and the encryption algorithm are implemented using VHDL language on a Xilinx ZedBoard Zynq Development Kit platform. The generated key sequences are tested using the NIST SP 800-22 test suite. The hardware implementation of the proposed algorithm is capable of securing different gray-scale images while maintaining low power usage, a good frequency of 129.9038 MHz, and a high throughput of 1651.201 Mbps. Both statistical and differential attacks demonstrate that the cryptosystem is capable of effective security against the next generation of cryptographic attacks.

In recent years, multimedia techniques have rapidly advanced. Video discs with massive storage capacities exemplify widely used multimedia storage mediums. However, online multimedia services face strict storage limits, and compression rates impact streaming quality. This poses a dilemma—data hiding risks exceeding storage limits and affecting the quality of video streams. To address this, we propose a lossless data hiding technique focusing on cover data size. The technique embeds new data into compressed cover video data, decreasing data size post-embedding. Extraction requires no key. In addition, the processing time is short enough that the technique can be used in real-time applications. We verified the technique through experiments with MPEG-2 videos. Video data size decreased after hiding data, and quality degradation remained acceptable. Results confirm the technique’s efficacy.

Unsupervised salient object detection aims to automatically detect important or attractive target objects in the scene without any user annotation. Compared to fully supervised, it can save a lot of manpower and resources invested in pixel-level annotation. However, the performance of existing unsupervised methods is still far from satisfactory because of the interference of complex backgrounds and the lack of knowledge in the semantic layer of scene objects. To solve these challenging problems, in this paper, we propose a new bottom-up method to better highlight the integral salient object region by exploiting the co-occurrence of saliency and objectness properties. Specifically, saliency estimation effectively utilizes low-level visual features based on high-quality hierarchical image segmentation (convolutional oriented boundaries). The objectness scoring is mainly obtained through the improved object proposals mapping map. Experiment results demonstrate that the proposed method can efficiently remove the interference of non-salient objects and complex background while preserving more salient object regions.

Visual object tracking is a crucial task across numerous computer vision applications. However, object tracking algorithms face significant challenges stemming from deformation and fast motion, which frequently incur dramatic changes to the target’s appearance. To address this problem, we propose a Siamese-network-based object tracking method that combines contrastive learning and adaptive template updating. First, we designed a contrastive branch based on a Siamese network, which constructs a contrastive network with a template branch. The objective of this network is to capture the invariance associated with different images of the same target. Subsequently, we implemented an adaptive template updating strategy for the timely capture of appearance changes in targets and adjusted the size and position of the bounding box. Finally, the α-complete intersection over union function was introduced to continuously optimize the generation of bounding boxes during training, guiding the model to produce more accurate tracking boxes and further improving the algorithm performance. The experimental results demonstrate that our algorithm achieves advanced performance on five datasets, namely GOT-10k, LaSOT, UAV123, OTB2015, and NFS.

With the development of technology and the diversification of information acquisition methods, video has become an important component of our daily life and work, and the security of video information is of increasing concern. In this work, a general video encryption algorithm is proposed and implemented. The algorithm extracts a video file frame by frame to obtain video frames and encrypts the video by encrypting its frames. The bilinear interpolation method is used to reduce the size of the video frames, and then one of the four fractional order chaotic systems and one of three scrambling methods are selected to scramble and diffuse the video frames. This work conducted in-depth feasibility testing on the encryption and decryption process, analyzed the resistance to statistical attacks, and completed tests on the histogram, correlation of adjacent pixels, and information entropy, as well as sensitivity analysis and noise resistance analysis. Moreover, the ability of encryption and restoration were compared with other methods. In addition, the performance comparison of three scrambling methods was also conducted. The encryption effect of these methods is not very different, but the restoration ability of the pixel scrambling is not as good as the other two methods. The experimental results show that this algorithm has better security and encryption efficiency.

At present, most of the reversible data hiding (RDH) algorithms divide images into regular rectangular blocks for data embedding, and most of the RDH algorithms for color images apply the steganography algorithms of grayscale images directly to each color channel of color images. We propose a color image RDH algorithm combining superpixel segmentation and pixel value ordering (PVO), which uses a superpixel segmentation technique to segment the image into irregular pixel blocks with high color similarity to improve the number of smooth blocks and the accuracy of pixel prediction. We then propose a hybrid embedding algorithm based on PVO for B and R channels to reduce the modification of G channel pixels to reduce the steganographic image distortion level. Experimental results show that the proposed algorithm obtains higher quality steganographic images than the existing RDH algorithms.

Generative adversarial networks have shown remarkable success in image synthesis, especially StyleGANs. Equipped with delicate and specific designs, StyleGANs are capable of synthesizing high-resolution and high-fidelity images. Previous works aiming at improving StyleGANs mainly focus on modifying the architecture of StyleGANs or transferring knowledge from other domains. However, the knowledge contained in StyleGANs trained in the same domain is still unexplored. We aim to further boost the performance of StyleGANs from the perspective of knowledge distillation, i.e., improving uncompressed StyleGANs with the aid of teacher StyleGANs trained in the same domain. Motivated by the implicit distribution contained in the pretrained teacher discriminator, we propose to exploit the teacher discriminator to additionally supervise the student generator of StyleGANs so as to leverage the knowledge in the teacher discriminator. With the proposed distillation scheme, our method can outperform original StyleGANs on several large-scale datasets, achieving state-of-the-art on AFHQv2.

In recent years, deep learning-based objects detection algorithms have demonstrated exceptional performance in natural environments. These algorithms have been extensively used in various remote sensing applications, which include the detection of structures and roads as well as flood and earthquake disasters. In these applications, remote sensing images may be captured by satellites, drones, and other equipment. Compared with conventional images, they often feature substantial occlusion, intricate backgrounds and numerous small targets, which are difficult to detect because of the high resolution and large data volume. The existing algorithms focus on detection accuracy or speed, which often fail to achieve a balance between these. To solve this problem, we proposed a single-stage object detection algorithm MA-YOLO based on YOLOv4. We first design a backbone network aimed to enhance feature extraction capabilities while maintaining inference speed. Second, we introduced a parallel attention mechanism, which is to improve the detection performance of small targets. Finally, we applied an attention mechanism to the path aggregation network, which is to enhance the fusion effect of multi-scale features for detecting multi-scale targets. To validate the efficacy of our proposed approach, we evaluated MA-YOLO on three datasets: DIOR, RSOD, and NWPU VHR-10. The experimental results show that our proposed network achieves detection accuracy of 68.87%, 94.13%, and 93.77% on these datasets while ensuring the reasoning speed of 28.4 frames per second and realizes the effective balance between detection accuracy and speed.

Due to absorption and scattering effects in the underwater medium, acquired underwater images often suffer from hazy content and color casts, which lead to significant degradation of visual quality. In this work, a dehazing model is proposed; it is divided into three parts and effectively eliminates the problem of visual degradation caused by hazy content. The attenuation characteristics of light at different wavelengths are different, which usually leads to significant color bias in the obtained underwater images. Therefore, underwater images are first color corrected, namely by red channel compensation and white balance processing. Next, because the original hazy images are usually underexposed, we manually adjust the luminance of the images. The resulting multi-level luminance images are then fused using the Gaussian–Laplacian pyramid scheme to produce clear underwater images. To verify the validity of our method, we evaluate several representative underwater image datasets and compare our method with several advanced traditional methods and deep learning methods developed in recent years. Our method shows excellent results in both qualitative analysis and quantitative comparison.

Semantic segmentation plays a crucial role in practical applications, such as autonomous driving and robot navigation. However, prevalent semantic segmentation networks suffer from two primary challenges: oversized networks with redundant parameters that hinder network inference speed and excessively lightweight network structures that sacrifice semantic segmentation accuracy. Therefore, it is essential to design a semantic segmentation network that strikes a balance between accuracy and inference speed. We propose the asymmetric residual bottleneck module, which incorporates dilated convolution, depth-wise separable asymmetric convolution, channel attention mechanism, and a channel shuffle unit. By utilizing these components, model parameters are effectively reduced, and inference speed is accelerated. Furthermore, a feature aggregation module is designed to integrate features from feature maps with various resolutions, thereby enhancing segmentation accuracy. Based on these advancements, an efficient and lightweight real-time semantic segmentation network called efficiently lightweight asymmetrical network (ELANet) is proposed. Experimental results of the Cityscapes and CamVid datasets demonstrate that ELANet strikes a favorable balance between speed and accuracy. Notably, without any pretrained model and postprocessing scheme, ELANet achieves an impressive mean intersection over union of 72.5% on the Cityscapes test dataset with only 0.82 million parameters, operating at an inference speed of 173.5 frames per second on a single NVIDIA GTX 3090 GPU, with a 512×1024 input image. These findings underscore ELANet’s tremendous potential for real-time applications.

Underwater images received by underwater robots in unrestricted environment during underwater operations are characterized by overall bluish and greenish tones, blurrier edge details, and low contrast. This phenomenon is due to the attenuation and scattering of light in the water and the influence of artificial light sources. To improve the visual performance of underwater imaging, we propose a two-step method of improved gated context aggregation network and gray world algorithms. First, based on similarities between the underwater optical imaging model and the atmosphere model, a gray world algorithm is used to calibrate the image. Then, the corrected underwater images are inputted into an improved gated contextual aggregation network, which is utilized to fuse the features at different levels within the images. The introduction of convolutional block attention module and residual structure can effectively improve the feature extraction ability and prevent network degradation. The method eliminates the grid artifact phenomenon and improves the flexibility of channel information to achieve image enhancement. By conducting experiments, we then compare the performance of the proposed method with six classical state-of-the-art methods. The qualitative results confirm that the proposed method is capable of effectively removing haze, correcting the color deviation of underwater images, and maintaining the naturalness of images. We further perform a quantitative evaluation and show that the proposed method outperforms the other methods that are compared with the proposed in terms of peak signal-to-noise ratio and structural similarity . The enhancement results are also measured by employing the information entropy and underwater color image quality evaluation index, indicating that the proposed method exhibits the highest mean values of 8.0579, 8.1194, and 0.6182, 0.5914 on the two datasets. The experimental results collectively validate the effectiveness of the proposed method in improved underwater image blur.

Shadows in polarized images often interfere with the acquisition and analysis of the polarization state of light. By removing the shadows, these interferences can be eliminated, and the polarization information can be extracted more accurately for subsequent processing. To solve the problem of insufficient illumination and information recovery in the shadow region of polarization images, we propose an attention condition generative adversarial network (GAN) for shadow removal in polarized imaging. The method uses conditional GAN s as its basic framework and introduces attention modules into the generator, which enhances the network’s ability to localize and recognize shadows. At the same time, the polarization shadow removal images from different directions are fused to maximize the generation of polarization shadow-free images. The acquisition of shadow images in four different scenarios uses a polarization camera. The proposed method is compared with the recent method. Experimental results show that the polarization image after shadow removal using our method is closer to the real ground image, and the evaluation index is better than other methods.

Robust video hashing is an effective method of copy detection. But most existing schemes do not make effective classifications; thus their copy detection performances are unsatisfactory. To address these problems, we propose a robust video hashing based on deep feature and the quaternion generic Fourier descriptor (QGFD) for copy detection. In the proposed hashing scheme, an entropy-weighted secondary frame is calculated by weighting all frames in a video group via color entropy. Because color entropy can reflect frame color information, the incorporation of color entropy into each frame guarantees the discrimination of the proposed hashing scheme and makes the obtained entropy-weighted secondary frame discriminative. Next, the entropy-weighted secondary frames are used to extract deep features and geometric invariant features. The deep features are extracted from the low-frequency discrete cosine transform (DCT) coefficients of feature maps learned by three Inception modules of GoogLeNet. As the low-frequency DCT coefficients indicate feature map content and are stable, the deep features provide good discrimination and robustness. The geometric invariant features are extracted by QGFD. Because QGFD are geometric invariant, the use of QGFD can improve the robustness of the proposed hashing scheme. Hash is ultimately determined by quantifying and combining deep features and geometric invariant features. Many experiments were performed on open datasets and indicate that the proposed hashing scheme is superior to the evaluated hashing schemes in the task of copy detection and classification.

Visual object tracking algorithms based on Siamese networks yield promising results through offline training on large benchmarks. However, they cannot adapt well to changes in the target’s appearance and tracking scenarios during online tracking because they rely on a single initial template. Most existing template update algorithms use the tracking result of the previous frame to update the template. If the tracking results deteriorate, it can cause the tracker to accumulate incorrect templates, leading to tracking drift. In this work, we proposed a dynamic template updating Siamese network called dynamic template updating (DTU)-Track that utilizes status feedback with quality evaluation for visual object tracking. This network dynamically selects the tracking template for the next frame based on feedback from the tracking result of the previous frame. It calculates the updating quality score and tracking quality score by using the tracking result of the previous frame. The updating quality score determines whether the current tracking result can be stored in the template library. The tracking quality score quantifies the quality of the tracking status of the previous frame. The quality conversion module determines the number of templates required for the next frame by evaluating the tracking quality of the previous frame. The template extraction mechanism selects high-quality, diverse templates from the template library, thereby enabling the tracker to effectively adapt to changes in tracking scenarios and target’s appearance in the subsequent frame. Extensive experiments on large-scale benchmarks, such as OTB-2015, UAV123, GOT-10k, TrackingNet, and LaSOT, demonstrated our method’s superiority over other algorithms in various tracking scenarios, yielding a real-time speed of 36 FPS.

As deep learning moves toward edge computing, researchers have developed techniques for efficient resource usage and accurate inference on mobile devices. Quantization, as one of the key approaches, enables the deployment of deep learning models on embedded platforms. However, MobileNet’s accuracy suffers due to quantization errors in depth-wise separable convolutions. To reach a smaller model size, we turn to a mixed-precision quantization strategy instead of uniform quantization. Motivated to gain a higher precision, a quantization-friendly separable convolution architecture has been conducted in a mixed precision quantization strategy search. Our approach introduces a quantization-friendly separable convolution architecture, enhancing MobileNet’s accuracy by addressing redundancy and quantization loss. Our framework demonstrates an eight times model size reduction with minimal accuracy loss compared to fixed-bit quantization. Evaluating on the ImageNet dataset and common objects in context dataset, our modified MobileNets almost closed the gap to the floating pipeline across 2-, 4-, 6-, and 8-bit settings. In the ablation experiment, after mixed quantization, our model can still maintain an accuracy of 72.84%, whereas our model has been compressed more than eight times.

Super-resolution techniques are crucial in improving image granularity, particularly in complex urban scenes, where preserving geometric structures is vital for data-informed cultural heritage applications. We propose a city scene super-resolution method via geometric error minimization. The geometric-consistent mechanism leverages the Hough transform to extract regular geometric features in city scenes, enabling the computation of geometric errors between low-resolution and high-resolution images. By minimizing mixed mean square error and geometric align error during the super-resolution process, the proposed method efficiently restores details and geometric regularities. Extensive validations on the SET14, BSD300, Cityscapes, and GSV-Cities datasets demonstrate that the proposed method outperforms existing state-of-the-art methods, especially in urban scenes.

In today’s digital era, network communication has become ubiquitous, evincing pressing concerns regarding the confidentiality of transmitted information. Given heightened public scrutiny of information security, image steganalysis has emerged as a pivotal concern within the ambit of information security. To further optimize the image steganalysis algorithm, attention mechanism is introduced into convolutional neural network, which further improves the accuracy and recognition efficiency of the algorithm. The experimental results show that by conducting steganalysis on 20,000 images in the database, the recognition accuracy of the research model is 92.58%, and the error recognition rate is 13.44%, which basically meets the research requirements. After conducting a performance test of the attention module and implementing it, the analysis model’s error rate has decreased to different extents. Comparing the performance of the algorithms using the wavelet obtained weights and spatial universal wavelet relative distortion algorithms as detection criteria, the error rates of steganalysis under four parameter settings are 21.4%, 11.6%, 20.8%, and 13.9%, respectively. These are the lowest values in each model, further verifying the optimization performance.

In recent years, significant progress has been made in single-image super-resolution (SISR) with the emergence of convolutional neural networks (CNNs). However, the application of SISR on low computing power devices is hindered by the massive number of parameters and computational costs. Despite the focus on lightweight SISR models in many studies, the majority still struggles to balance performance and model size, making it difficult to apply them in real-life situations. Therefore, we propose to construct an SISR network termed 3D lightweight image super-resolution (3DLSR) network by introducing 3DCNN to this task. By leveraging the additional dimension of 3D convolution, the proposed 3DLSR can extract the interchannel and innerchannel information of color images, thereby aiding the reconstruction of high-resolution images while maintaining a small model size. Furthermore, we redesign a best-fitting network structure for 3DLSR based on the difference between 3D convolution and 2D convolution. The experimental results demonstrate the superiority of our 3DLSR, as it can achieve a competitively quantitative metric with a parameter size one order of magnitude smaller than the majority, compared with the state-of-the-art methods.

Event cameras record moving objects with asynchronous event streams. It remains a challenge to make full use of the spatiotemporal information of event streams to extract high-quality features and to make event camera object recognition. We propose an event-based event camera object recognition system, which includes a denoising module and an object recognition module. The denoising module removes noise events using spatiotemporal information of the events. The object recognition module extracts primary spatiotemporal features based on the event time surface prototypes obtained by clustering, then further extracts complex diagnostic features, and makes object recognition using spiking neural networks with reward-modulated spike-timing-dependent plasticity learning rule. Experimental results demonstrate that our system has better performance than baseline methods on five popular event camera datasets, especially on datasets with rich spatiotemporal dynamics. Our denoising method improves the noise robustness of our event camera object recognition system on high-speed moving datasets that are greatly affected by noises. Moreover, our method has much better recognition ability than baseline methods when using short input event streams. Our method is very beneficial for developing event-based event camera object recognition algorithm when event streams are short or noises are serious.

This paper presents a run length encoding (RLE) based reversible data hiding (RDH) scheme designed specifically for higher order pixel group values of image intensities. RDH aims to embed additional data into the host image while maintaining the ability to completely recover the original image without any loss. The proposed scheme utilizes the RLE technique to exploit the statistical characteristics of higher order intensity values in images. By identifying suitable run-length blocks within the image, the scheme intelligently embeds the additional data, ensuring efficient utilization of the available capacity. The model again encrypts the image after data embedding to establish the confidentiality as well as integrity of the hidden data, adding an extra layer of security to protect against unauthorized access and alterations. To achieve reversibility, the scheme carefully encodes the embedded data within the run-length segments, ensuring that the original image can be perfectly reconstructed. By embedding data, the visual quality of the image is intentionally reduced. However, using an encryption technique randomness is increased there by frustrating potential attackers when they attack. The average embedding capacity for a 512×512 image is approx. 177463 bits and the entropy after encryption is 7.999 bpp. Experimentation results demonstrate the efficacy of the proposed scheme in terms of reversibility, embedding capacity, and entropy values, thereby showcasing its potential for practical applications.

In recent years, unsupervised learning has gained significant attention as a promising approach for monocular depth estimation. We propose an unsupervised monocular depth learning approach that combines self-teaching method and contrast-enhanced structural similarity (SSIM) loss. The self-teaching method involves learning from both a teacher and a student network. The teacher network generates a pseudo-ground truth depth map from non-augmented images and highest predicted resolution. The student network is trained on augmented images and related low resolution to supervise the training process. This approach allows the student network to learn from the teacher network and improves the quality of the depth predictions. In addition, we introduce a new contrast-enhanced version of the SSIM loss that improves the learning process by emphasizing the contrast between the predicted depth and the ground truth. Our experiments on the KITTI datasets demonstrate that our approach outperforms many unsupervised depth learning methods.

In this work, a secure encryption technique for implementing reversible data hiding, named the Huffman Ciphersystem, is developed. First, a cover image is decomposed into two sub-images. The first sub-image consists of four most significant bits (MSBs), and the second sub-image consists of four least significant bits. The difference of adjacent MSB values for the first sub-image is calculated using inverse S scan order because it gives a confined and small range of difference values. It helps to attain an efficient lossless compression, enhancing our proposed work’s embedding capacity (EC). On the other hand, security is imparted by our proposed Huffman Ciphersystem by same level node exchange strategy followed by encryption of the dictionary via scrambling. The proposed scheme achieves a peak signal-to-noise ratio value of infinite for all the tested standard grayscale images at the recovery time. Experimental results demonstrate the proposed work’s superiority in EC and security of the encrypted images.

In the field of automatic defect detection, a major challenge in training accurate classifiers using supervised learning is the insufficient and limited diversity of datasets. Obtaining an adequate amount of image data depicting defective surfaces in an industrial setting can be costly and time-consuming. Furthermore, the collected dataset may suffer from selection bias, resulting in an underrepresentation of certain defect classes. Our research aims to address surface defect detection in titanium metal spacer rings by introducing an approach leveraging a digital twin framework. The behavior of the digital twin is optimized using a reinforcement learning (RL) algorithm. The optimized digital twin is then used to generate synthetic data, which is employed to train a spacer defect detection classifier. The classifier’s performance is evaluated using real-world data. The results show that the model trained with synthetic data outperforms the one trained on a limited amount of real data. Our work highlights the potential of digital twin-based synthetic data generation and RL optimization in enhancing spacer surface defect detection and addressing the data scarcity challenge in the field. When the inspection network is trained solely using generated synthetic data, it achieves an inspection precision of 96.10%, with a recall of 85.77%. Incorporating real data alongside synthetic data for training further enhances performance, yielding an inspection precision of 93.07% and a recall of 94.20%. This surpasses the defect detection capabilities observed when training the inspection network exclusively with real data.

To tackle the problem of human trajectory prediction in complex scenes, we propose a model using hypergraph convolutional neural networks for social interaction (HGCNSI). Our model leverages a hypergraph structure to capture both high-order interactions and complex social dynamics among pedestrians (who often influence each other in a nonlinear and structured manner). First, we propose a social interaction module that improves the accuracy of interaction modeling by distinguishing between interacting and non-interacting pedestrians. Then, the hypergraph structure that can capture the complex and nonlinear relations among multiple pedestrians from the social interaction module is constructed. Furthermore, we exploit an improved attention mechanism called scene-coordinates attention that fuses the spatial and temporal features and models the unique historical movement information of each person. Finally, we introduce the SIRF module that filters the trajectories within one iteration to reduce the computational complexity and improve the prediction performance. We evaluate the proposed HGCNSI model on five publicly available datasets and demonstrate that it achieves state-of-the-art results. Specifically, the experiments show that our model outperforms existing methods in terms of prediction accuracy, using evaluation metrics, such as the average displacement error and the final displacement error.

Aerial image object detection has a wide range of application values in civilian or military fields. Due to its unique high-altitude imaging viewpoint and the multiangle shooting method, aerial images lead to problems, such as small objects being detected in the image, large variations in object scales, and dense distribution. To alleviate the above problems, we propose an improved aerial image object detection algorithm aerial images object detection based on YOLO (AIOD-YOLO) based on YOLOv8-s. First, we propose the multibranch contextual information aggregation module. It enhances the network’s perception of small objects by associating object information with the surrounding environment, thereby compensating for the lack of feature information for small objects. In addition, we propose the multilayer feature cascade efficient aggregation network, which leverages multigradient flow fusion of features at different scales. This approach aids the network in capturing a wide range of scale information and effectively mitigates the issue of missed detections caused by variations in object scales. Finally, we propose the adaptive task alignment label assignment strategy to address the issue of dense object distribution. The strategy incorporates the cosine similarity calculation to assess alignment globally and simultaneously adjusts the weights of positive and negative samples. We optimize the precision of label assignment for dense objects in aerial images, effectively resolving the challenges posed by their close proximity. The experiments on the VisDrone dataset demonstrate that AIOD-YOLO achieves a significant 7.2% improvement in mAP compared to the baseline model YOLOv8-s. The mAP0.5 of AIOD-YOLO is also improved by 14.1%, 7.9%, and 7.5% on SeaDronesSee v2, AI-TOD, and TinyPerson datasets, respectively, which validates the generalization of our proposed algorithm. AIOD-YOLO offers a superior information processing approach for tasks related to aerial image object detection in both civilian and military applications.

Real-time and reliable three-dimensional (3D) object detection is critical for the autonomous driving system. Many 3D object detectors have recently adopted the pillar-based method represented by PointPillars due to its benefits of simplicity, speed, modularity, and ease of expansion. However, the detection effectiveness of the pillar-based method is limited by the lack of intra-pillar feature learning and insufficient multiscale pseudo-image feature extraction. Therefore, we present a multilayer pseudo-image 3D object detection method to enhance the performance of the pillar-based method. First, the proposed detector utilizes a multiscale feature extraction module to learn the point cloud representation within the pillars. The pillarized point cloud space is then converted into two pseudo-images of the same size, each encoding unique spatial information. After being individually processed via an inter-channel attention mechanism, these two pseudo-images get concatenated to form a global pseudo-feature map. Finally, a multiscale feature extraction backbone network with upper- and lower-level fusion is employed to process the global pseudo-feature map into a high-level representation. The multi-task detection head then generates the final detection results. Experimental findings on the KITTI dataset validate the effectiveness and superiority of the proposed method. The 3D mAP, BEV mAP, and mAOS of the proposed method are improved by 2.85%, 2.48%, and 2.76% compared to the traditional PointPillars, respectively.

Gait recognition plays an important role in the area of biometric recognition. Despite that much progress has been made for gait recognition in recent years, most of them are based on lateral-view gait characteristics. These methods usually require a large data collection area to capture full gait sequences, which are only applicable in wide outdoor spaces. In this paper, we propose a new frontal-view method for human gait recognition based on gait dynamics and deep transfer learning. The frontal human silhouettes are characterized with four kinds of time-varying gait features, namely the lower limb length ratio, lower limb area ratio, swing area of lower limb, and swing angle of lower limb. In the training phase, gait dynamics underlying the aforementioned time-varying gait features is approximated by radial basis function neural networks through deterministic learning theory. This kind of dynamics information reflects the essential dynamical attribute of human walking and is insensitive to the deformation of gait silhouettes caused by frontal-view walking process. To further improve the recognition rates, the extracted gait dynamics and deep transfer learning technique are incorporated, and a deep transfer learning based feature fusion strategy are investigated for human identification task in the recognition phase. To further improve the robustness against different walking conditions, three kinds of gait representations are fused to provide a comprehensive characterization of human walking movement. Finally, comprehensive experiments are carried out on the well-known public gait databases to demonstrate the recognition performance of the proposed algorithm.

Few-shot classification aims to classify samples with a limited quantity of labeled training data, and it can be widely applied in practical scenarios such as wastewater treatment plants and healthcare. Compared with traditional methods, existing deep metric-based algorithms have excelled in few-shot classification tasks, but some issues need to be further investigated. While current standard convolutional networks can extract expressive depth features, they do not fully exploit the relationships among input sample attributes. Two problems are included here: (1) how to extract more expressive features and transform them into attributes, and (2) how to obtain the optimal combination of sample class attributes. This paper proposes a few-shot classification method based on manifold metric learning (MML) with feature space embedded in symmetric positive definite (SPD) manifolds to overcome the above limitations. First, significant features are extracted using the proposed joint dynamic convolution module. Second, the definition and properties of Riemannian popular strictly convex geodesics are used to minimize the proposed MML loss function and obtain the optimal attribute correlation matrix A. We theoretically prove that the MML is popularly strictly convex in the SPD and obtain the global optimal solution in the closed space. Extensive experimental results on popular datasets show that our proposed approach outperforms other state-of-the-art methods.

Cracks are one of the major threats to the safe operation of civil infrastructures, so timely and accurate detection of cracks is crucial for accident prevention. However, in practical applications, the poor continuity and low contrast of many cracks (e.g., pavement cracks) pose a great challenge to image-based crack detection. In previous approaches, the detection results are often unsatisfactory due to the limitation of the dataset and the singularity of the detection method. To solve this problem, we propose a frequency and spatial dual guide network for crack detection. First, global frequency domain information is processed by the fast Fourier transform to extract low-level features, and then the convolutional neural network uses global frequency domain information to learn local features of the image to extract high-level features. Finally, the feature maps of the two are weighted and fused element by element to obtain the final detection result. Extensive experiments show that our method is superior to any other existing method.

Grouping images into different themes is a challenging task in photo book curation. Unlike image object recognition, image theme recognition focuses on the understanding of the main subject or overall meaning conveyed by an image. However, it is challenging to achieve satisfactory performance using existing general image recognition methods. In this work, we aim to solve the image theme recognition task with few-shot training samples using pre-trained contrastive language-image models. A text-prompt-guided few-shot image adaptation framework is proposed, which incorporates a text-embedding-guided classifier and an auxiliary classification loss to exploit embedded visual and text features, stabilize the network training, and enhance recognition performance. We also present an annotated dataset Theme25 for studying image theme recognition. We conducted experiments on our Theme25 dataset as well as the publicly available CIFAR100 and ImageNet datasets to demonstrate the superiority of our method over the compared state-of-the-art methods.

Video denoising is a fundamental task in low-level computer vision. Most existing denoising algorithms use synthetic data learning. However, there is a significant difference between the noise distributions of synthetic and natural data, which leads to poor generalization performance of the model in actual scenes. Hence, a video method based on an offset optical flow-guided transformer is proposed. The proposed method adopts a semisupervised framework to improve the model’s generalization performance, designs the offset optical flow to guide the transformer in capturing critical information, and performs global self-similarity modeling using neighboring spatiotemporal domain features to improve the denoising performance. In addition, contrastive learning is introduced in the supervised branch to prevent the fitting of wrong labels, imaging prior information to mine sequence features in the unsupervised branch, and a two-branch memory loss is introduced to reduce the difference of double branch training. Experimental results on synthetic and real videos demonstrate that our method has obvious quantitative and qualitative improvements over state-of-the-art methods with fewer parameters.

Global contextual features are essential in computer vision tasks. Traditional convolutional networks are limited by the size of the convolutional kernel, resulting in a limited receptive field for each layer of the network. To address this issue, transformers introduced global attention, which has demonstrated excellent performance in natural language processing and has been widely applied in visual tasks. However, both convolutional networks and transformer models are constrained by the arrangement of data in Euclidean space, making it challenging to effectively extract features of irregular objects and complex scenes. We propose a graph neural network-based module (Graphion) for extracting global contextual features. Graphion maps the feature maps into a non-Euclidean space, establishes adjacency relationships among image patches, and uses a graph neural network for message passing and aggregation operations on the obtained node features, enabling the learning of correlation information from the graph structure. Consequently, it facilitates more efficient feature extraction of irregular object instances. Graphion is flexible and portable, allowing integration into visual feature extraction backbones. Extensive experiments were conducted on multiple datasets, including MS COCO, ADE20K, and ImageNet64. The proposed Graphion method demonstrates outstanding performance in object detection, segmentation, and classification.

A new type of motion saliency—that is, temporal context-aware motion saliency (TCAMS)—was proposed to detect the saliency of motion using its temporal context, presenting the semantic information of an event in a highly informative manner. Our definition differs from the typical definition of motion saliency. According to our definition of TCAMS, a novel detection method is proposed based on the interaction between the human visual and working memory systems, transferring visual intelligence to machine intelligence. Experimental evaluations demonstrated that the proposed TCAMS detection method exhibited accurate motion-saliency detection and good temporal context description performance. The benefits of the proposed TCAMS detection method were also comprehensively demonstrated using a wide range of applications, from low-level to high-level computer vision tasks—such as object detection, video synopsis, and action recognition—where the temporal context of motion was as essential as the motion itself. We showed that TCAMS could present an accurate motion state in a clear and informative manner.

Remote photoplethysmography (rPPG) is an optical technique that measures physiological signals from facial videos by analyzing subtle changes in the skin blood volume. However, rPPG signals generated in practical applications are easily affected by external environmental factors and the state of individuals, leading to irregular waveform variations that increase the difficulty in heart rate estimation. To improve the regularity of generated rPPG signals, we propose a standardized rPPG signal generation method. Specifically, facial videos are fed into the generator of a generative adversarial network (GAN) to predict a rough rPPG signal by supervised learning. In addition, a mathematical signal synthesizer model is used to generate noise-free standardized rPPG signals, which are subsequently fed into a discriminator along with the predicted signal for adversarial learning. This enables the generator to learn more standardized waveforms. As a result, the predicted signal waveform by the generator becomes closer to the waveform distribution of real rPPG signals. The proposed method is validated on the widely used MAHNOB-HCI, UBFC-rPPG, and MMSE-HR databases and shows significant improvements in the prediction accuracy and signal-to-noise ratio.

A common problem in the field of object detection is that the image features could not be fully expressed. And another issue is that the static query selection in the detection transformer (DETR)-like models cannot adapt well to different datasets due to the fixed number of selected object queries. To solve these problems, hollow attention (HA) and dynamic query selection (DQS) modules were proposed, and a network HA-DQS-Net was further formed. HA integrates specially designed masks into self-attention to better combine channel and spatial directional feature information, thereby learning more complex and comprehensive target features. DQS improves the idea of static query selection in the current DETR-like model by dynamically selecting the number of object queries based on the actual number of targets in the image, which enhances the accuracy of the model. HA-DQS-Net, which combines the advantages of HA and DQS, has a competitive performance in the field of object detection. The excellent detection effectiveness of our viewpoint is validated based on PASVAL VOC and a homemade smoking dataset. It is worth noting that all APs have been improved when HA is applied to different DETR-like models, which improves the universality of the HA module.

Snapshot spectral imaging enables the acquisition of hyperspectral images (HSI) employing specialized optical systems, such as the coded aperture snapshot spectral imager (CASSI). Specifically, the CASSI system performs spatiospectral codification of light obtaining two-dimensional projected measurements, and these measurements are then processed by computational algorithms to obtain the desired spectral images. However, because HSIs often have a high spatial or spectral resolution, the sensing matrix related to the acquisition protocol becomes very large, leading to a high computational storage cost and long computation times. In this work, we propose an algebraic framework for computing the relevant operations in a tensorial form based on the nature of the codification protocol. We then test our framework against some comparison methods based on linear algebra decomposition, factorization, or block-operations, demonstrating that the proposed method is between 3 and 20 times faster than the best-competing method. Moreover, the gain becomes larger when the matrices become bigger, corresponding to realistic HSI sizes for spectral imaging applications. In extreme cases, our method can still operate when the competing methods stall due to memory shortage.

Reinforcement learning (RL) has recently made several significant advances using video games as a testbed. While many of these games are relatively self-contained, there has been a recent push to develop agents capable of tackling massive, open-ended environments that are more reminiscent of the real world. One of the most popular of these platforms is Minecraft, but to attain human-level performance, agents must be able to learn, plan, and reason using high-dimensional image input. Commonly, an agent will attempt to extract lower-dimensional features that assist with downstream tasks. However, representation learning techniques have primarily been applied to real-world, natural image datasets, and it is unclear how these same methods might translate to an artificial world with non-natural images. We therefore present MiDaS, a novel large-scale Minecraft dataset featuring 36,000 labeled images across 60 classes. MiDaS contains information about both the blocks in the image, critical to solving the game, as well as auxiliary information such as time of day and biome. Further, we perform an evaluation of various models to benchmark performance on this new dataset. Since RL agents must be capable of learning features without labels, we include benchmarks of various self-supervised learning approaches on the dataset. Our results indicate that self-supervised methods perform best in the linear evaluation paradigm, particularly in low-label settings with a ResNet-based backbone, whereas ImageNet-pretraining assists more in the fine-tuning setting. The full dataset is available at https://github.com/MinecraftDataset/MiDaS.

Partial convolution and gated convolution (GC) have been widely used to solve the limitations of vanilla convolution. However, both approaches have their respective drawbacks. For example, the single-channel binary mask used in partial convolution restricts its flexibility, and the accuracy of the gating values learned from GC cannot be guaranteed. To overcome these limitations, we propose an approach called bi-gated convolution. It adaptively integrates the binary mask and the gating values learned from the network to obtain refined gating values that effectively characterize the features, thereby increasing the recognition accuracy of the gating value. Furthermore, we propose a feature-adaptive supplementation operation designed specifically for repairing damaged areas within the encoder features. Finally, we present a mutual encoder-decoder architecture with bi-gated convolution. Experiments conducted on two benchmark datasets show that the proposed method has the capability to generate visually plausible results.

Due to the rise of social media, synthesized or composited photos are becoming increasingly widespread, and image relighting is one of the crucial technologies that is capable of creating convincingly realistic images. Our study proposes a framework for relighting a portrait subject when superimposing it onto a 360-deg image. In most image compositions, it is difficult to acquire the 3D shapes of subjects directly to rerender them in a virtual environment. A well-diffused color portrait image with a corresponding normal map is generated from our photo booth using a photometric method. In addition, a virtual environment based on a principled bidirectional scattering distribution shader and environmental 360-deg texture in the Blender software is utilized to create composite images. After considering different situations, including gender, postures, indoor or outdoor scenes, and color or color-free subjects, each of 128 composite images was played as a 4-s video clip, and various scenarios were conducted for subjective assessment. From the evaluation scores of the 30 participants, the overall satisfaction with the image composition based on the proposed framework was above average (5-point Likert scale > 3 points), and the color-free subject in the 360-deg image was significantly preferred.

Real-time image processing and regions of interest extraction are crucial in the non-standard welding process guided by structured light vision. However, due to the impact of detection speed, accuracy, and applicability, existing methods are difficult to apply directly. To address these issues, we have screened various improvement methods through experiments and provided a practical lightweight algorithm, YOLO-DGB, based on YOLOv5s, a current mainstream object detection algorithm. The proposed algorithm introduces depthwise separable convolution and Ghost modules into the backbone network to reduce the number of parameters and floating-point operations per second (FLOPs) in the detection process. To meet the accuracy requirements of the detection network, a bottleneck transformer is introduced after the spatial pyramid pooling fast, which improves the detection effect while ensuring a reasonable number of parameters and computation. To address the issue of insufficient datasets, we propose an improved DCGAN network to enhance the collected images. Compared with the original YOLOv5s network, our proposed algorithm reduces the number of parameters and FLOPs by 35.5% and 44.8%, respectively, while increasing the mAP of the model from 96.5% to 98.1%. Experimental results demonstrate that our algorithm can effectively meet the requirements of actual production processes.

Infrared and visible image fusion aims to generate a more comprehensive image for improving measurement and analysis accuracy. Existing Swin Transformer based methods effectively extract features from medium-large scale receptive fields. However, these methods ignore the complementary role of features extracted from the small receptive field, resulting in the loss of essential fine information. To this end, we propose MCFusion, a dual-branch framework based on a multiscale receptive field and cross-modal enhanced attention mechanism. MCFusion is composed of an attention-guided coarse branch (AGCB) and a fine branch (FineB). First, AGCB and FineB are designed to extract distinct yet complementary features from medium-large and small receptive fields, respectively. Second, a cross-modal enhanced attention mechanism is designed to enhance shared features across different modalities. Third, a loss function fully considering contrast, texture, and illumination intensity is proposed to generate fused images. The test results demonstrate that MCFusion is superior to other state-of-the-art methods and can notably enhance the detection and recognition of targets.

To solve the blurred image edge segmentation problem and the influence of a complex background on the segmentation accuracy of the clustering algorithm, this paper proposes an edge gravity mean shift (EGMS) image segmentation algorithm. Aiming at the problem of partial edge information loss when the Canny algorithm processes images, the L0 gradient minimization method is introduced to iteratively optimize the gradient map to extract more edge information, and then the L0 Canny edge detection algorithm is proposed. Aiming at the problem that the Epanechnikov mean shift algorithm will converge to the local optimum due to the non-smooth Epanechnikov kernel function, the optimal bandwidth is obtained, and the smoothness of the kernel function shape is improved by cross-validation and spline function. Then, the sleek Epanechnikov kernel function is constructed by integrating the weight function, which can reflect the richness of clustering information at the peak so that the kernel function can more accurately reflect the data characteristics of the peak. Based on the sleek Epanechnikov kernel function, the sleek Epanechnikov mean shift (SEMS) clustering algorithm is proposed. The EGMS image segmentation algorithm is constructed by fusing the L0 Canny and SEMS. The experimental results indicate that EGMS can retain more edge contour information when segmenting images, and it can generate better clustering clusters for clustered datasets. In addition, compared to other clustering segmentation algorithms, the proposed EGMS algorithm demonstrates superior clustering accuracy and convergence on the majority of image datasets. Our code is available at: https://gitee.com/zhiweilu111/edge-gravity-mean-shift.

The U-Former model is proposed in this work to segment the COVID-19 lung computed tomography images of patients. U-Former introduces the transformer architecture, based on the traditional U-Net segmentation network, which effectively improves the network’s ability to capture global features. The mixed module is presented in this work to capture long-range dependencies and extract local information. In the mixed module, the computationally expensive self-attention mechanism is enhanced and combined with convolution to enable the network to efficiently capture global information while taking into account local details. The multi-scale attention module is utilized to fuse the multi-scale features to enhance the segmentation effect for details. Experimental results show that the proposed U-Former model outperforms other state-of-the-art segmentation models, including both convolutional neural network-based and transformer-based models, with a mean Dice score of 82.54%, a mean intersection over union of 80.01%, and a mean sensitivity of 85.70%. The code and models are publicly available at https://github.com/tianyuzhou668/U-Former

In the context of real-time video defogging, accurately estimating the transmittance of sky regions and deep fault regions remains a challenging task due to severe color distortion and halo effects. To address this issue, we propose an innovative adaptive wavelet transform defogging (AWTD) scheme aimed at precisely modifying the transmittance of the sky and deep fault regions. Additionally, we present a dedicated field programmable gate array defogging architecture capable of direct application for real-time synchronous defogging of Full HD (1920×1080) videos. Through extensive experiments and comparisons, we demonstrate the effectiveness and practicality of the proposed AWTD approach in mitigating color distortion, halo effects, and other fog-induced artifacts. The results show that the dehazing algorithm attains 60 frames per second for the image resolution of 1920×1080, which is suitable of real time applications.

Deep learning models have achieved great success in the field of ship detection, but these models often require a large amount of computing and storage resources, and are not suitable for some resource-constrained situations. To solve the above problems, we propose a lightweight Swin-YOLOFormer ship detection method. First, in terms of the backbone network, the Swin transformer lightweight model is introduced to reduce the redundancy parameters of the backbone network. Second, in the feature fusion network, an improved ghost-efficient long-range attention network-hierarchy module is proposed to extract features and reduce the burden of model parameters. Finally, in order to prevent feature loss, an improved SPPCPSC module is proposed to enhance the feature of receptive field. Through experimental verification, compared with the YOLOv7 benchmark model, the number of proposed model parameters was reduced by 66.05% to 13.501 M, which not only accelerated the model training speed but also reached 97.81% accuracy. The results show that the proposed method achieves model lightweight and maintain high precision.

Underwater images suffer from localized distortion and blurred degradation of edge structures due to light absorption and scattering by water. However, existing super-resolution (SR) methods for underwater images cannot effectively solve the above problems and encounter model sizes that are too large. To this end, we propose an attention-guided hybrid transformer-CNN network (AHTCN) to improve the SR reconstruction of underwater images through the interaction of local and multiscale global information, as well as the long-range dependencies modeling capability. Specifically, AHTCN mainly consists of several cascaded transformer-CNN feature extraction blocks (TCFEB) and an image reconstruction module. In TCFEB, the designed attention-based channel separation mechanism can adaptively separate the weighted features while reducing the number of model parameters and then extract the local details and global structural information at different scales through the dual-stream structure. Moreover, we replace the feedforward layer in the transformer with the blueprint separable convolutional feedforward layer and propose an enhanced pyramid pooling transformer layer, which helps to strengthen the feature perception of the model. Experimental results demonstrate that AHTCN outperforms the state-of-the-art algorithms in terms of both subjective visual effects and objective quality assessment, while requiring fewer parameters.

Person re-identification (Re-ID) is a significant research field that seeks to enable the identification of individuals among different surveillance cameras, making it a fundamental technology in the field of computer vision. Despite its significance, current models addressing this problem encounter difficulties when tasked with re-identifying individuals over an extended period, owing to changes in appearance such as clothing and hairstyles. We propose an innovative Re-ID network that considers facial features to enhance the model training and to improve the Re-ID performance by extracting pertinent facial information. Specifically, we filter the face images in the training set and optimize the training dataset, enabling our model to focus on essential facial features. In addition to the global stream in the training process, we introduce a distinct facial stream to train the facial features, which is subsequently merged into the overall flow at the loss function stage to enable knowledge transfer between the two branches. Experimental results on different datasets have shown that our method performs much better than the previous state-of-the-art methods in Rank-1 and mAP metrics. This proposed approach therefore presents a innovative and promising solution for the research of cloth-changing person Re-ID, and the associated code and dataset are made available on https://github.com/Titor99/FaceCC

Mask wearing is known as one of the most effective and convenient methods for preventing the transmission of viruses. Accurately determining whether a person is wearing a mask correctly is crucial in curbing the widespread transmission of viruses in public places. To address the limitations of traditional approaches, such as insufficient feature extraction and limited generalizability, an attention-enhanced multiscale fusion approach is proposed for assisting face mask-wearing detection. We first incorporate the simple, parameter-free attention module, adaptively spatial feature fusion module, and wise intersection over union loss function into a multiscale feature fusion framework, constructing the YSAW mask-wearing detection network, resulting in the development of the YSAW mask-wearing detection network. The network was then trained and evaluated on the WIDER face dataset and MAFA dataset, which includes faces, masks, and face masks. Experimental results demonstrate that the proposed YSAW network achieves mean average precision at intersection over union threshold 0.5 (mAP50=94.3%), indicating superior detection performance in comparison to existing methods.

Color-to-gray image conversion is commonly used in applications such as printing, e-ink display, image stylization. Effective decolorization methods aim to maintain the naturalness and the image contrast after the conversion. We propose a method for this conversion problem using the image’s salient colors and radial basis functions. There are three steps in the proposed method. First, we quantize the image colors to a small number of salient representative colors, which essentially provides an adaptive mechanism for choosing the dominant image color regime that can be mapped to the grayscale with the maximum possible contrast structure. A carefully designed criterion is developed for choosing the appropriate number of quantization points in the k-means algorithm based on the elbow method. Second, two methods are proposed for ordering the quantized colors on the grayscale intensities, using the guideline that the ordering should agree with the human perceptual comfort when looking at common grayscale images. Finally, radial basis functions are used to extend the mapping from the quantized colors to the grayscale intensities to all colors of the original image’s pixels in a continuous fashion. Experimental results on the three benchmark datasets of color-to-gray conversion show significant performance improvement over state-of-the-art methods both qualitatively and quantitatively.

The conditional generative adversarial network (cGAN) is widely used in image-to-image translation for natural images by implementing supervised learning loss in the GAN framework. It has also been extended to biomedical images for image segmentation. However, during the training phase, it lacks proper guidance for smooth convergence, and the model requires a large dataset, which is often unavailable for biomedical images. To address this limitation, we integrated a multi-task learning technique using a cGAN-based adversarial framework for lesion segmentation in ultrasound (US) breast images. Our proposition defines an auxiliary task of finding the mask similarity score between the input US image and the ground-truth segmentation mask, which is trained on features extracted from the available data for the segmentation task. This learning methodology is imposed within a cGAN framework to boost the original segmentation task of the generator and guide it through the auxiliary task. The proposed method shows improved convergence and stability compared to the core adversarial network, and its performance has been assessed on different datasets and model architectures, demonstrating significant improvement over the baseline.

For image denoising problems, the structure tensor total variation (STV)-based models show good performances when compared with other competing regularization approaches. However, the STV regularizer does not couple the local information of the image and may not maintain the image details. Therefore, we employ the anisotropic weighted matrix introduced in the anisotropic total variation (ATV) model to improve the STV model. By applying the weighted matrix to the discrete gradient of the patch-based Jacobian operator in STV, our proposed weighted STV (WSTV) model can effectively capture local information from images and maintain their details during the denoising process. The optimization problem in the model is solved by a fast first-order gradient projection algorithm with a complexity result of O(1/i2). For images with different Gaussian noise levels, the experimental results demonstrate that the WSTV model can effectively improve the quality of restored images compared to other TV and STV-based models.

With the rapid development of machinery and equipment modernization, more and more non-standard shaped parts are designed and put into specific occasions to use to meet the needs of special circumstances. Therefore, how to quickly recognize the shaped parts has become an urgent need for a technology. To recognize shaped parts, deep learning methods such as the widely used YOLOv5s network are commonly employed. However, directly deploying the official network model has drawbacks, including heavy reliance on data, poor detection results for small target objects, and high hardware requirements. These issues increase the threshold for non-professionals to use it. For this reason, this paper designs an improved network based on YOLOv5s. This paper proposes improvements in terms of both lightness and accuracy. In terms of light weight, the backbone of YOLOv5s is replaced by MobileNetV3; and the convolution and C3 module of the head part of YOLOv5s is replaced by phantom convolution and C3Ghost module, and the attention mechanism layer is trimmed to reduce the number of computational parameters and model size. In terms of accuracy, non-maximum suppression (NMS) is improved to Soft-NMS; intersection over union (IoU) loss function is replaced with distance-IoU loss function. And trained on the homemade shaped parts dataset, the results show that the average accuracy of the improved network is 99.2% in the test case, the model size is 2.4M, and the detection time is 1.5 ms per image, which is a significant increase in speed and accuracy compared with other unmodified networks, and a substantial decrease in the model size and the number of parameters.

The detection of surface defects on a printed circuit board (PCB) plays a crucial role in ensuring the quality of PCB products. To address the diversity of defect poses and the challenges associated with detecting small objects on PCB, this paper proposes an improved YOLOv7 model for small objects-oriented PCB defect detection. First, this paper improves the regression loss function of YOLOv7 by incorporating wise-IoU (WIoU) and replacing IoU with an outlier degree to develop a gradient-boosting allocation strategy, thereby increasing the network’s accuracy. Second, this paper proposes a coordinate attention dynamic mechanism that performs convolution operations with deformable convolutional networks v2 (DCNv2) using coordinate attention. This mechanism effectively suppresses redundant information. Finally, this paper proposes a dynamic head diverse (DyHead-d) module that prioritizes spatial awareness over scale awareness, building on DyHead. This module improves the network’s ability to localize small targets. Experimental results show that the WDC-YOLO achieves a mean average precision of 98.4% on public datasets, demonstrating a 3.1% improvement compared to the original network. The significantly enhanced detection accuracy meets the real-time requirements of PCB defect diagnosis, which is of great importance for quality control and cost reduction in PCB industrial production.

Real-time human pose estimation (HPE) using convolutional neural networks (CNN) is critical for enabling machines to better understand human beings based on images and videos, and for assisting supervisors in identifying human behavior. However, CNN-based systems are susceptible to adversarial attacks, and the attacks specifically targeting HPE have received little attention. We present a gradient-based adversarial example generation method, named AdaptiveFool, which is designed to effectively perform a keypoints-invisible attack against OpenPose by aggregating the loss function of human keypoints and generating adaptive adversarial perturbations. In addition, we introduce an object-oriented perturbation generation method during the AdaptiveFool process to eliminate background perturbations. Our proposed method adapts the adversarial perturbations and generates object-oriented perturbations. On COCO 2017 datasets, our method achieves 6.3% mean average precision on OpenPose. This research provides inspiration for future work on developing efficient and effective adversarial example defense methods for HPE.

Video synopsis aims to condense long videos containing temporal redundancy into a more compact form by rearranging the active objects in the video. However, since each object in the video belongs to a different tube, moving the objects during the tube rearrangement process may disrupt the interaction relationship between the objects. To address this issue, we proposed a video synopsis optimization method based on interaction determination using human pose estimation. First, the inverse perspective mapping method is used to calculate the distance between objects from a bird’s-eye view. Then the body orientation is determined using the coordinates of human skeletal points obtained from the human pose estimation model. These two pieces of information are then combined to determine the interactions between objects. Second, the interaction energy cost term is introduced. By penalizing the cost term, the interactions between objects in the original video are preserved in the synopsis video. Finally, we propose a fusion optimization algorithm (FSOSA) that combines snake optimization and simulated annealing. FSOSA aims to minimize the energy cost function and achieve optimal tube rearrangement. Experimental results demonstrate that the proposed method effectively preserves the interactions between objects while improving the convergence speed.

Palm vein biometric technology is widely regarded as highly secure due to its challenging-to-forge characteristics. However, recent empirical studies have revealed that forged vein patterns printed on paper can deceive palm vein recognition systems, thereby leading to security breaches. The conventional approach to address this issue involves performing liveness detection followed by preprocessing the palm vein image prior to recognition, which increases the algorithmic complexity and might adversely affect overall performance. To overcome these limitations, we propose a multibranch network (PVCodeNet++) for end-to-end integration of palm vein recognition and liveness detection using a multitask learning approach. Specifically, our proposed model leverages network weight sharing and mutual assistance between network branches to enhance overall performance. We utilize the transformer encoder as the underlying shared component, employ central difference convolution for the liveness detection branch, introduce the normalized attention mechanism, and balance the multitask loss through the uncertainty weighting method. Experiments on palm vein liveness and spoofing datasets show that the proposed PVCodeNet++ has an equal error rate of 0 for recognition performance metrics on various datasets, a significant improvement in the intraclass compactness and interclass separability separation metric, increasing from 7.88 to 9.37 on the PolyU dataset; and an average classification error rate of 0 for liveness detection performance metrics, demonstrating the feasibility and effectiveness of the method proposed.

Image-to-image translation with unaligned domains is a seminal work that is capable of automatically selecting suitable image of unaligned dataset for translation. Unaligned datasets often contain irrelevant images that disrupt translation. However, the accuracy of selecting appropriate images is an issue that needs to be urgently improved. To improve the grasp of image features and enhance the performance of unaligned image-to-image translation without changing much semantic information, we propose a novel framework unaligned image-to-image translation model based on contrastive learning and generative adversarial networks (UCGAN), which can focus on both local and global features of the image. Inspired by contrastive learning, here we propose the patch-by-patch feature contrastive (PPFC). PPFC reuses the encoder of the generator to extract multi-layer features of different image patches. We then normalize the various features using L1-norm instead of L2-norm. To further improve translation performance, we also set up paired feature comparisons. We compare our proposed method UCGAN with state-of-the-art algorithms and the experimental results have demonstrated the superiority and practicality of our method.

The quality of the images captured by the camera deteriorates in hazy weather, which affects the effectiveness of subsequent high-level vision applications. Extensive methods have been proposed to remove haze and restore clear and haze-free images. However, these methods mainly focus on the accuracy of the model, while ignoring the computational complexity and inference speed that make it difficult to deploy on resource-constrained devices. To address the aforementioned problems, we propose a simple but effective single-image dehazing network. The network is based on the classical U-Net architecture. First, the encoder uses a down-sampling operation to extract shallow hierarchical features and reduce the dimension of the features. Then, the degraded features are gradually restored through the cascaded feature recovery module. Finally, the deep and shallow features are fused through the decoder to obtain the recovered images. The proposed method leverages the stacked larger kernel convolutions to enhance the local and global feature learning capability, and better cope with the severe degradation of image quality in dense and non-homogeneous haze weather through the hybrid attention mechanism.