The latest high efficiency video coding (HEVC) standard significantly increases the encoding complexity for improving its coding efficiency. Due to the limited computational capability of handheld devices, complexity constrained video coding has drawn great attention in recent years. A complexity control algorithm based on adaptive mode selection is proposed for interframe coding in HEVC. Considering the direct proportionality between encoding time and computational complexity, the computational complexity is measured in terms of encoding time. First, complexity is mapped to a target in terms of prediction modes. Then, an adaptive mode selection algorithm is proposed for the mode decision process. Specifically, the optimal mode combination scheme that is chosen through offline statistics is developed at low complexity. If the complexity budget has not been used up, an adaptive mode sorting method is employed to further improve coding efficiency. The experimental results show that the proposed algorithm achieves a very large complexity control range (as low as 10%) for the HEVC encoder while maintaining good rate-distortion performance. For the lowdelayP condition, compared with the direct resource allocation method and the state-of-the-art method, an average gain of 0.63 and 0.17 dB in BDPSNR is observed for 18 sequences when the target complexity is around 40%.

Video anomaly event detection is the process of finding an abnormal event deviation compared with the majority of normal or usual events. The main challenges are the high structure redundancy and the dynamic changes in the scenes that are in surveillance videos. To address these problems, we present a framework for anomaly detection and localization in videos that is based on locality-constrained affine subspace coding (LASC) and a model updating procedure. In our algorithm, LASC attempts to reconstruct the test sample by its top-k nearest subspaces, which are obtained by segmenting the normal samples space using a clustering method. A sample with a large reconstruction cost is detected as abnormal by setting a threshold. To adapt to the scene changes over time, a model updating strategy is proposed. We experiment on two public datasets: the UCSD dataset and the Avenue dataset. The results demonstrate that our method achieves competitive performance at a 700 fps on a single desktop PC.

A framing camera based on the pulse-dilation technology is simulated by using the Monte Carlo method. The camera uses pulse dilation of an electron signal from a pulsed photocathode (PC) to achieve high temporal resolution. When the PC is not pulsed, the theoretical temporal resolution of the camera without pulse dilation is about 90 ps. When the excitation pulse is applied on the PC, the theoretical temporal resolution is improved to 1.9 ps using the pulse-dilation technology. The relationship between the temporal resolution and the PC bias voltage is obtained. The variation of the temporal resolution with the gradient of the PC excitation pulse is also provided. Furthermore, the connection between the temporal resolution and the drift length is acquired.

In texture-plus-depth format of three-dimensional visual data, both texture and depth maps are required to synthesize a desired view via depth-image-based rendering. However, the depth maps captured or estimated always exist with low resolution compared to their corresponding texture images. We introduce a joint edge-guided convolutional neural network that upsamples the resolution of a depth map on the premise of synthesized view quality. The network takes the low-resolution depth map as an input using a joint edge feature extracted from the depth map and the registered texture image as a reference, and then produces a high-resolution depth map. We further use local constraints that preserve smooth regions and sharp edges so as to improve the quality of the depth map and synthesized view. Finally, a global looping optimization is performed with virtual view quality as guidance in the recovery process. We train our model using pairs of depth maps and texture images and then make tests on other depth maps and video sequences. The experimental results demonstrate that our scheme outperforms existing methods both in the quality of the depth maps and synthesized views.

We tackle the problem of natural image segmentation by proposing a statistical approach that is based on spatially variant finite mixture models with generalized means. The contributions can be summarized as follows: first, the proposed spatially variant mixture model exploits beta-Liouville as basic distributions for describing the underlying data structure, which demonstrated better segmentation performance than commonly used distributions, such as Gaussian; second, the mixing proportions (i.e., the probabilities of class labels) in our model are modeled via the Dirichlet compound multinomial probability density, and the spatial smoothness is imposed by adopting the function of generalized mean over the mixture model as well as mixing proportions; and finally, a variational Bayes learning approach is developed to estimate model parameters and model complexity simultaneously with closed-form solutions. The robustness, accuracy, and effectiveness of the proposed model in image segmentation are demonstrated through experiments on both natural images and synthetic images degraded by noise compared with other state-of-the-art image segmentation methods.

For stereo matching, it is hard to find accurate correspondence for saturated regions, such as too dark or too bright regions, because there is rarely reliable information to match. In this situation, conventional high-dynamic range (HDR) imaging techniques combining multiple exposures for each viewpoint can be adopted to generate well-exposed stereo images. This approach is, however, time-consuming and needs much memory to store multiple exposures for each viewpoint. We propose an efficient method to generate HDR multiview images as well as corresponding accurate depth maps. First, we take a single exposure for each viewpoint with alternating exposure setting, such as short and long exposure, as functions of viewpoint changes. Then, we compute an initial depth map for each view only using neighboring images that have the same exposure. To reduce the error of the initial depth maps for the saturated regions, we adopt the fusion move algorithm fusing neighboring depth maps that have different error regions. Finally, using the enhanced depth maps, we generate artifact-free and sharp HDR images using the joint bilateral filtering and a detail-transfer technique. Experimental results show that our method produces both consistent HDR images and accurate depth maps for various indoor and outdoor multiview images.

Due to advances in technology, today’s biometric systems become vulnerable to spoof attacks made by fake faces. These attacks occur when an intruder attempts to fool an established face-based recognition system by presenting a fake face (e.g., print photo or replay attacks) in front of the camera instead of the intruder’s genuine face. For this purpose, face antispoofing has become a hot topic in face analysis literature, where several applications with antispoofing task have emerged recently. We propose a solution for distinguishing between real faces and fake ones. Our approach is based on extracting features from the difference between successive frames instead of individual frames. We also used a multilevel representation that divides the frame difference into multiple multiblocks. Different texture descriptors (local binary patterns, local phase quantization, and binarized statistical image features) have then been applied to each block. After the feature extraction step, a Fisher score is applied to sort the features in ascending order according to the associated weights. Finally, a support vector machine is used to differentiate between real and fake faces. We tested our approach on three publicly available databases: CASIA Face Antispoofing database, Replay-Attack database, and MSU Mobile Face Spoofing database. The proposed approach outperforms the other state-of-the-art methods in different media and quality metrics.

This paper proposes a joint image encryption and compression scheme based on a new hyperchaotic system and curvelet transform. A new five-dimensional hyperchaotic system based on the Rabinovich system is presented. By means of the proposed hyperchaotic system, a new pseudorandom key stream generator is constructed. The algorithm adopts diffusion and confusion structure to perform encryption, which is based on the key stream generator and the proposed hyperchaotic system. The key sequence used for image encryption is relation to plain text. By means of the second generation curvelet transform, run-length coding, and Huffman coding, the image data are compressed. The joint operation of compression and encryption in a single process is performed. The security test results indicate the proposed methods have high security and good compression effect.

An improvement in the method of automatic vehicle classification is investigated. The challenges are to correctly classify vehicles regardless of changes in illumination, differences in points of view of the camera, and variations in the types of vehicles. Our proposed appearance-based feature extraction algorithm is called linked visual words (LVWs) and is based on the existing technique bag-of-visual word (BoVW) with the addition of spatial information to improve accuracy of classification. In addition, to prevent over-fitting due to a large number of LVWs, four common sampling techniques with LVWs are investigated. Our results suggest that the sampling of LVWs using TF-IDF with grouping improved the accuracy of classification for the test dataset. In summary, the proposed system is able to classify nine types of vehicles and work with surveillance cameras in real-world scenarios. The classification accuracy of the proposed system is 5.58% and 4.27% higher on average for three datasets when compared with BoVW + SVM and Lenet-5, respectively.

Image intensity value is determined by both the albedo component and the shading component. The albedo component describes the physical nature of different objects at the surface of the earth, and land-cover classes are different from each other because of their intrinsic physical materials. We, therefore, recover the intrinsic albedo feature of the hyperspectral image to exploit the spatial semantic information. Then, we use the support vector machine (SVM) to classify the recovered intrinsic albedo hyperspectral image. The SVM tries to maximize the minimum margin to achieve good generalization performance. Experimental results show that the SVM with the intrinsic albedo feature method achieves a better classification performance than the state-of-the-art methods in terms of visual quality and three quantitative metrics.

Banknote discrimination with image recognition technology is significant in many applications. The traditional methods based on image recognition only recognize the banknote denomination without discriminating the counterfeit banknote. To solve this problem, we propose a systematical banknote discrimination approach with the dominant color and texture features. After capturing the visible and infrared images of the test banknote, we first implement the tilt correction based on the principal component analysis (PCA) algorithm. Second, we extract the dominant color feature of the visible banknote image to recognize the denomination. Third, we propose an adaptively weighted local binary pattern with “delta” tolerance algorithm to extract the texture features of the infrared banknote image. At last, we discriminate the genuine or counterfeit banknote by comparing the texture features between the test banknote and the benchmark banknote. The proposed approach is tested using 14,000 banknotes of six different denominations from Chinese yuan (CNY). The experimental results show 100% accuracy for denomination recognition and 99.92% accuracy for counterfeit banknote discrimination.

With the remarkably increasing interest directed to the security dimension, the iris recognition process is considered to stand as one of the most versatile technique critically useful for the biometric identification and authentication process. This is mainly due to every individual’s unique iris texture. A modestly conceived efficient approach relevant to the feature extraction process is proposed. In the first place, iris zigzag “collarette” is extracted from the rest of the image by means of the circular Hough transform, as it includes the most significant regions lying in the iris texture. In the second place, the linear Hough transform is used for the eyelids’ detection purpose while the median filter is applied for the eyelashes’ removal. Then, a special technique combining the richness of Gabor features and the compactness of ordinal measures is implemented for the feature extraction process, so that a discriminative feature representation for every individual can be achieved. Subsequently, the modified Hamming distance is used for the matching process. Indeed, the advanced procedure turns out to be reliable, as compared to some of the state-of-the-art approaches, with a recognition rate of 99.98%, 98.12%, and 95.02% on CASIAV1.0, CASIAV3.0, and IIT Delhi V1 iris databases, respectively.

This paper addresses the problem of high-computational complexity decoding in traditional Wyner–Ziv video coding (WZVC). The key focus is the migration of two traditionally high-computationally complex encoder algorithms, namely motion estimation and mode decision. In order to reduce the computational burden in this process, the proposed architecture adopts the partial boundary matching algorithm and four flexible types of block mode decision at the decoder. This approach does away with the need for motion estimation and mode decision at the encoder. The experimental results show that the proposed padding block-based WZVC not only decreases decoder complexity to approximately one hundredth that of the state-of-the-art DISCOVER decoding but also outperforms DISCOVER codec by up to 3 to 4 dB.

The objective function of sparse representation to be minimized mainly consists of two parts: reconstruction error and sparsity-inducing regularization (e.g., ℓ₀ norm or ℓ₁ norm). A regularization parameter allows a trade-off between the two parts. In practical applications, solutions for sparse representation models are highly sensitive to the regularization parameter. To alleviate this phenomenon and improve the discrimination of sparse representation, a reweighted ℓ₁ norm minimization-based sparse representation, named class-specific adaptive sparse representation (CSASR), is proposed. In CSASR, a class-specific weight matrix derived from observations of a class is applied to substitute the regularization parameter. The CSASR model mainly has the following two advantages: (i) the class-specific weight matrix can be calculated adaptively from observations; therefore, it is a parameter-free model and (ii) each class has a unique class-specific weight matrix, which enhances the discrimination of sparse representation. The experimental results show that the proposed algorithm achieves superior classification performance compared to traditional sparse representation-based classification algorithms.

A color filter array is placed on the image sensor of a digital camera to acquire color images. Each pixel uses only one color, since the image sensor can measure only one color per pixel. Therefore, empty pixels are filled using an interpolation process called demosaicing. The original and the interpolated pixels have different statistical characteristics. If the image is modified by manipulation or forgery, the color filter array pattern is altered. This pattern change can be a clue for image forgery detection. However, most forgery detection algorithms have the disadvantage of assuming the color filter array pattern. We present an identification method of the color filter array pattern. Initially, the local mean is eliminated to remove the background effect. Subsequently, the color difference block is constructed to emphasize the difference between the original pixel and the interpolated pixel. The variance measure of the color difference image is proposed as a means of estimating the color filter array configuration. The experimental results show that the proposed method is effective in identifying the color filter array pattern. Compared with conventional methods, our method provides superior performance.

A hybrid image allows multiple image interpretations to be modulated by the viewing distance. It can be constructed on the basis of the multiscale perceptual mechanisms of the human visual system by combining the low and high spatial frequencies of two different images. The hybrid image was introduced as an experimental tool for visual recognition study in terms of spatial frequency perception. To produce a compelling hybrid image, the original hybrid image synthesis method could only use similar shapes of source images that were aligned in the edges. If any two different images can be hybrid, it would be beneficial as a new experimental tool. In addition, there is no measure for the actual perception of spatial frequency, whether a single spatial frequency or both spatial frequencies are perceived from the hybrid stimulus. This paper describes two methods for synthesizing a hybrid image from dissimilar shape images or unaligned images; this hybrid image is known as an “edge-alignment-free hybrid image.” A noise-inserted method can be done by intentionally inserting and enhancing noises into the high-frequency image. With this method, the low-frequency blobs are covered with high-frequency noises when viewed up close. A color-inserted method uses complementary color gratings in the background of the high-frequency image to emphasize the high-frequency image when viewed up close, whereas the gratings disappear when viewed from far away. To ascertain that our approach successfully separates the spatial frequency at each viewing distance, we measured this property using our proposed assessment method. Our proposed method allows the experimenter to quantify the probability of perceiving both spatial frequencies and a single spatial frequency in a hybrid image. The experimental results confirmed that our proposed synthesis methods successfully hid the low-frequency image and emphasized the high-frequency image at a close viewing distance. At the same time, the perception of the low-frequency image was not disturbed when the image was viewed from far away.

We propose a method to enhance the color image of a low-light scene using a single sensor that simultaneously captures red, green, blue (RGB), and near-infrared (NIR) information. Typical image enhancement methods require two sensors to simultaneously capture color and NIR images. In contrast, our proposed system utilizes a single sensor but achieves accurate color image restoration. We divide the captured multispectral data into RGB and NIR information based on the spectral sensitivity of our imaging system. Using the NIR information for guidance, we reconstruct the corresponding color image based on a joint demosaicking and denoising technique. Subsequently, we restore the estimated color image iteratively using the constructed guidance image. Our experiments demonstrate the effectiveness of our method using synthetic data, and real raw data captured by our imaging system.

The main goal of image quality assessment (IQA) methods is the emulation of human perceptual image quality judgments. Therefore, the correlation between objective scores of these methods with human perceptual scores is considered as their performance metric. Human judgment of the image quality implicitly includes many factors when assessing perceptual image qualities such as aesthetics, semantics, context, and various types of visual distortions. The main idea of this paper is to use a host of features that are commonly employed in image aesthetics assessment in order to improve blind image quality assessment (BIQA) methods accuracy. We propose an approach that enriches the features of BIQA methods by integrating a host of aesthetics image features with the features of natural image statistics derived from multiple domains. The proposed features have been used for augmenting five different state-of-the-art BIQA methods, which use statistical natural scene statistics features. Experiments were performed on seven benchmark image quality databases. The experimental results showed significant improvement of the accuracy of the methods.

We consider image intrinsic structure in two aspects: foreground saliency computation and background saliency calculation. On the one hand, a segmented image is represented as a weighted undirected graph, and we calculate the foreground saliency by choosing some superpixels as ranking queries, where we apply a robust background measure to select those for accuracy. On the other hand, we compute the posterior probabilities to measure the background saliency and in terms of the probability, we construct a probability tree via multimerging on superpixels, and then apply the optimization strategy to the background saliency. We evaluate our proposed algorithm on two benchmark datasets and our algorithm yields the competitive results when compared with nine state-of-the-art algorithms in terms of five evaluation metrics.

Pedestrian path forecasting is one of the recently emerging applications in visual crowd analysis and modeling. Moreover, of the attempts proposed to date, only a few have considered that the undergoing interaction among agents is a key factor in determining their walking trends in a given scene. To this end, we propose a simple yet efficient framework for pedestrian path prediction in crowded scenes. First, we extract motion features related to the target pedestrian and its nearby neighbors. Second, we adopt an autoencoder feature-learning model to further enhance the representation of the extracted features. Finally, we utilize a Gaussian process regression model to infer the potential future trajectories of the target pedestrians given their walking history in the scene. We performed experiments using a challenging dataset, and our method yielded promising results and outperformed traditional methods proposed in the literature.

Background prior selection is the key step in current ranking-based saliency detection approaches. The existing related methods usually choose boundary regions of an image or the region with low initial saliency value in single image scale as the background. Then, the saliency map is obtained by ranking the inside similarity and correlation. However, these methods cannot handle situations in which the salient object lies on the image boundary (boundary-salient) multiple salient objects exist in a single image (multisalient). To this end, this paper proposes an adaptive background selection method by exploiting the background invariance in different image scales within distinct color spaces. Through embedding the selected background prior into multiple newly proposed ranking-based saliency methods, the superiority of the obtained background prior is strongly verified. Exhaustive experiments on four challenging datasets demonstrate that the proposed method outperforms the state-of-the-art methods in handling the boundary-salient and multisalient situations.

This paper proposes a privacy-preserving retrieval scheme for JPEG images based on local variance. Three parties are involved in the scheme: the content owner, the server, and the authorized user. The content owner encrypts JPEG images for privacy protection by jointly using permutation cipher and stream cipher, and then, the encrypted versions are uploaded to the server. With an encrypted query image provided by an authorized user, the server may extract blockwise local variances in different directions without knowing the plaintext content. After that, it can calculate the similarity between the encrypted query image and each encrypted database image by a local variance-based feature comparison mechanism. The authorized user with the encryption key can decrypt the returned encrypted images with plaintext content similar to the query image. The experimental results show that the proposed scheme not only provides effective privacy-preserving retrieval service but also ensures both format compliance and file size preservation for encrypted JPEG images.

As the newest international video compression standard, high efficiency video coding (HEVC) achieves a higher compression ratio and better video quality, compared with the previous standard, H.264/advanced video coding. However, higher compression efficiency is obtained at the cost of extraordinary computational load, which obstructs the implementation of the HEVC encoder for real-time applications and mobile devices. Intracoding is one of the high computational stages due to the flexible coding unit (CU) sizes and high density of angular prediction modes. This paper presents an intraencoding technique to speed up the process, which is composed of an early coding tree unit (CTU) depth interval prediction and an efficient CU partition method. The encoded CU depth information in the already encoded surrounding CTUs is utilized to predict the encoding CU search depth interval of the current CTU. By analyzing the textural features of CU, an early CU splitting termination is proposed to decide whether a CU should be decomposed into four lower-dimensions CUs or not. The experimental results indicate that the proposed algorithm outperforms the reference software HM16.7 by decreasing the coding time up to 53.67% with a negligible bit rate increase of 0.52%, and peak signal-to-noise ratio losses lower 0.06 dB, respectively.

Motion trajectory recognition is one of the most important means to determine the identity of a moving object. A compact and discriminative feature representation method can improve the trajectory recognition accuracy. This paper presents an efficient framework for action recognition using a three-dimensional skeleton kinematic joint model. First, we put forward a rotation-scale-translation-invariant shape descriptor based on point context (PC) and the normal vector of hypersurface to jointly characterize local motion and shape information. Meanwhile, an algorithm for extracting the key trajectory based on the confidence coefficient is proposed to reduce the randomness and computational complexity. Second, to decrease the eigenvalue decomposition time complexity, a tensor shape descriptor (TSD) based on PC that can globally capture the spatial layout and temporal order to preserve the spatial information of each frame is proposed. Then, a multilinear projection process is achieved by tensor dynamic time warping to map the TSD to a low-dimensional tensor subspace of the same size. Experimental results show that the proposed shape descriptor is effective and feasible, and the proposed approach obtains considerable performance improvement over the state-of-the-art approaches with respect to accuracy on a public action dataset.

This paper proposes a robust method based on a local intensity binary pattern for interest feature description. To avoid estimating reference orientation, local features were calculated on a rotation invariant system. Different from the local binary patterns (LBP) and center-symmetric-LBP operator, our proposed local circular contrast pattern (LCCP) operator calculates a local binary feature by comparing the relative intensity order information of each two adjacent elements in the sequence consisting of the sampling point and its neighboring points. To evaluate the performance of our proposed descriptor LCCP and other existing descriptors (e.g., scale-invariant feature transform, DAISY, HRI-CSLTP, multisupport region order-based gradient histogram-single, local intensity order pattern), image matching experiments were first conducted on the Oxford dataset, additional image pairs with complex light changes, image sequences with different noise, and three-dimensional objects dataset. To further evaluate the discriminative ability of local descriptors, a simple object recognition experiment was carried out on three public datasets. The experimental results show that our descriptor LCCP exhibits a better performance and robustness than other evaluated descriptors.

In movies and TV shows, it is common that several scenes repeat alternately. These videos are characterized with the long-term temporal correlation, which can be exploited to improve video coding efficiency. However, in applications supporting random access (RA), a video is typically divided into a number of RA segments (RASs) by RA points (RAPs), and different RASs are coded independently. In such a way, the long-term temporal correlation among RASs with similar scenes cannot be used. We present a scene-library-based video coding scheme for the coding of videos with repeated scenes. First, a compact scene library is built by clustering similar scenes and extracting representative frames in encoding video. Then, the video is coded using a layered scene-library-based coding structure, in which the library frames serve as long-term reference frames. The scene library is not cleared by RAPs so that the long-term temporal correlation between RASs from similar scenes can be exploited. Furthermore, the RAP frames are coded as interframes by only referencing library frames so as to improve coding efficiency while maintaining RA property. Experimental results show that the coding scheme can achieve significant coding gain over state-of-the-art methods.