This PDF file contains the editorial “2016 List of Reviewers” for JEI Vol. 26 Issue 01

A multisensor image fusion approach established on the hybrid-domain image enhancement and double nonsubsampled contourlet transform (NSCT) is proposed. The hybrid-domain pre-enhancement algorithm can promote the contrast of the visible color image. Different fusion rules are, respectively, selected and applied to obtain fusion results. The double NSCT framework is introduced to obtain better fusion performance than the general single NSCT framework. Experimental outcomes in fused images and performance results demonstrate that the presented approach is apparently more advantageous.

This PDF file contains the editorial “Special Section Guest Editorial:Image Processing for Cultural Heritage” for JEI Vol. 26 Issue 01

The existing methods—such as sieving, microscope, light scattering, sedimentation, and electrical induction for pigment size detection—require sampling or scattering the mineral pigments, which will inevitably cause damage to the films painted by mineral pigments. A new detection method based on run length texture analysis is proposed to nondestructively detect the pigment size in the mineral paint film. The films painted by mineral pigments with preknown pigment sizes are contactlessly captured by CCD microscope under diffused light. Gray transform, histogram equalization, and median filtering are implemented to preprocess the captured images, and then the run length texture parameters are extracted from the preprocessed images. A parametric relationship between the extracted parameters and the preknown size is established to predict the pigment size in mineral paint film nondestructively. Burnt carnelian is selected as the sample to verify the feasibility of the proposed method. Results show that the max detection error of the proposed method is 5.548  μm and can be applied to the size detection of the mineral pigments used in mineral paint film.

The focus of this study was to investigate the potential of hyperspectral imaging (HSI) in the monitoring of commercial consolidant products applied on wood samples. Poplar and walnut were chosen for the consolidant application. Both traditional and innovative products were selected, based on acrylic, epoxy, and aliphatic compounds. Wood samples were stressed by freeze/thaw cycles in order to cause material degradation without the loss of wood components. Then the consolidant was applied under vacuum. The samples were finally artificially aged for 168 h in a solar box chamber. The samples were acquired in the short wave infrared (1000 to 2500 nm) range by SISUChema XL™ device (Specim, Finland) after 168 h of irradiation. As comparison, color measurement was also used as an economic, simple, and noninvasive technique to evaluate the deterioration and consolidation effects on wood. All data were then processed adopting a chemometric approach finalized to define correlation models, HSI based, between consolidating materials, wood species, and short-time aging effects.

In recent years, agencies around the world have invested huge amounts of effort toward digitizing many aspects of the world’s cultural heritage. Of particular importance is the digitization of outdoor archaeological sites. In the spirit of valorization of this digital information, many groups have developed virtual or augmented reality (AR) computer applications themed around a particular archaeological object. The problem of pose tracking in outdoor AR applications is addressed. Different positional systems are analyzed, resulting in the selection of a monocular camera-based user tracker. The limitations that challenge this technique from map generation, scale, anchoring, to lighting conditions are analyzed and systematically addressed. Finally, as a case study, our pose tracking system is implemented within an AR experience in the Byblos Roman theater in Lebanon.

The effectiveness of the convolutional neural network (CNN) has already been demonstrated in many challenging tasks of computer vision, such as image retrieval, action recognition, and object classification. This paper specifically exploits CNN to design local descriptors for content-based retrieval of complete or nearly complete three-dimensional (3-D) vessel replicas. Based on vector quantization, the designed descriptors are clustered to form a shape vocabulary. Then, each 3-D object is associated to a set of clusters (words) in that vocabulary. Finally, a weighted vector counting the occurrences of every word is computed. The reported experimental results on the 3-D pottery benchmark show the superior performance of the proposed method.

Generating automatically three-dimensional (3-D) models of heritage monuments such as medieval buildings is a very challenging task, particularly when only archival documents of that buildings remain. This paper presents a set of algorithms dedicated to the 3-D modeling of historical buildings from a collection of old architecture plans, including floor plans, elevations, and cutoffs. Image processing algorithms help to detect and localize main structures of the building from the floor plans (thick and thin walls, openings). The extrusion of the walls allow us to build a first 3-D model. We compute height informations and add textures to the model by analyzing the elevation images from the same collection of documents. We applied this pipeline to 18th century plans of the Château de Versailles and show results for two different parts of the Château.

In recent years, we have witnessed a huge diffusion of building information modeling (BIM) approaches in the field of architectural design, although very little research has been undertaken to explore the value, criticalities, and advantages attributable to the application of these methodologies in the cultural heritage domain. Furthermore, the last developments in digital photogrammetry lead to the easy generation of reliable low-cost three-dimensional textured models that could be used in BIM platforms to create semantic-aware objects that could compose a specific library of historical architectural elements. In this case, the transfer between the point cloud and its corresponding parametric model is not so trivial and the level of geometrical abstraction could not be suitable with the scope of the BIM. The aim of this paper is to explore and retrace the milestone works on this crucial topic in order to identify the unsolved issues and to propose and test a unique and simple workflow practitioner centered and based on the use of the latest available solutions for point cloud managing into commercial BIM platforms.

We propose a method to extract special objects in images of medieval books, which generally represent, for example, figures and capital letters. Instead of working on the single-pixel level, we consider superpixels as the basic classification units for improved time efficiency. More specifically, we classify superpixels into different categories/objects by using a bag-of-features approach, where a superpixel category classifier is trained with the local features of the superpixels of the training images. With the trained classifier, we are able to assign the category labels to the superpixels of a historical document image under test. Finally, special objects can easily be identified and extracted after analyzing the categorization results. Experimental results demonstrate that, as compared to the state-of-the-art algorithms, our method provides comparable performance for some historical books but greatly outperforms them in terms of generality and computational time.

During the past years, research has focused on the reconstruction of three-dimensional point cloud models from unordered and uncalibrated sets of images. Most of the proposed solutions rely on the structure-from-motion algorithm, and their performances significantly degrade whenever exchangeable image file format information about focal lengths is missing or corrupted. We propose a preprocessing strategy that permits estimating the focal lengths of a camera more accurately. The basic idea is to cluster the input images into separate subsets according to an array of interpolation-related multimedia forensic clues. This operation permits having a more robust estimate and improving the accuracy of the final model.

With advances in technology, a large part of our cultural heritage is becoming digitally available. In particular, in the field of historical document image analysis, there is now a growing need for indexing and data mining tools, thus allowing us to spot and retrieve the occurrences of an object of interest, called a pattern, in a large database of document images. Patterns may present some variability in terms of color, shape, or context, making the spotting of patterns a challenging task. Pattern spotting is a relatively new field of research, still hampered by the lack of available annotated resources. We present a new publicly available dataset named DocExplore dedicated to spotting patterns in historical document images. The dataset contains 1500 images and 1464 queries, and allows the evaluation of two tasks: image retrieval and pattern localization. A standardized benchmark protocol along with ad hoc metrics is provided for a fair comparison of the submitted approaches. We also provide some first results obtained with our baseline system on this new dataset, which show that there is room for improvement and that should encourage researchers of the document image analysis community to design new systems and submit improved results.

Due to their specific characteristics, palm leaf manuscripts provide new challenges for text line segmentation tasks in document analysis. We investigated the performance of six text line segmentation methods by conducting comparative experimental studies for the collection of palm leaf manuscript images. The image corpus used in this study comes from the sample images of palm leaf manuscripts of three different Southeast Asian scripts: Balinese script from Bali and Sundanese script from West Java, both from Indonesia, and Khmer script from Cambodia. For the experiments, four text line segmentation methods that work on binary images are tested: the adaptive partial projection line segmentation approach, the A* path planning approach, the shredding method, and our proposed energy function for shredding method. Two other methods that can be directly applied on grayscale images are also investigated: the adaptive local connectivity map method and the seam carving-based method. The evaluation criteria and tool provided by ICDAR2013 Handwriting Segmentation Contest were used in this experiment.

One of the most critical aspects of documenting distinct sites is acquiring detailed and accurate range information. Several three-dimensional (3-D) acquisition techniques are available, but each has its own limitations. This paper presents a range data fusion method with the aim to enhance the descriptive contents of the entire 3-D reconstructed model. A kernel function is introduced for supervised classification of the range data using a kernelized support vector machine. The classification method is based on the local saliency features of the acquired range data. The range data acquired from heterogeneous range sensors are transformed into a defined common reference frame. Based on the segmentation criterion, the fusion of range data is performed by integrating finer regions of range data acquired from a laser range scanner with the coarser region of Kinect’s range data. After fusion, the Delaunay triangulation algorithm is applied to generate the highly accurate, realistic 3-D model of the scene. Finally, experimental results show the robustness of the proposed approach.

A system designed and developed for the three-dimensional (3-D) reconstruction of cultural heritage (CH) assets is presented. Two basic approaches are presented. The first one, resulting in an “approximate” 3-D model, uses images retrieved in online multimedia collections; it employs a clustering-based technique to perform content-based filtering and eliminate outliers that significantly reduce the performance of 3-D reconstruction frameworks. The second one is based on input image data acquired through terrestrial laser scanning, as well as close range and airborne photogrammetry; it follows a sophisticated multistep strategy, which leads to a “precise” 3-D model. Furthermore, the concept of change history maps is proposed to address the computational limitations involved in four-dimensional (4-D) modeling, i.e., capturing 3-D models of a CH landmark or site at different time instances. The system also comprises a presentation viewer, which manages the display of the multifaceted CH content collected and created. The described methods have been successfully applied and evaluated in challenging real-world scenarios, including the 4-D reconstruction of the historic Market Square of the German city of Calw in the context of the 4-D-CH-World EU project.

This paper describes a computational approach for analyzing and visualizing the aesthetics of color from the perspective of color theory. Our study is grounded in the works of Johannes Itten, one of the most remarkable theorists of color aesthetics. Our focus lies on the computational analysis of a specific aspect of color usage in paintings, namely modulation. We, therefore, propose the three-dimensional (3-D) color palette, a visualization of the chromatic information of an image in the hue-saturation-lightness space. Using the proposed palette, we derive a set of simple hue-specific descriptors for color modulation. Our experimental results involve a selection of digital reproductions of paintings discussed extensively by Itten. They show that the proposed modulation measures yield results that are consistent with Itten’s comments and explanations. Future work involves further exploration of the proposed 3-D color palette, in terms of its ability to discriminate between different artists and painting styles.

The Morgantina silver treasure belonging to the Archaeological Museum of Aidone (Sicily) was involved in a three-dimensional (3-D) survey and diagnostics campaign for monitoring the collection over time in anticipation of their temporary transfer to the Metropolitan Museum of Art in New York for a period of 4 years. Using a multidisciplinary approach, a scientific and methodological protocol based on noninvasive techniques to achieve a complete and integrated knowledge of the precious items and their conservation state, as well as to increase their valorization, has been developed. All acquired data, i.e., 3-D models, ultraviolet fluorescence, x-ray images, and chemical information, will be made available, in an integrated way, within a web-oriented platform, which will present an in-progress tool to deepen existing archaeological knowledge and production technologies and to obtain referenced information of the conservation state before and after moving of the collection from its exposure site.

We propose a convolutional neural network to classify images of buildings using sparse features at the network’s input in conjunction with primary color pixel values. As a result, a trained neuronal model is obtained to classify Mexican buildings in three classes according to the architectural styles: prehispanic, colonial, and modern with an accuracy of 88.01%. The problem of poor information in a training dataset is faced due to the unequal availability of cultural material. We propose a data augmentation and oversampling method to solve this problem. The results are encouraging and allow for prefiltering of the content in the search tasks.

A method is presented for increasing the spatial resolution of the three-dimensional (3-D) digital representation of coins by combining fine photometric detail derived from a set of photographic images with accurate geometric data from a 3-D laser scanner. 3-D reconstructions were made of the obverse and reverse sides of two ancient Roman denarii by processing sets of images captured under directional lighting in an illumination dome. Surface normal vectors were calculated by a “bounded regression” technique, excluding both shadow and specular components of reflection from the metallic surface. Because of the known difficulty in achieving geometric accuracy when integrating photometric normals to produce a digital elevation model, the low spatial frequencies were replaced by those derived from the point cloud produced by a 3-D laser scanner. The two datasets were scaled and registered by matching the outlines and correlating the surface gradients. The final result was a realistic rendering of the coins at a spatial resolution of 75  pixels/mm (13-μm spacing), in which the fine detail modulated the underlying geometric form of the surface relief. The method opens the way to obtain high quality 3-D representations of coins in collections to enable interactive online viewing.

We propose a method to find characteristic landmarks and recognize ancient Roman imperial coins using deep convolutional neural networks (CNNs) combined with expert-designed domain hierarchies. We first propose a framework to recognize Roman coins that exploits the hierarchical knowledge structure embedded in the coin domain, which we combine with the CNN-based category classifiers. We next formulate an optimization problem to discover class-specific salient coin regions. Analysis of discovered salient regions confirms that they are largely consistent with human expert annotations. Experimental results show that the proposed framework is able to effectively recognize ancient Roman coins as well as successfully identify landmarks on the coins. For this research, we have collected a Roman coin dataset where all coins are annotated and consist of obverse (head) and reverse (tail) images.

The rapid growth of image digitization and collections in recent years makes it challenging and burdensome to organize, categorize, and retrieve similar images from voluminous collections. Content-based image retrieval (CBIR) is immensely convenient in this context. A considerable number of local feature detectors and descriptors are present in the literature of CBIR. We propose a model to anticipate the best feature combinations for image retrieval-related applications. Several spatial complementarity criteria of local feature detectors are analyzed and then engaged in a regression framework to find the optimal combination of detectors for a given dataset and are better adapted for each given image; the proposed model is also useful to optimally fix some other parameters, such as the k in k-nearest neighbor retrieval. Three public datasets of various contents and sizes are employed to evaluate the proposal, which is legitimized by improving the quality of retrieval notably facing classical approaches. Finally, the proposed image search engine is applied to the cultural photographic collections of a French museum, where it demonstrates its added value for the exploration and promotion of these contents at different levels from their archiving up to their exhibition in or ex situ.

Cultural heritage digitization has been of research interest for several decades. For such, the quality of the stored images should be pleasant to see. However, as images captured by digital devices may include undesirable effects, conducting an enhancement on the image is essential. In this context, we present a framework for the purpose of cultural heritage image illumination enhancement. First, a mapping curve based on saturation feedback is created to adjust the contrast. Then illumination is enhanced by applying a modified homomorphic filter in the frequency domain. The technique employs an optimization search process based on the efficient golden section search algorithm to compute the optimal parameters to produce the enhanced image. Finally, a color restoration function is applied to overcome the problem of color violation. The resulted image represents a trade-off among local contrast improvement, detail enhancement, and preserving the naturalness of the image. Experiments are conducted on a collected dataset of cultural heritage images and compared to some of the state-of-the-art image enhancement methods using a set of quantitative assessments criteria. Results have shown that our proposed approach is able to accomplish a wide set of the performance goals.

Digital restoration of film content that has historical value is crucial for the preservation of cultural heritage. Also, digital restoration is not only a relevant application area of various video processing technologies that have been developed in computer graphics literature but also involves a multitude of unresolved research challenges. Currently, the digital restoration workflow is highly labor intensive and often heavily relies on expert knowledge. We revisit some key steps of this workflow and propose semiautomatic methods for performing them. To do that we build upon state-of-the-art video processing techniques by adding the components necessary for enabling (i) restoration of chemically degraded colors of the film stock, (ii) removal of excessive film grain through spatiotemporal filtering, and (iii) contrast recovery by transferring contrast from the negative film stock to the positive. We show that when applied individually our tools produce compelling results and when applied in concert significantly improve the degraded input content. Building on a conceptual framework of film restoration ensures the best possible combination of tools and use of available materials.

The study of cultural heritage objects with embellished realistic and abstract designs made up of connected and intertwined curves crosscuts a number of related disciplines, including archaeology, art history, and heritage management. However, many objects, such as pottery sherds found in the archaeological record, are fragmentary, making the underlying complete designs unknowable at the scale of the sherd fragment. The challenge to reconstruct and study complete designs is stymied because (1) most fragmentary cultural heritage objects contain only a small portion of the underlying full design, (2) in the case of a stamping application, the same design may be applied multiple times with spatial overlap on one object, and (3) curve patterns detected on an object are usually incomplete and noisy. As a result, traditional curve-pattern matching algorithms, such as Chamfer matching, may perform poorly in identifying the underlying design. We develop a new partial-to-global curve matching algorithm to address these challenges and better identify the full design from a fragmented cultural heritage object. Specifically, we develop the algorithm to identify the designs of the carved wooden paddles of the Southeastern Woodlands from unearthed pottery sherds. A set of pottery sherds, curated at Georgia Southern University, are used to test the proposed algorithm, with promising results.

The reconstruction of broken objects is an important field of research for many applications, such as art restoration, surgery, forensics, and solving puzzles. In archaeology, the reconstruction of broken artifacts is a very time-consuming task due to the handling of fractured objects, which are generally fragile. However, it can now be supported by three-dimensional (3-D) data acquisition devices and computer processing. Those techniques are very useful in this domain because they allow the remote handling of very accurate models of fragile parts, they permit the extensive testing of reconstruction solutions, and they provide access to the parts for the entire research community. An interesting problem has recently been proposed by archaeologists in the form of a huge puzzle composed of a thousand fragments of Pentelic marble of different sizes found in Autun (France), and all attempts to reconstruct the puzzle during the last two centuries have failed. Archaeologists are sure that some fragments are missing and that some of the ones we have come from different slabs. We propose an inexpensive transportable system for 3-D acquisition setup and a 3-D reconstruction method that is applied to this Roman inscription but is also relevant to other applications.

We present a technique for automated restoration of digital images obtained from faded photographic prints. The proposed defading technique uses our early proposed image contrast enhancement algorithm based on a contrast measure of images in the Chebyshev moment transform domain. Obtained experimental results demonstrate some advantages of the technique as compared to other widely used image enhancement methods.

This paper deals with a virtual anastylosis of a Greek Archaic statue from ancient Sicily and the development of a public outreach protocol for those with visual impairment or cognitive disabilities through the application of three-dimensional (3-D) printing and haptic technology. The case study consists of the marble head from Leontinoi in southeastern Sicily, acquired in the 18th century and later kept in the collection of the Museum of Castello Ursino in Catania, and a marble torso, retrieved in 1904 and since then displayed in the Archaeological Museum of Siracusa. Due to similar stylistic features, the two pieces can be dated to the end of the sixth century BC. Their association has been an open problem, largely debated by scholars, who have based their hypotheses on comparisons between pictures, but the reassembly of the two artifacts was never attempted. As a result the importance of such an artifact, which could be the only intact Archaic statue of a kouros ever found in Greek Sicily, has not fully been grasped by the public. Consequently, the curatorial dissemination of the knowledge related with such artifacts is purely based on photographic material. As a response to this scenario, the two objects have been 3-D scanned and virtually reassembled. The result has been shared digitally with the public via a web platform and, in order to include increased accessibility for the public with physical or cognitive disabilities, copies of the reassembled statue have been 3-D printed and an interactive test with the 3-D model has been carried out with a haptic device.

Characterization and interpretation of flat ancient material objects, such as those found in archaeology, paleoenvironments, paleontology, and cultural heritage, have remained a challenging task to perform by means of conventional x-ray tomography methods due to their anisotropic morphology and flattened geometry. To overcome the limitations of the mentioned methodologies for such samples, an imaging modality based on Compton scattering is proposed in this work. Classical x-ray tomography treats Compton scattering data as noise in the image formation process, while in Compton scattering tomography the conditions are set such that Compton data become the principal image contrasting agent. Under these conditions, we are able, first, to avoid relative rotations between the sample and the imaging setup, and second, to obtain three-dimensional data even when the object is supported by a dense material by exploiting backscattered photons. Mathematically this problem is addressed by means of a conical Radon transform and its inversion. The image formation process and object reconstruction model are presented. The feasibility of this methodology is supported by numerical simulations.

This paper outlines a low-cost, user-friendly photogrammetric technique with nonmetric cameras to obtain excavation site digital sequence images, based on photogrammetry and computer vision. Digital camera calibration, automatic aerial triangulation, image feature extraction, image sequence matching, and dense digital differential rectification are used, combined with a certain number of global control points of the excavation site, to reconstruct the high precision of measured three-dimensional (3-D) models. Using the acrobatic figurines in the Qin Shi Huang mausoleum excavation as an example, our method solves the problems of little base-to-height ratio, high inclination, unstable altitudes, and significant ground elevation changes affecting image matching. Compared to 3-D laser scanning, the 3-D color point cloud obtained by this method can maintain the same visual result and has advantages of low project cost, simple data processing, and high accuracy. Structure-from-motion (SfM) is often used to reconstruct 3-D models of large scenes and has lower accuracy if it is a reconstructed 3-D model of a small scene at close range. Results indicate that this method quickly achieves 3-D reconstruction of large archaeological sites and produces heritage site distribution of orthophotos providing a scientific basis for accurate location of cultural relics, archaeological excavations, investigation, and site protection planning. This proposed method has a comprehensive application value.

Labanotation uses symbols to describe human motion and is an effective means of protecting folk dance. We use motion capture data to automatically generate Labanotation. First, we convert the motion capture data of the biovision hierarchy file into three-dimensional coordinate data. Second, we divide human motion into element movements. Finally, we analyze each movement and find the corresponding notation. Our work has been supervised by an expert in Labanotation to ensure the correctness of the results. At present, the work deals with a subset of symbols in Labanotation that correspond to several basic movements. Labanotation contains many symbols and several new symbols may be introduced for improvement in the future. We will refine our work to handle more symbols. The automatic generation of Labanotation can greatly improve the work efficiency of documenting movements. Thus, our work will significantly contribute to the protection of folk dance and other action arts.

There is an increasing demand for high-resolution recording of in situ underwater cultural heritage. Reflectance transformation imaging (RTI) has a proven track record in terrestrial contexts for acquiring high-resolution diagnostic data at small scales. The research presented here documents the first adaptation of RTI protocols to the subaquatic environment, with a scuba-deployable method designed around affordable off-the-shelf technologies. Underwater RTI (URTI) was used to capture detail from historic shipwrecks in both the Solent and the western Mediterranean. Results show that URTI can capture submillimeter levels of qualitative diagnostic detail from in situ archaeological material. In addition, this paper presents the results of experiments to explore the impact of turbidity on URTI. For this purpose, a prototype fixed-lighting semisubmersible RTI photography dome was constructed to allow collection of data under controlled conditions. The signal-to-noise data generated reveals that the RGB channels of underwater digital images captured in progressive turbidity degraded faster than URTI object geometry calculated from them. URTI is shown to be capable of providing analytically useful object-level detail in conditions that would render ordinary underwater photography of limited use.

Point matching under illumination changes is significant for many vision information applications. However, the uneven and dramatic illumination variations model is rarely considered in existing point matching algorithms. Therefore, a method to match features efficiently under uneven and dramatic illumination changes is presented. This method extracts and describes illumination invariant interesting points from matched multibrightness layers that are obtained by a set of contrast stretching functions and prior information based on original images. Layers matching is insensitive to large unevenness of illumination changes and provides similar images in brightness and structure, so the effects of large uneven illumination changes can be reduced greatly. This algorithm is compatible with most detectors and descriptors. To accelerate the computing speed, the features from the accelerated segment test detector and the improved speeded up robust features descriptor are chosen in this paper. In addition, the combination of priority Hamming distance matching and Lowe’s matching algorithms is first proposed to increase the matching speed. This method is generic and can be used in most point matching under all varying illumination conditions. Experimental results demonstrate that the proposed method improves the quality of matched points significantly.

Recently, discriminative correlation filter (DCF)-based trackers have achieved extremely successful results in many competitions and benchmarks. These methods utilize a periodic assumption of the training samples to efficiently learn a classifier. However, this assumption will produce unwanted boundary effects, which severely degrade the tracking performance. Correlation filters with limited boundaries and spatially regularized DCFs were proposed to reduce boundary effects. However, their methods used the fixed mask or predesigned weights function, respectively, which was unsuitable for large appearance variation. We propose an accurate mask-based spatially regularized correlation filter for visual tracking. Our augmented objective can reduce the boundary effect even in large appearance variation. In our algorithm, the masking matrix is converted into the regularized function that acts on the correlation filter in frequency domain, which makes the algorithm fast convergence. Our online tracking algorithm performs favorably against state-of-the-art trackers on OTB-2015 Benchmark in terms of efficiency, accuracy, and robustness.

Concept factorization (CF), as a popular matrix factorization technique, has recently attracted increasing attention in image clustering, due to the strong ability of dimension reduction and data representation. Existing CF variants only consider the local structure of data, but ignore the global structure information embedded in data, which is very crucial for data representation. To address the above issue, we propose an improved CF method, namely local and global regularized concept factorization (LGCF), by considering the local and global structures simultaneously. Specifically, the local geometric structure is depicted in LGCF via a hypergraph, which is capable of precisely capturing high-order geometrical information. In addition, to discover the global structure, we establish an unsupervised discriminant criterion, which characterizes the between-class scatter and the total scatter of the data with the help of latent features in LGCF. For the formulated LGCF, a multiplicative update rule is developed, and the convergence is rigorously proved. Extensive experiments on several real image datasets demonstrate the superiority of the proposed method over the state-of-the-art methods in terms of clustering accuracy and mutual information.

We present a line-scan stereo system and descriptor-based dense stereo matching for high-performance vision applications. The stochastic binary local descriptor (STABLE) descriptor is a local binary descriptor that builds upon the principles of compressed sensing theory. The most important properties of STABLE are the independence of the descriptor length from the matching window size and the possibility that more than one pair of pixels contributes to a single-descriptor bit. Individual descriptor bits are computed by comparing image intensities over pairs of balanced random subsets of pixels chosen from the whole described area. On a synthetic as well as real-world examples, we demonstrate that STABLE provides competitive or superior performance than other state-of-the-art local binary descriptors in the task of dense stereo matching. The real-world example is derived from line-scan binocular stereo imaging, i.e., two line-scan cameras are observing the same object line and 2-D images are generated due to relative motion. We show that STABLE performs significantly better than the census transform and local binary patterns (LBP) in all considered geometric and radiometric distortion categories to be expected in practical applications of stereo vision. Moreover, we show as well that STABLE provides comparable or better matching quality than the binary robust-independent elementary features descriptor. The low computational complexity and flexible memory footprint make STABLE well suited for most hardware architectures. We present quantitative results based on the Middlebury stereo dataset as well as illustrative results for road surface reconstruction.

Endoscopy is an imaging procedure used for diagnosis as well as for some surgical purposes. The camera used for the endoscopy should be small and able to produce a good quality image or video, to reduce discomfort of the patients, and to increase the efficiency of the medical team. To achieve these fundamental goals, a small endoscopy camera with a footprint of 1  mm×1  mm×1.65  mm is used. Due to the physical properties of the sensors and human vision system limitations, different image-processing algorithms, such as noise reduction, demosaicking, and gamma correction, among others, are needed to faithfully reproduce the image or video. A full image-processing pipeline is implemented using a field-programmable gate array (FPGA) to accomplish a high frame rate of 60 fps with minimum processing delay. Along with this, a viewer has also been developed to display and control the image-processing pipeline. The control and data transfer are done by a USB 3.0 end point in the computer. The full developed system achieves real-time processing of the image and fits in a Xilinx Spartan-6LX150 FPGA.

High efficiency video coding (HEVC) not only improves the coding efficiency drastically compared to the well-known H.264/AVC but also introduces coding tools for parallel processing, one of which is tiles. Tile partitioning is allowed to be arbitrary in HEVC, but how to decide tile boundaries remains an open issue. An adaptive tile boundary (ATB) method is proposed to select a better tile partitioning to improve load balancing (ATB-LoadB) and coding efficiency (ATB-Gain) with a unified scheme. Experimental results show that, compared to ordinary uniform-space partitioning, the proposed ATB can save up to 17.65% of encoding times in parallel encoding scenarios and can reduce up to 0.8% of total bit rates for coding efficiency.

We formulate object tracking under the particle filter framework as a collaborative tracking problem. The priori information from training data is exploited effectively to online learn a discriminative and reconstructive dictionary, simultaneously without losing structural information. Specifically, the class label and the semantic structure information are incorporated into the dictionary learning process as the classification error term and ideal coding regularization term, respectively. Combined with the traditional reconstruction error, a unified dictionary learning framework for robust object tracking is constructed. By minimizing the unified objective function with different mixed norm constraints on sparse coefficients, two robust optimizing methods are developed to learn the high-quality dictionary and optimal classifier simultaneously. The best candidate is selected by minimizing the reconstructive error and classification error jointly. As the tracking continues, the proposed algorithms alternate between the robust sparse coding and the dictionary updating. The proposed trackers are empirically compared with 14 state-of-the-art trackers on some challenging video sequences. Both quantitative and qualitative comparisons demonstrate that the proposed algorithms perform well in terms of accuracy and robustness.

Recently, research on and applications of human fixation detection in video compressed domain have gained increasing attention. However, prediction accuracy and computational complexity still remain a challenge. This paper addresses the problem of compressed domain fixations detection in the videos based on residual discrete cosine transform coefficients norm (RDCN) and Markov random field (MRF). RDCN feature is directly extracted from the compressed video with partial decoding and is normalized. After spatial–temporal filtering, the normalized map [Smoothed RDCN (SRDCN) map] is taken to the MRF model, and the optimal binary label map is obtained. Based on the label map and the center saliency map, saliency enhancement and nonsaliency inhibition are done for the SRDCN map, and the final SRDCN-MRF salient map is obtained. Compared with the similar models, we enhance the available energy functions and introduce an energy function that indicates the positional information of the saliency. The procedure is advantageous for improving prediction accuracy and reducing computational complexity. The validation and comparison are made by several accuracy metrics on two ground truth datasets. Experimental results show that the proposed saliency detection model achieves superior performances over several state-of-the-art compressed-domain and pixel-domain algorithms on evaluation metrics. Computationally, our algorithm reduces 26% more computational complexity with comparison to similar algorithms.

A technique appropriate for extracting textual information from documents with complex layouts, such as newspapers and journals, is presented. It is a combination of a foreground analysis and a text localization method. The first one is used to segment the page in text and nontext blocks, whereas the second one is used to detect text that may be embedded inside images, charts, diagrams, tables, etc. Detailed experiments on two public databases showed that mixing layout analysis and text localization techniques can lead to improved page segmentation and text extraction results.

We present a probabilistic collaborative representation method under Bayesian framework for visual tracking. First, principal component analysis (PCA) basis vectors and squared templates are used to model the appearance of tracked object. Second, to decline the high complexity in traditional tracking methods via sparse representation, we demonstrate the mechanism of a probabilistic collaborative representation method and propose a fast method for computing the coefficients. Third, we introduce a PCA basis vectors update mechanism for the appearance change of the tracked object. Experiments on challenging videos demonstrate that our method can achieve better tracking results in terms of lower center location error and higher overlap rate.

A steganalysis feature extraction method based on Gauss partial derivative filter bank is proposed in this paper to improve the detection performance for content-adaptive JPEG steganography. Considering that the embedding changes of content-adaptive steganographic schemes are performed in the texture and edge regions, the proposed method generates filtered images comprising rich texture and edge information using Gauss partial derivative filter bank, and histograms of absolute values of filtered subimages are extracted as steganalysis features. Gauss partial derivative filter bank can represent texture and edge information in multiple orientations with less computation load than conventional methods and prevent redundancy in different filtered images. These two properties are beneficial in the extraction of low-complexity sensitive features. The results of experiments conducted on three selected modern JPEG steganographic schemes—uniform embedding distortion, JPEG universal wavelet relative distortion, and side-informed UNIWARD—indicate that the proposed feature set is superior to the prior art feature sets—discrete cosine transform residual, phase aware rich model, and Gabor filter residual.

How to transfer the trained detector into the target scenarios has been an important topic for a long time in the field of computer vision. Unfortunately, most of the existing transfer methods need to keep source samples or label target samples in the detection phase. Therefore, they are difficult to apply to real applications. For this problem, we propose a framework that consists of a controlled convolutional neural network (CCNN) and a modulating neural network (MNN). In a CCNN, the parameters of the last layer, i.e., the classifier, are dynamically adjusted by a MNN. For each target sample, the CCNN adaptively generates a proprietary classifier. Our contributions include (1) the first detector-based unsupervised transfer method that is very suitable for real applications and (2) a new scheme of a dynamically adjusting classifier in which a new object function is invented. Experimental results confirm that our method can achieve state-of-the-art results on two pedestrian datasets.

Weighted prediction (WP) is an efficient video coding tool that was introduced since the establishment of the H.264/AVC video coding standard, for compensating the temporal illumination change in motion estimation and compensation. WP parameters, including a multiplicative weight and an additive offset for each reference frame, are required to be estimated and transmitted to the decoder by slice header. These parameters cause extra bits in the coded video bitstream. High efficiency video coding (HEVC) provides WP parameter prediction to reduce the overhead. Therefore, WP parameter prediction is crucial to research works or applications, which are related to WP. Prior art has been suggested to further improve the WP parameter prediction by implicit prediction of image characteristics and derivation of parameters. By exploiting both temporal and interlayer redundancies, we propose three WP parameter prediction algorithms, enhanced implicit WP parameter, enhanced direct WP parameter derivation, and interlayer WP parameter, to further improve the coding efficiency of HEVC. Results show that our proposed algorithms can achieve up to 5.83% and 5.23% bitrate reduction compared to the conventional scalable HEVC in the base layer for SNR scalability and 2× spatial scalability, respectively.

A comprehensive radiometric characterization of raw-data format imagery acquired with the Raspberry Pi 3 and V2.1 camera module is presented. The Raspberry Pi is a high-performance single-board computer designed to educate and solve real-world problems. This small computer supports a camera module that uses a Sony IMX219 8 megapixel CMOS sensor. This paper shows that scientific and engineering-grade imagery can be produced with the Raspberry Pi 3 and its V2.1 camera module. Raw imagery is shown to be linear with exposure and gain (ISO), which is essential for scientific and engineering applications. Dark frame, noise, and exposure stability assessments along with flat fielding results, spectral response measurements, and absolute radiometric calibration results are described. This low-cost imaging sensor, when calibrated to produce scientific quality data, can be used in computer vision, biophotonics, remote sensing, astronomy, high dynamic range imaging, and security applications, to name a few.

Image representation and classification are two fundamental tasks toward version understanding. Shape and texture provide two key features for visual representation and have been widely exploited in a number of successful local descriptors, e.g., scale invariant feature transform (SIFT), local binary pattern descriptor, and histogram of oriented gradient. Unlike these gradient-based descriptors, this paper presents a simple yet efficient local descriptor, named local color contrastive descriptor (LCCD), which captures the contrastive aspects among local regions or color channels for image representation. LCCD is partly inspired by the neural science facts that color contrast plays important roles in visual perception and there exist strong linkages between color and shape. We leverage f-divergence as a robust measure to estimate the contrastive features between different spatial locations and multiple channels. Our descriptor enriches local image representation with both color and contrast information. Due to that LCCD does not explore any gradient information, individual LCCD does not yield strong performance. But we verified experimentally that LCCD can compensate strongly SIFT. Extensive experimental results on image classification show that our descriptor improves the performance of SIFT substantially by combination on three challenging benchmarks, including MIT Indoor-67 database, SUN397, and PASCAL VOC 2007.

In visual tracking, learning a robust and efficient appearance model is a challenging task. Model learning determines both the strategy and the frequency of model updating, which contains many details that could affect the tracking results. Self-paced learning (SPL) has recently been attracting considerable interest in the fields of machine learning and computer vision. SPL is inspired by the learning principle underlying the cognitive process of humans, whose learning process is generally from easier samples to more complex aspects of a task. We propose a tracking method that integrates the learning paradigm of SPL into visual tracking, so reliable samples can be automatically selected for model learning. In contrast to many existing model learning strategies in visual tracking, we discover the missing link between sample selection and model learning, which are combined into a single objective function in our approach. Sample weights and model parameters can be learned by minimizing this single objective function. Additionally, to solve the real-valued learning weight of samples, an error-tolerant self-paced function that considers the characteristics of visual tracking is proposed. We demonstrate the robustness and efficiency of our tracker on a recent tracking benchmark data set with 50 video sequences.

A new method, MICO-LDASR, is proposed to improve the classification accuracy of fused radar and optical data. The proposed algorithm combines three algorithms: multiplicative intrinsic component optimization (MICO), linear discriminant analysis (LDA), and sparse regularization (SR). MICO-LDASR first corrects the bias fields of the input images by an energy minimization process and then selects the most discriminative image features using a combination of LDA and SR (LDASR) based on a supervised feature selection and learning. Two pairs of fused radar and optical data were used in this study. Features, such as non-negative matrix factorization and textural features, were extracted from the original and bias corrected images, and, following the formation of two different types of feature matrices, the matrices were optimized based on LDASR and utilized in the two learned and unlearned forms as the inputs to rotation forest and support vector machine classifiers. The results showed that classification accuracy is greatly improved when implementing MICO-LDASR on feature matrices of Sentinel and ALOS-fused data.

Two types of scores extracted from two-dimensional (2-D) and three-dimensional (3-D) palmprint for personal recognition systems are merged, introducing a local image descriptor for 2-D palmprint-based recognition systems, named bank of binarized statistical image features (B-BSIF). The main idea of B-BSIF is that the extracted histograms from the binarized statistical image features (BSIF) code images (the results of applying the different BSIF descriptor size with the length 12) are concatenated into one to produce a large feature vector. 3-D palmprint contains the depth information of the palm surface. The self-quotient image (SQI) algorithm is applied for reconstructing illumination-invariant 3-D palmprint images. To extract discriminative Gabor features from SQI images, Gabor wavelets are defined and used. Indeed, the dimensionality reduction methods have shown their ability in biometrics systems. Given this, a principal component analysis (PCA)+linear discriminant analysis (LDA) technique is employed. For the matching process, the cosine Mahalanobis distance is applied. Extensive experiments were conducted on a 2-D and 3-D palmprint database with 10,400 range images from 260 individuals. Then, a comparison was made between the proposed algorithm and other existing methods in the literature. Results clearly show that the proposed framework provides a higher correct recognition rate. Furthermore, the best results were obtained by merging the score of B-BSIF descriptor with the score of the SQI+Gabor wavelets+PCA+LDA method, yielding an equal error rate of 0.00% and a recognition rate of rank-1=100.00%.

Intraprediction has been significantly improved in high efficiency video coding over H.264/AVC with quad-tree-based coding unit (CU) structure from size 64×64 to 8×8 and more prediction modes. However, these techniques cause a dramatic increase in computational complexity. An intracoding algorithm is proposed that consists of perceptual fast CU size decision algorithm and fast intraprediction mode decision algorithm. First, based on the visual saliency detection, an adaptive and fast CU size decision method is proposed to alleviate intraencoding complexity. Furthermore, a fast intraprediction mode decision algorithm with step halving rough mode decision method and early modes pruning algorithm is presented to selectively check the potential modes and effectively reduce the complexity of computation. Experimental results show that our proposed fast method reduces the computational complexity of the current HM to about 57% in encoding time with only 0.37% increases in BD rate. Meanwhile, the proposed fast algorithm has reasonable peak signal-to-noise ratio losses and nearly the same subjective perceptual quality.

The removal of rain streaks is critical for cameras in rainy weather conditions. Traditional single-image-based methods suffer from the defects of losing detail information and being time consuming. This study presents a single-image-based rain streak removal using multidimensional variational mode decomposition (MVMD). Rainy image is decomposed using MVMD into several subimages with specific directional and frequency characteristics. Thus, the rain streak components will be limited to one or two subimages, which are called rainy subimages. Rainy subimages are determined based on the directional characteristics. The subimages are narrow band-limited. Therefore, the rainy subimages contain few outline and detail information with other frequency and directional characteristics. Thus, the derain procedure that is performed on the rainy subimages will preserve most of the detailed information. The rain components are removed from the rainy subimages by a bilateral filter. The rain components can be successfully removed from the rainy image through the summation of the subimages without rain components and the filtered rainy subimages. The experimental results show that the proposed method has good rain removal capability while retaining most of the details with a lower running time compared with other existing methods.

The hyperchaotic sequence and the DNA sequence are utilized jointly for image encryption. A four-dimensional hyperchaotic system is used to generate a pseudorandom sequence. The main idea is to apply the hyperchaotic sequence to almost all steps of the encryption. All intensity values of an input image are converted to a serial binary digit stream, and the bitstream is scrambled globally by the hyperchaotic sequence. DNA algebraic operation and complementation are performed between the hyperchaotic sequence and the DNA sequence to obtain a robust encryption performance. The experiment results demonstrate that the encryption algorithm achieves the performance of the state-of-the-art methods in term of quality, security, and robustness against noise and cropping attack.

We aim to apply sparse autoencoder-based unsupervised feature learning to emotional semantic analysis for textile images. To tackle the problem of limited training data, we present a cross-domain feature learning scheme for emotional textile image classification using convolutional autoencoders. We further propose a correlation-analysis-based feature selection method for the weights learned by sparse autoencoders to reduce the number of features extracted from large size images. First, we randomly collect image patches on an unlabeled image dataset in the source domain and learn local features with a sparse autoencoder. We then conduct feature selection according to the correlation between different weight vectors corresponding to the autoencoder’s hidden units. We finally adopt a convolutional neural network including a pooling layer to obtain global feature activations of textile images in the target domain and send these global feature vectors into logistic regression models for emotional image classification. The cross-domain unsupervised feature learning method achieves 65% to 78% average accuracy in the cross-validation experiments corresponding to eight emotional categories and performs better than conventional methods. Feature selection can reduce the computational cost of global feature extraction by about 50% while improving classification performance.

A good image feature representation is crucial for image classification tasks. Many traditional applications have attempted to design single-modal features for image classification; however, these may have difficulty extracting sufficient information, resulting in misjudgments for various categories. Recently, researchers have focused on designing multimodal features, which have been successfully employed in many situations. However, there are still some problems in this research area, including selecting efficient features for each modality, transforming them to the subspace feature domain, and removing the heterogeneities among different modalities. We propose an end-to-end multimodal deep neural network (MDNN) framework to automate the feature selection and transformation procedures for image classification. Furthermore, inspired by Fisher’s theory of linear discriminant analysis, we improve the proposed MDNN by further proposing a multimodal multitask deep neural network (M2DNN) model. The motivation behind M2DNN is to improve the classification performance by incorporating an auxiliary discriminative constraint to the subspace representation. Experimental results on five representative datasets (NUS-WIDE, Scene-15, Texture-25, Indoor-67, and Caltech-101) demonstrate the effectiveness of the proposed MDNN and M2DNN models. In addition, experimental comparisons of the Fisher score criterion exhibit that M2DNN is more robust and has better discriminative power than other approaches.

To predict the peak signal-to-noise ratio (PSNR) quality of decoded images in fractal image coding more efficiently and accurately, an improved method is proposed. After some derivations and analyses, we find that the linear correlation coefficients between coded range blocks and their respective best-matched domain blocks can determine the dynamic range of their collage errors, which can also provide the minimum and the maximum of the accumulated collage error (ACE) of uncoded range blocks. Moreover, the dynamic range of the actual percentage of accumulated collage error (APACE), APACEmin to APACEmax, can be determined as well. When APACEmin reaches a large value, such as 90%, APACEmin to APACEmax will be limited in a small range and APACE can be computed approximately. Furthermore, with ACE and the approximate APACE, the ACE of all range blocks and the average collage error (ACER) can be obtained. Finally, with the logarithmic relationship between ACER and the PSNR quality of decoded images, the PSNR quality of decoded images can be predicted directly. Experiments show that compared with the previous similar method, the proposed method can predict the PSNR quality of decoded images more accurately and needs less computation time simultaneously.

Extracting informative statistic features is the most essential technical issue of steganalysis. Among various steganalysis methods, probability density function (PDF) and characteristic function (CF) moments are two important types of features due to the excellent ability for distinguishing the cover images from the stego ones. The two types of features are quite similar in definition. The only difference is that the PDF moments are computed in the spatial domain, while the CF moments are computed in the Fourier-transformed domain. Then, the comparison between PDF and CF moments is an interesting question of steganalysis. Several theoretical results have been derived, and CF moments are proved better than PDF moments in some cases. However, in the log prediction error wavelet subband of wavelet decomposition, some experiments show that the result is opposite and lacks a rigorous explanation. To solve this problem, a comparison result based on the rigorous proof is presented: the first-order PDF moment is proved better than the CF moment, while the second-order CF moment is better than the PDF moment. It tries to open the theoretical discussion on steganalysis and the question of finding suitable statistical features.

Many computer vision applications require finding corresponding points between images and using the corresponding points to estimate disparity. Today’s correspondence finding algorithms primarily use image features or pixel intensities common between image pairs. Some 3-D computer vision applications, however, do not produce the desired results using correspondences derived from image features or pixel intensities. Two examples are the multimodal camera rig and the center region of a coaxial camera rig. We present an image correspondence finding technique that aligns pairs of image sequences using optical flow fields. The optical flow fields provide information about the structure and motion of the scene, which are not available in still images but can be used in image alignment. We apply the technique to a dual focal length stereo camera rig consisting of a visible light—infrared camera pair and to a coaxial camera rig. We test our method on real image sequences and compare our results with the state-of-the-art multimodal and structure from motion (SfM) algorithms. Our method produces more accurate depth and scene velocity reconstruction estimates than the state-of-the-art multimodal and SfM algorithms.