This PDF file contains the front matter associated with SPIE Proceedings Volume 7744, including the Title Page, Copyright information, Table of Contents, Introduction, Sponsor Page, and the Conference Committee listing.

Subjectively-rated image databases have become increasingly popular in the evaluation of image quality measurement algorithms. Several groups recently have improved their metrics' performance in matching these databases, using particular HVS (human visual system) properties or image statistical models. However, it is difficult to know whether these improvements are due to progress towards mimicking the perceptual properties, or are due to matching some characteristics of the databases. This paper demonstrates an inherent limitation in using such databases, showing that our very simple metric, built on the contrast masking effect, is able to perform as good as many state-of-the-art metrics. It is also argued that existent databases neither contain enough images with particularly biased distortions to test the significance of single HVS property, nor cover diverse distortion types to reflect the requirement of emerging applications.

Image quality assessment (IQA) is useful in many visual processing systems but challenging to perform in line with the human perception. A great deal of recent research effort has been directed towards IQA. In order to overcome the difficulty and infeasibility of subjective tests in many situations, the aim of such effort is to assess visual quality objectively towards better alignment with the perception of the Human Visual system (HVS). In this work, we review and analyze the recent progress in the areas related to IQA, as well as giving our views whenever possible. Following the recent trends, we discuss the engineering approach in more details, explore the related aspects for feature pooling, and present a case study with machine learning.

The perception of video is different from that of image because of the motion information in video. Motion objects lead to the difference between two neighboring frames which is usually focused on. By far, most papers have contributed to image saliency but seldom to video saliency. Based on scene understanding, a new video saliency detection model with multi-features is proposed in this paper. First, background is extracted based on binary tree searching, then main features in the foreground is analyzed using a multi-scale perception model. The perception model integrates faces as a high level feature, as a supplement to other low-level features such as color, intensity and orientation. Motion saliency map is calculated using the statistic of the motion vector field. Finally, multi-feature conspicuities are merged with different weights. Compared with the gaze map from subjective experiments, the output of the multi-feature based video saliency detection model is close to gaze map.

The human figure with skin colors is always the focus of the people watching images displayed on the display devices like LCDs. The human perception of skin colors is always influenced by surrounding colors due to simultaneous color contrast effect. This paper presents a scheme for analyzing the perception shift of some representative skin colors which are arranged to be against a set of pre-selected background colors. The shift in perception is quantified by the differences of gamma parameters that characterize the GOG models of the skin color surrounded by the mid-gray color and the skin color surrounded by a specified background color, while both skin colors give the same appearance. With the results of the analysis on the gamma deviation of the reference skin colors, a scheme for correcting skin colors has been developed such that the appearance of the skin color against any background color can be very close to that of the same skin color against the mid-gray color. The results of subjective experiments indicate that the proposed scheme is feasible in removing the effect simultaneous effect.

In this paper, distortions caused by packet loss during video transmission are evaluated with respect to their perceived annoyance. In this respect, the impact of visual saliency on the level of annoyance is of particular interest, as regions and objects in a video frame are typically not of equal importance to the viewer. For this purpose, gaze patterns from a task free eye tracking experiment were utilised to identify salient regions in a number of videos. Packet loss was then introduced into the bit stream such as that the corresponding distortions appear either in a salient region or in a non-salient region. A subjective experiment was then conducted in which human observers rated the annoyance of the distortions in the videos. The outcomes show a strong tendency that distortions in a salient region are indeed perceived as much more annoying as compared to distortions in the non-salient region. The saliency of the distorted image content was further found to have a larger impact on the perceived annoyance as compared to the distortion duration. The findings of this work are considered to be of great use to improve prediction performance of video quality metrics in the context of transmission errors.

The current rate distortion optimization (RDO) coding schemes usually use the Sum of Absolute Difference (SAD) or Sum of Square Difference (SSD) as the distortion metric. However, neither SAD nor SSD correlates with the human visual system (HVS) very well. To develop a perceptual distortion based video encoder, we employ the Structural Similarity (SSIM) Index as the distortion metric and propose a SSIM based Lagrangian perceptual distortion rate optimization (PDRO) method in this paper. Furthermore, to adapt the different input sequences dynamically, we present an adaptive Lagrange multiplier selection scheme based on the properties of the input sequences. By modeling the transformed residuals with Laplace distribution, the statistical SSIM and rate models are deduced to obtain the adaptive Lagrange multiplier. Extensive experiments show that the proposed scheme can achieve better perceptual distortion rate performance and provide better visual quality than the SAD/SSD based RDO coding scheme.

Music notation has evolved to the point in which music scores can be digitalized to give composers a different dimension of music composition. Traditional method of music notation using pen and paper requires much time and effort especially during reviewing and editing of hand written music scores. On the other hand of the spectrum, computerizing the entire music composition process can potentially reduce the workload; however, previous approaches in digitizing music notation suffer from the overwhelming functions and the lack of human touch. In this paper, we designed, implemented, and evaluated a multi-touch application called Musicflow, which allows for automatic transcription of composers' music into digital music scores through one's fingertips. To facilitate natural, efficient interaction, MusicFlow supports many multi-touch gestures such as music notation, editing and playing back for reviewing. In addition, Musicflow includes a collaborative teaching tool which further enhances music education by engaging both the teacher and students actively on a multi-touch table. Our initial evaluation indicates that MusicFlow is intuitive to use and effective for music composition.

Photo clustering has been widely explored in many applications such as album management. But automatic clustering can hardly achieve satisfying performance due to the large variety of photos' content. This paper proposes a semi-automatic photo clustering scheme that attempts to improve clustering performance with users' interactions. Users can adjust the results of automatic clustering, and a set of constraints among photos are generated accordingly. A distance metric is then learned with these constraints and we can re-implement clustering with this metric. We conduct experiments on different photo albums, and experimental results have demonstrated that our approach is able to improve automatic photo clustering results, and it is better than pure manual adjustment approach by exploring distance metric learning.

Building recognition is a nontrivial task in computer vision research which can be utilized in robot localization, mobile navigation, etc. However, existing building recognition systems usually encounter the following two problems: 1) extracted low level features cannot reveal the true semantic concepts; and 2) they usually involve high dimensional data which require heavy computational costs and memory. Relevance feedback (RF), widely applied in multimedia information retrieval, is able to bridge the gap between the low level visual features and high level concepts; while dimensionality reduction methods can mitigate the high-dimensional problem. In this paper, we propose a building recognition scheme which integrates the RF and subspace learning algorithms. Experimental results undertaken on our own building database show that the newly proposed scheme appreciably enhances the recognition accuracy.

In the paper, we describe a mechanism to identify important social characters by analyzing the social structures embedded in photo collections. We first construct a weighted undirected graph from photo collections by examining the co-appearance of individuals in photos, wherein the weights of edges are measures of the social closeness of the involved individuals (vertices in the graph). Then a graph clustering algorithm that maximizes the modularity of the graph partition is applied to detect the embedded social clusters. Once the social clusters are identified, we measure individual's contribution to the formation of a social cluster to quantify the social importance of each character in the cluster. To compensate the discrepancy between the user-perceived important social characters and the algorithmically computed ones, we propose an interactive browsing scheme to enable viewers quickly identify import social characters that conform to their subjectivity. The effectiveness of the proposed mechanism is demonstrated through experiments on consumer photo collections.

As the performance-indicator of the image processing algorithms or systems, image quality assessment (IQA) has attracted the attention of many researchers. Aiming to the widely used compression standards, JPEG and JPEG2000, we propose a new no reference (NR) metric for compressed images to do IQA. This metric exploits the causes of distortion by JPEG and JPEG2000, employs the directional discrete cosine transform (DDCT) to obtain the detail and direction information of the images and incorporates with the visual perception to obtain the image quality index. Experimental results show that the proposed metric not only has outstanding performance on JPEG and JPEG2000 images, but also applicable to other types of artifacts.

An overview of existing and upcoming 3D video coding standards is given. Various different 3D video formats are available, each with individual pros and cons. The 3D video formats can be separated into two classes: video-only formats (such as stereo and multiview video) and depth-enhanced formats (such as video plus depth and multiview video plus depth). Since all these formats exist of at least two video sequences and possibly additional depth data, efficient compression is essential for the success of 3D video applications and technologies. For the video-only formats the H.264 family of coding standards already provides efficient and widely established compression algorithms: H.264/AVC simulcast, H.264/AVC stereo SEI message, and H.264/MVC. For the depth-enhanced formats standardized coding algorithms are currently being developed. New and specially adapted coding approaches are necessary, as the depth or disparity information included in these formats has significantly different characteristics than video and is not displayed directly, but used for rendering. Motivated by evolving market needs, MPEG has started an activity to develop a generic 3D video standard within the 3DVC ad-hoc group. Key features of the standard are efficient and flexible compression of depth-enhanced 3D video representations and decoupling of content creation and display requirements.

We have developed a new type of television named FTV (Free-viewpoint TV). FTV is the ultimate 3DTV that enables us to view a 3D scene by freely changing our viewpoints. We proposed the concept of FTV and constructed the world's first real-time system including the complete chain of operation from image capture to display. FTV is based on the rayspace method that represents one ray in real space with one point in the ray-space. We have developed ray capture, processing and display technologies for FTV. FTV can be carried out today in real time on a single PC or on a mobile player. We also realized FTV with free listening-point audio. The international standardization of FTV has been conducted in MPEG. The first phase of FTV was MVC (Multi-view Video Coding) and the second phase is 3DV (3D Video). MVC was completed in May 2009. The Blu-ray 3D specification has adopted MVC for compression. 3DV is a standard that targets serving a variety of 3D displays. The view generation function of FTV is used to decouple capture and display in 3DV. FDU (FTV Data Unit) is proposed as a data format for 3DV. FTU can compensate errors of the synthesized views caused by depth error.

New data formats including 2D video and the corresponding depth maps enable new video applications in which virtual views can be rendered, such as 3DTV and free-viewpoint video (FVV). Different from video frames, depth maps typically consist of homogeneous areas (with no textures) separated by sharp edges representing depth value changes such as between foreground and background. Conventional video coding techniques with transforms followed by quantization typically result in large artifacts along such sharp edges. To suppress these coding artifacts while preserving edges, we propose in this paper a novel filtering method for depth coding, joint trilateral filter. The main contribution in the proposed filter design is the utilization of edge information in the collocated video frame as well as in the depth map. The filtering weights are determined by the following three factors: a domain (spatial) filter which measures the proximity of pixel positions, and two range filters. One range filter takes into account the similarity among depth samples and the other one considers the similarity among the collocated pixels in the video frame. By replacing the deblocking filter in H.264/AVC with the proposed trilateral filter, simulation results demonstrate up to 0.8 dB gain in rendering quality at given bitrate for depth signal.

This paper presents a hypothesis that stereoscopic perception requires a short adjustment period after a scene change before it is fully effective. A compression method based on this hypothesis is proposed - instead of coding pictures from the left and right views conventionally, a view in the middle of the left and right view is coded for a limited period after a scene change. The coded middle view can be utilized in two alternative ways in rendering. First, it can be rendered as such, which causes an abrupt change from conventional monoscopic video to stereoscopic video. Second, the layered depth video (LDV) coding scheme can be used to associate depth, background texture, and background depth to the middle view, enabling view synthesis and gradual view disparity increase in rendering. Subjective experiments were conducted to evaluate and validate the presented hypothesis and compare the two rendering methods. The results indicate that when the maximum disparity between the left and right views was relatively small, the presented time-variable camera separation method was imperceptible. A compression gain, the magnitude of which depended on the scene duration, was achieved with half of the sequences having a suitable disparity for the presented coding method.

The reconstruction of a high resolution (HR) image from its low resolution (LR) counterpart is a challenging problem. The recently developed sparse representation (SR) techniques provide new solutions to this inverse problem by introducing the l1-norm sparsity prior into the super-resolution reconstruction process. In this paper, we present a new SR based image super-resolution by optimizing the objective function under an adaptive sparse domain and with the nonlocal regularization of the HR images. The adaptive sparse domain is estimated by applying principal component analysis to the grouped nonlocal similar image patches. The proposed objective function with nonlocal regularization can be efficiently solved by an iterative shrinkage algorithm. The experiments on natural images show that the proposed method can reconstruct HR images with sharp edges from degraded LR images.

We address an important issue of fully low-cost and low-complexity video encoding for use in resource limited sensors/devices. Conventional distributed video coding (DVC) does not actually meet this requirement because the acquisition of video sequences still relies on the high-cost mechanism (sampling + compression). Recently, we have proposed a distributed compressive video sensing (DCVS) framework to directly capture compressed video data called measurements, while exploiting correlations among successive frames for video reconstruction at the decoder. The core is to integrate the respective characteristics of DVC and compressive sensing (CS) to achieve CS-based single-pixel camera-compatible video encoder. At DCVS decoder, video reconstruction can be formulated as a convex unconstrained optimization problem via solving the sparse coefficients with respect to some basis functions. Nevertheless, the issue of measurement rate allocation has not been considered yet in the literature. Actually, different measurement rates should be adaptively assigned to different local regions by considering the sparsity of each region for improving reconstructed quality. This paper investigates dynamic measurement rate allocation in block-based DCVS, which can adaptively adjust measurement rates by estimating the sparsity of each block via feedback information. Simulation results have indicated the effectiveness of our scheme. It is worth noting that our goal is to develop a novel fully low-complexity video compression paradigm via the emerging compressive sensing and sparse representation technologies, and provide an alternative scheme adaptive to the environment, where raw video data is not available, instead of competing compression performances against the current compression standards (e.g., H.264/AVC) or DVC schemes which need raw data available for encoding.

Conventional wisdom in signal processing heavily relies on the concept of inner product defined in the Hilbert space. Despite the popularity of Hilbert-space formulation, we argue it is overly-structured to account for the complexity of signals arising from the real-world. Inspired by the works on fractal image decoding and nonlocal image processing, we propose to view an image as the fixed-point of some nonexpansive mapping in the metric space in this paper. Recently proposed BM3D-based denoising and nonlocal TV filtering can be viewed as the special cases of nonexpansive mappings while differ on the choice of clustering techniques. The physical interpretation of clustering-based nonexpansive mappings is that they convey organizational principles of the dynamical system underlying the signals of interest. There is an interesting analogy between phases of matters in statistical physics and types of structures in image processing. From this perspective, image reconstruction can be solved by a deterministic-annealing based global optimization approach which collectively exploits the a priori information about unknown image. The potential of this new paradigm, which we call "ollective sensing" is demonstrated on the lossy compression application where significant gain over current state-of-the-art (SPIHT) coding scheme has been achieved.

Compressive sensing encodes a signal into a relatively small number of incoherent linear measurements. In theory, the optimal incoherence is achieved by completely random measurement matrices. However, such matrices are often difficult and costly to implement in hardware realizations. Random Toeplitz and circulant matrices can be easily (or even naturally) realized in various applications. This paper introduces fast algorithms for reconstructing signals from incomplete Toeplitz and circulant measurements. Computational results are presented to show that Toeplitz and circulant matrices are not only as effective as random matrices for signal encoding, but also permit much faster decoding.

Compressive sensing (CS) reconstructs images from a small number of projections. We propose EdgeCS - edge guided CS reconstruction - to recover images of higher qualities from fewer measurements than the current state-of-the-art methods. Accurate edge information can significantly improve image recovery quality and speed, but such information is encoded in the CS measurements of an image. To take advantage of edge information in CS recovery, EdgeCS alternatively performs CS reconstruction and edge detection in a way that each benefits from the latest solution of the other. EdgeCS is fast and returns high-quality images. It exactly recovers the 256 × 256 Shepp-Logan phantom from merely 7 radial lines (or 3.03% k-space), which is impossible for most existing algorithms. It accurately reconstructs a 512 × 512 magnetic resonance image from 21% noisy samples. Moreover, it is also able to reconstruct complex-valued images. Each took about 30 seconds on an ordinary laptop. The algorithm can be easily ported to GPUs for a speedup of more than 10 folds.

New 3D video representations enable new modalities of interaction, such as haptic interaction, with 2D and 3D video for truly immersive media applications. Haptic interaction with video includes haptic structure and haptic motion for new immersive experiences. It is possible to compute haptic structure signals from 3D scene geometry or depth information. This paper introduces the concept of haptic motion, as well as new methods to compute haptic structure and motion signals for 2D video-plus-depth representation. The resulting haptic signals can be rendered using a haptic cursor attached to a 2D or 3D video display.

Three-dimensional (3-D) video provides an immersing experience for users. In recent years, many attempts have been made to capture the complex surface shape and highly detailed texture of real-world moving objects, which results in a huge amount of data. In this paper, we discuss compression issues for 3-D video. We introduce 3-D video, which is classified into two categories. Then we survey compression methods that have been investigated for each category. We present our compression methods for temporally varying mesh sequences. In addition, we show comparison results for our algorithm with respect to previous work.

Interactive media streaming is the communication paradigm where an observer periodically requests new desired subsets from the streaming sender in real-time, upon which the sender sends the appropriate media data, corresponding to the received requests, for immediate decoding and display. This is in contrast to non-interactive media streaming, e.g., TV broadcast, where the entire media set is compressed and delivered to the observer before the observer interacts with the data (such as switching TV channels). Examples of interactive streaming abound in different media modalities: interactive browsing of JPEG2000 images, interactive light field or multiview video streaming, etc. Interactive media streaming has the obvious advantage of bandwidth efficiency: only the media subsets corresponding to observer's requests are transmitted. This is important when an observer only views a small subset out of a very large media data set during a typical streaming session. The technical challenge is how to structure media data such that good compression efficiency can be achieved by exploiting correlation among media subsets (thus inducing a particular decoding order if correlation is exploited during encoding), while providing sufficient flexibility for the observer to freely navigate the media data set in his/her desired unique order. In this overview paper, we survey different proposals in the literature that simultaneously achieve the conflicting objectives of compression efficiency and decoding flexibility.

In immersive communication applications, knowing the user's viewing position can help improve the efficiency of multiview compression and streaming significantly, since often only a subset of the views are needed to synthesize the desired view(s). However, uncertainty regarding the viewer location can have negative impacts on the rendering quality. In this paper, we propose an algorithm to improve the robustness of view-dependent compression schemes by jointly performing user tracking and compression. A face tracker tracks the user's head location and sends the probability distribution of the face locations as one or many particles. The server then applies motion model to the particles and compresses the multiview video accordingly in order to improve the expected rendering quality of the viewer. Experimental results show significantly improved robustness against tracking errors.

We propose a framework for popularity-driven rate allocation in H.264/MVC-based multi-view video communications when the overall rate and the rate necessary for decoding each view are constrained in the delivery architecture. We formulate a rate allocation optimization problem that takes into account the popularity of each view among the client population and the rate-distortion characteristics of the multi-view sequence so that the performance of the system is maximized in terms of popularity-weighted average quality. We consider the cases where the global bit budget or the decoding rate of each view is constrained. We devise a simple ratevideo- quality model that accounts for the characteristics of interview prediction schemes typical of multi-view video. The video quality model is used for solving the rate allocation problem with the help of an interior point optimization method. We then show through experiments that the proposed rate allocation scheme clearly outperforms baseline solutions in terms of popularity-weighted video quality. In particular, we demonstrate that the joint knowledge of the rate-distortion characteristics of the video content, its coding dependencies, and the popularity factor of each view is key in achieving good coding performance in multi-view video systems.

In this paper, we address the joint optimization problem of rate allocation and distortion control for scalable video coding (SVC) multicast networks. Firstly, we construct video distribution meshes by coupling network coding and multipath routing with multi-rate control to seek optimal routing paths and associated transmission rates. Secondly, a particular minimum bandwidth consumption scheme is presented for each video layer to reduce video distortion, where the content priority of the base layer and the minimum bandwidth consumption at higher layers are taken into account. Finally, a convex mathematic model combining these considerations is proposed. Through decomposition and dual approach, the target convex optimization problem is solved by a fully decentralized algorithm which is decomposed into a two-level optimization procedure. Simulation results validate the convergence behavior and performance of the proposed algorithm.

P2P content delivery techniques for video transmission have become of high interest in the last years. With the involvement of client into the delivery process, P2P approaches can significantly reduce the load and cost on servers, especially for popular services. However, previous studies have already pointed out the unreliability of P2P-based live streaming approaches due to peer churn, where peers may ungracefully leave the P2P infrastructure, typically an overlay networks. Peers ungracefully leaving the system cause connection losses in the overlay, which require repair operations. During such repair operations, which typically take a few roundtrip times, no data is received from the lost connection. While taking low delay for fast-channel tune-in into account as a key feature for broadcast-like streaming applications, the P2P live streaming approach can only rely on a certain media pre-buffer during such repair operations. In this paper, multi-tree based Application Layer Multicast as a P2P overlay technique for live streaming is considered. The use of Flow Forwarding (FF), a.k.a. Retransmission, or Forward Error Correction (FEC) in combination with Scalable video Coding (SVC) for concealment during overlay repair operations is shown. Furthermore the benefits of using SVC over the use of AVC single layer transmission are presented.

Emerging IPTV technology uses source-specific IP multicast to deliver television programs to end-users. To provide reliable IPTV services over the error-prone DSL access networks, a combination of multicast forward error correction (FEC) and unicast retransmissions is employed to mitigate the impulse noises in DSL links. In existing systems, the retransmission function is provided by the Retransmission Servers sitting at the edge of the core network. In this work, we propose an alternative distributed solution where the burden of packet loss repair is partially shifted to the peer IP set-top boxes. Through Peer-Assisted Repair (PAR) protocol, we demonstrate how the packet repairs can be delivered in a timely, reliable and decentralized manner using the combination of server-peer coordination and redundancy of repairs. We also show that this distributed protocol can be seamlessly integrated with an application-layer source-aware error protection mechanism called forward and retransmitted Systematic Lossy Error Protection (SLEP/SLEPr). Simulations show that this joint PARSLEP/ SLEPr framework not only effectively mitigates the bottleneck experienced by the Retransmission Servers, thus greatly enhancing the scalability of the system, but also efficiently improves the resistance to the impulse noise.

Quality of Experience (QoE) refers to subjective criteria for evaluating multimedia content. Methods have been devised to study the design of Voice over IP systems and video codecs. In recent years, due to more abundant network bandwidth, it has become quite popular to watch video streamed over the Internet, whether by clientserver method or through a peer-to-peer (P2P) network. In this paper, we describe our experience in conducting QoE studies of P2P streaming using a chunk-level model. Instead of considering fine-grained network service impairments such as bit errors, packet losses or delays, we focus on chunk level delays. We carry out some preliminary QoE experiments on low-bit rate, low-frame rate and low-resolution video (3L-video) sequences. We apply the chunk-level model to help improve the design of the P2P streaming algorithms, and the design of video players that playback network streamed video.

Temporal fluctuation artifact is often observed in digitally compressed video. However, the fluctuation intensity cannot be correctly measured by the traditional image/video quality metric, e.g., the peak signal-to-noise ratio (PSNR), which only addresses on the quality of a single image. Although there are several metrics proposed for temporal fluctuation measurement, e.g., the sum of squared differences (SSD) and motion compensated SSD (MCSSD), these first difference based algorithms may falsely treat a smoothly continuous change of pixels as the temporal fluctuation artifact. To overcome this problem, this contribution proposes a second difference based temporal metric, named the motion estimated mean scaled absolute second difference (MEMSASD) The performance of the MEMSASD is examined using a number of video sequences with varying degrees of temporal fluctuation, which are generated by an H.264/AVC compliant codec. Compared with existing metrics such as the PSNR, the SSD and the MCSSD, the results of the proposed metric better reflect the temporal fluctuation intensity.

In this paper a frame-loss adaptive temporal pooling method for video quality assessment is proposed. Extensive subjective tests have been carried out to determine the duration of successive frames based on which steady quality judgment can be made by human observers. The resulting duration is applied to the determination of the length of Group of Frames (GOF), where a flexible algorithm is used to separate the input video into variable sized GOFs. Short-term temporal pooling is first performed for each of the GOF to get the GOF quality, where quality contribution of each frame is incorporated with the context and frame loss well taken into account. The video quality is then obtained by long-term temporal pooling of the GOF quality considering the fact that perceptual video quality is predominately determined by the worst parts of the video. Extensive experimental results have demonstrated the effectiveness of the proposed method both for regular and irregular frame loss.

3D Video system based on Depth-Image-Based Rendering relies on high quality depth data. Errors distributed randomly in depth map sequences induce annoying temporal noise, such as flickering and object shifting. Prior studies on video quality assessment focused mainly on spatial quality of the tested sequence and often ignored its temporal performance. In synthesized sequences, a large number of tiny geometric distortions and illumination differences are temporally constant and perceptually invisible. The dynamic noise impairs subjective quality of the sequences more greatly than the static spatial noise. Temporal quality plays a dominant role in overall quality assessment on the synthesized sequences with temporal instability problem. We propose a simple full-reference metric, Peak Signal to Perceptible Temporal Noise Ratio, to evaluate quality of synthesized sequences by measuring the perceptible temporal noise in them.

In this paper, we present a new no-reference quality assessment metric for color images by using biologically inspired features (BIFs) and machine learning. In this metric, we first adopt a biologically inspired model to mimic the visual cortex and represent a color image based on BIFs which unifies color units, intensity units and C1 units. Then, in order to reduce the complexity and benefit the classification, the high dimensional features are projected to a low dimensional representation with manifold learning. Finally, a multiclass classification process is performed on this new low dimensional representation of the image and the quality assessment is based on the learned classification result in order to respect the one of the human observers. Instead of computing a final note, our method classifies the quality according to the quality scale recommended by the ITU. The preliminary results show that the developed metric can achieve good quality evaluation performance.

This paper summaries the state-of-the-art of image quality assessment (IQA) and human visual system (HVS). IQA provides an objective index or real value to measure the quality of the specified image. Since human beings are the ultimate receivers of visual information in practical applications, the most reliable IQA is to build a computational model to mimic the HVS. According to the properties and cognitive mechanism of the HVS, the available HVS-based IQA methods can be divided into two categories, i.e., bionics methods and engineering methods. This paper briefly introduces the basic theories and development histories of the above two kinds of HVS-based IQA methods. Finally, some promising research issues are pointed out in the end of the paper.

No reference (NR) method is the most difficult issue of image quality assessment (IQA), which does not need the original image or its features as reference and only depends on the statistical law of the natural images. So, the NR-IQA is a high -level evaluation for image quality and simulates the complicated subjective process of human beings. This paper presents a NR-IQA metric based on Hidden Markov Tree (HMT) model. First, the HMT is utilized to model natural images, and the statistical properties of the model parameters are analyzed to mimic variation of image degradation. Then, by estimating the deviation degree of the parameters from the statistical law the distortion metric is constructed. Experimental results show that the proposed image quality assessment model is consistent well with the subjective evaluation results, and outperforms the existing models on difference distortions.

A novel multi-order-residual (MOR) coding approach is proposed for high-bit-rate video compression in this work. We show that a significant amount of short and medium-range correlations due to the use of a fine quantization parameter cannot be easily removed by the well known motion compensated prediction process in the high-bit- rate coding scenario. Consequently, the video coding performance degrades rapidly as quality increases. To address this issue, we propose a Multi-Order-Residual (MOR) coding scheme that encodes prediction residuals in multiple stages according to their correlation characteristics. Then, we adopt different compensation and coding schemes in different stages. Experimental results are given to show the proposed MOR scheme outperforms H.264/AVC by a significant margin.

Inter prediction is an important component for video coding which exploits the temporal correlation between frames and significantly reduces the redundancy in video sequences. In this paper, we propose a predictive patch matching for inter prediction based on template matching prediction. Besides the surrounding reconstructed pixels which form the template in template matching prediction, our proposed patch matching is able to utilize the predicted pixels generated by the traditional motion prediction. A linear combination of the reconstructed template and the predicted pixels permits to synthesize a prediction while maintaining the local variations of the target block. Furthermore, a mode selection mechanism is introduced to adaptively select the predictive patch matching at sub-block level. Experimental results demonstrate the effectiveness of our proposed predictive patch matching. Constant coding gain can be achieved by our scheme at both low and high bit rates

In multiuser rate allocation in a wireless network, strategic users can bias the rate allocation by misrepresenting their bandwidth demands to a base station, leading to an unfair allocation. Game-theoretical approaches have been proposed to address the unfair allocation problems caused by the strategic users. However, existing approaches rely on a timeconsuming iterative negotiation process. Besides, they cannot completely prevent unfair allocations caused by inconsistent strategic behaviors. To address these problems, we propose a Search Based Pricing Mechanism to reduce the communication time and to capture a user's strategic behavior. Our simulation results show that the proposed method significantly reduce the communication time as well as converges stably to an optimal allocation.

This paper proposes a new approach to improve the coding performance of intra block coding in H.264/AVC via finite state machine. Grounding on high correlation between neighboring blocks, finite state machine is employed both at encoder and decoder to reduce the number of bits required for encoding to enhance coding performance. Two extra intra prediction modes are created in our proposed method. Through these two modes, the number of bits required to denote the current block is greatly reduced and low bit rate can be achieved. Experimental results show that the proposed method can greatly improve coding efficiency of intra macroblock coding in H.264/AVC.

Differential motion estimation produces dense motion vector fields which are far too demanding in terms of coding rate in order to be used in video coding. However, a pel-recursive technique like that introduced by Cafforio and Rocca can be modified in order to work using only the information available at the decoder side. This allows to improve the motion vectors produced in the classical predictive modes of H.264. In this paper we describe the modification needed in order to introduce a differential motion estimation method into the H.264 codec. Experimental results will validate a coding mode, opening new perspectives in using differential-based motion estimation techniques into classical hybrid codecs.

Distributed video coding is typically treated as a channel coding problem among others. The encoder generates parity bits (or syndrome bits) for the source and transmits part of them to the decoder for a certain target quality. The decoder tries to reconstruct the source using the received parity bits along with the side information available. In this paper, we aim to estimate the critical number of parity bits to transmit. Having observed the uncertainty of the critical rate, we model it as a random variable, use its distribution to calculate the decoding failure probability and formulate the expected distortion. We allocate a certain bit budget among different bit-planes such that the expected distortion is minimized. Moreover, we introduce fast decoding at the encoder, which helps us to estimate the critical rate far more accurately. Eventually, we achieve up to 1.5 dB gain in rate-distortion performance at high bit rates.

An important issue in Wyner-Ziv video coding is the reconstruction of Wyner-Ziv frames with decoded bit-planes. So far, there are two major approaches: the Maximum a Posteriori (MAP) reconstruction and the Minimum Mean Square Error (MMSE) reconstruction algorithms. However, these approaches do not exploit smoothness constraints in natural images. In this paper, we model a Wyner-Ziv frame by Markov random fields (MRFs), and produce reconstruction results by finding an MAP estimation of the MRF model. In the MRF model, the energy function consists of two terms: a data term, MSE distortion metric in this paper, measuring the statistical correlation between side-information and the source, and a smoothness term enforcing spatial coherence. In order to better describe the spatial constraints of images, we propose a context-adaptive smoothness term by analyzing the correspondence between the output of Slepian-Wolf decoding and successive frames available at decoders. The significance of the smoothness term varies in accordance with the spatial variation within different regions. To some extent, the proposed approach is an extension to the MAP and MMSE approaches by exploiting the intrinsic smoothness characteristic of natural images. Experimental results demonstrate a considerable performance gain compared with the MAP and MMSE approaches.

Distributed Video Coding (DVC) is a video coding paradigm which mainly exploits the source statistics at the decoder based on the availability of side information at the decoder. This paper considers feedback channel based Transform Domain Wyner-Ziv (TDWZ) DVC. The coding efficiency of TDWZ video coding does not match that of conventional video coding yet, mainly due to the quality of side information and inaccurate noise estimation. In this context, a novel TDWZ video decoder with noise residue refinement (NRR) and side information refinement (SIR) is proposed. The proposed refinement schemes are successively updating the estimated noise residue for noise modeling and side information frame quality during decoding. Experimental results show that the proposed decoder can improve the Rate- Distortion (RD) performance of a state-of-the-art Wyner-Ziv video codec for the set of test sequences.

Noisy video occurs frequently in many visual communication applications, especially in video surveillance and video phone. The noise affects compression a lot by decreasing the accuracy of prediction and increasing the residual. In this paper, we propose a novel inter-frame filtering method based on motion compensation for reference picture in video coding. It constructs the low-pass temporal filter by modulating the constructed picture with previous reference picture. As a result, it greatly reduces the noise in reference picture, which results in the improvement of compression efficiency. As the replacement of traditional reference picture, this proposed dynamical motion compensation based Synthesized Reference (SR) picture is computation efficient and easy for implementation. It can be applied to any block-based hybrid coding structure, such as H.264/AVC, AVS (Audio Video Coding Standard), MPEG-4. Simulation results show that more than 20% bit rate is reduced when applying SR picture to the encoding of typical video surveillance sequences. At the same time, the visual quality is greatly improved.

Scalable Video Coding (SVC) is an extension of H.264/AVC standard. The base layer of SVC is compatible with H.264/AVC standard, while the enhancement layers provide desired temporal, quality and/or spatial scalabilities. Bit-stream rewriting in SVC standard allows an SVC bit-stream to be converted to an H.264/AVC bit-stream without quality loss and preferably with low computational complexity. However, current rewriting is only supported in quality scalability rather than spatial scalability, which limits the application in many practical scenarios. In this paper, a hybrid bit-stream rewriting approach to support both quality and spatial scalability is proposed based on the principle of residue upsampling in transform domain. The computational complexity of the proposed approach is much lower than the conventional scheme of cascading transcoding. Extensive experimental results demonstrate that the loss of the rate-distortion (RD) performance of the proposed rewritable SVC bit-stream is acceptable compared with the conventional SVC bit-stream, however, the RD performance is better than that of simulcast. Furthermore, the RD performance of the H.264/AVC bit-stream rewritten from the rewritable SVC bit-stream is even better than that of the input SVC bit-stream. Compared with the cascading transcoding scheme, the proposed hybrid rewriting can achieve 0.8 dB Y-PSNR gains while saving 80% processing time on average.

Since the ultimate receivers of encoded video are human eyes, the characteristics of human visual system should be taken into consideration in the design of bit allocation to improve the perceptual video quality. In this paper, we incorporate the structural similarity index as a distortion metric and propose a novel rate-distortion model to characterize the relationship between rate and the structural similarity index. Based on the model, we develop an optimum bit allocation and rate control scheme for H.264 encoders. Experimental results show that up to 25% bitrate reduction over the JM reference software can be achieved. Subjective evaluation further confirms that the proposed scheme preserves more structural information and improves the perceptual quality of the encoded video.

An interpolation oriented adaptive down-sampling algorithm is proposed for low bit-rate image coding in this paper. Given an image, the proposed algorithm is able to obtain a low resolution image, from which a high quality image with the same resolution as the input image can be interpolated. Different from the traditional down-sampling algorithms, which are independent from the interpolation process, the proposed down-sampling algorithm hinges the down-sampling to the interpolation process. Consequently, the proposed down-sampling algorithm is able to maintain the original information of the input image to the largest extent. The down-sampled image is then fed into JPEG. A total variation (TV) based post processing is then applied to the decompressed low resolution image. Ultimately, the processed image is interpolated to maintain the original resolution of the input image. Experimental results verify that utilizing the downsampled image by the proposed algorithm, an interpolated image with much higher quality can be achieved. Besides, the proposed algorithm is able to achieve superior performance than JPEG for low bit rate image coding.

Line-based coding has shown its potential in improving the coding efficiency of intra-frame/image coding due to its flexibility in prediction. In our previous work we proposed an efficient line-based image coding method (LIC) by adaptive line-by-line prediction (ALP) and adaptive residue coding. In this paper we further improve this line-based coding scheme by exploiting long-distance correlations in prediction. Line-by-line template matching (LTM) is introduced to perform the long-distance prediction. Experiments in the KTA software show that the LTM scheme can effectively improve the performance of LIC at low bitrates and also brings some improvement at high bitrates. Up to 1db gain is achieved compared to previous LIC and 1.5db compared to KTA on images with regular patterns or strong edges. Up to 20% rate reduction over KTA is also achieved on some standard video sequences with different resolutions.

A Pattern-based Assembled DCT (PADCT) scheme is proposed as a general framework to fully utilize texture structure property in images in a more feasible and flexible manner. Different patterns can be designed or extracted according to texture structures such as directionality and distribution patterns. Pixels in the block are divided into sub-partitions, followed by assembling pixels in each sub-partitions together into a compact and efficient form for the succeeding separated one dimensional transform operations, suitable DC prediction and entropy coding schemes are also introduced. PADCT has the advantages of advanced compact energy concentration, efficient usage of memory, fewer categories of one dimensional transform, and adaptability for hardware optimization. Experimental results show that the bitrate reduction over the 2-D DCT is up to 35%. It also outperforms significantly than previous typical directional transform schemes in comparison with PSNR or subjective quality.

Adaptive interpolation filtering (AIF) algorithms have been proposed to enhance the hybrid video coding scheme recently. Although these algorithms can improve coding efficiency significantly, their encoders suffer huge increase in complexity in terms of latency and memory access due to its inherent multi-pass encoding procedure. In this paper, we present a novel single-pass solution for these algorithms, which allows optimal selection among different interpolation filters. In this solution, time-delayed interpolation filters are used to achieve single-pass encoding, and localized ratedistortion (RD) selection is used to compensate the possible coding loss from time-delayed filters. Experimental results show that the proposed method is efficient for All AIF techniques in current ITU-T/SG16 reference software. By using the proposed method, single-pass encoding with multiple AIF filters can be achieved while maintaining similar coding efficiency as multi-pass AIF.

One of the important issues in multiple description coding (MDC) for image and video signals is to compose pictures of the "best" quality when more than one description is received at the decoder side. To this goal, a transform-domain overlapping technique combined with some necessary translations in the transform domain is proposed in the popular MDC schemes with staggered quantizers. However, the achievable gain is rather limited. In this paper, we formulate this enhancement problem as a total variation (TV) regularized optimization constrained by the knowledge of the quantization intervals of the DCT coefficients of each composed picture. In this TV-based approach, the transformdomain overlapping technique is used to find the more accurate quantization intervals in which the true DCT coefficients will fall while receiving more than one description. Simulation results demonstrate that such a TV-based enhancement yields a higher quality gain in both objective (e.g. PSNR-based) and subjective (i.e. visual perception) evaluations than using the transform-domain overlapping technique.

We propose a new scheme for wireless video multicast based on compressed sensing. It has the property of graceful degradation and, unlike systems adhering to traditional separate coding, it does not suffer from a cliff effect. Compressed sensing is applied to generate measurements of equal importance from a video such that a receiver with a better channel will naturally have more information at hands to reconstruct the content without penalizing others. We experimentally compare different random matrices at the encoder side in terms of their performance for video transmission. We further investigate how properties of natural images can be exploited to improve the reconstruction performance by transmitting a small amount of side information. And we propose a way of exploiting inter-frame correlation by extending only the decoder. Finally we compare our results with a different scheme targeting the same problem with simulations and find competitive results for some channel configurations.

In this paper, we study the optimal bandwidth allocation for scalable video coding (SVC) streaming in multiple overlays. We model the whole bandwidth request and distribution process as a set of decentralized auction games between the competing peers. For the upstream peer, a bandwidth allocation mechanism is introduced to maximize the aggregate revenue. For the downstream peer, a dynamic bidding strategy is proposed. It achieves maximum utility and efficient resource usage by collaborating with a content-aware layer dropping/adding strategy. Also, the convergence of the proposed auction games is theoretically proved. Experimental results show that the auction strategies can adapt to dynamic join of competing peers and video layers.

One of the most challenging issues in video transmission over wireless networks is to address the rigid time bounded constraint for video delivery. We propose in this paper a deadline-aware transmission framework (DATF) for video over IEEE 802.11e EDCA wireless networks. In this new framework, we estimate the deadline for time bounded video delivery for each packet in queues at MAC layer according to the sequence number of a frame a given video data packet belonging to as well as the current network delay. Then, the MAC layer determines whether a packet should be sent or should be dropped based on the estimated deadline information. To accomplish the scheme of DATF, we propose to modify the mapping scheme in IEEE 802.11e EDCA to facilitate unequal deadline requirement of video packets. Instead of mapping all video packets into class AC_VI, which is defined for video data in IEEE 802.11e EDCA, we differentiate video packets further based on the dependency characteristics of a given frame type. The proposed DATF scheme has been implemented with NS-2 simulation based on the scenario of wired-cum-wireless network architecture. We compare the proposed approach with several competing schemes and the simulation results show that the proposed scheme outperforms these competing schemes in terms of both wireless networking metrics and received video quality.

Packet-layer video quality assessment model is a light-weight model that is useful for application scenarios like inservice video quality monitoring and network service planning, which is under standardization in ITU-T. In this paper we first differentiate the requirements for video quality assessment model from these two different application scenarios. Then different criteria and methods are analyzed and used to construct two types of test datasets for model building and evaluation. Finally, a novel packet-layer video quality assessment model dealing with video quality monitoring is proposed by incorporating the estimation of the spatiotemporal complexity of video content. The interaction between content features and the error concealment effects plus error propagation effects is considered. Experiment results demonstrate that the proposed model achieves preferable and robust performance improvement compared with the existing models in both datasets for scenarios of video quality planning and monitoring. Especially, larger Pearson correlation increase from 0.75 to 0.93 and RMSE decrease from 0.56 to 0.31 are obtained in dataset for video quality monitoring.

In this paper, we propose a unified framework targeting at two important issues in a distributed wireless camera network, i.e., object tracking and network communication, to achieve reliable multi-object tracking over distributed wireless camera networks. In the object tracking part, we propose a fully automated approach for tracking of multiple objects across multiple cameras with overlapping and non-overlapping field of views without initial training. To effectively exchange the tracking information among the distributed cameras, we proposed an idle probability based broadcasting method, iPro, which adaptively adjusts the broadcast probability to improve the broadcast effectiveness in a dense saturated camera network. Experimental results for the multi-object tracking demonstrate the promising performance of our approach on real video sequences for cameras with overlapping and non-overlapping views. The modeling and ns-2 simulation results show that iPro almost approaches the theoretical performance upper bound if cameras are within each other's transmission range. In more general scenarios, e.g., in case of hidden node problems, the simulation results show that iPro significantly outperforms standard IEEE 802.11, especially when the number of competing nodes increases.

Traditional particle filter which uses simple geometric shapes for representation cannot track objects with complex shape accurately. In this paper, we propose a refined particle filter method for contour tracking based on a binary level set model. In contrast with other previous work, the computational efficiency is greatly improved due to the simple form of the level set function. In addition, we perform curve evolution in the update step to make good use of the observation at current time. Finally, we consider some appearance information as well as the energy function to measure the weight for particles, which can identify the target more accurately. Experiment results on several challenging video sequences have verified the proposed algorithm is efficient and effective in many complicated scenes.

In this paper, we propose a novel and robust object tracking algorithm based on sparse representation. Object tracking is formulated as a object recognition problem rather than a traditional search problem. All target candidates are considered as training samples and the target template is represented as a linear combination of all training samples. The combination coefficients are obtained by solving for the minimum l1-norm solution. The final tracking result is the target candidate associated with the non-zero coefficient. Experimental results on two challenging test sequences show that the proposed method is more effective than the widely used mean shift tracker.

Vision based human motion tracking has drawn considerable interests recently because of its extensive applications. In this paper, we propose an approach to tracking the body motion of human balancing on each foot. The ability to balance properly is an important indication of neurological condition. Comparing with many other human motion tracking, there is much less occlusion in human balancing tracking. This less constrained problem allows us to combine a 2D model of human body with image analysis techniques to develop an efficient motion tracking algorithm. First we define a hierarchical 2D model consisting of six components including head, body and four limbs. Each of the four limbs involves primary component (upper arms and legs) and secondary component (lower arms and legs) respectively. In this model, we assume each of the components can be represented by quadrangles and every component is connected to one of others by a joint. By making use of inherent correlation between different components, we design a top-down updating framework and an adaptive algorithm with constraints of foreground regions for robust and efficient tracking. The approach has been tested using the balancing movement in HumanEva-I/II dataset. The average tracking time is under one second, which is much shorter than most of current schemes.

Image segmentation is very important in the field of image processing. The pulse coupled neural network (PCNN) has been efficiently applied to image processing, especially for image segmentation. In this study, a simplified PCNN (S-PCNN) model is proposed, the fuzzy mutual information (FMI) is improved as optimization criterion for S-PCNN, and then the S-PCNN and improved FMI (IFMI) based segmentation algorithm is proposed and applied for the segmentation of breast tumor in ultrasound image. To validate the proposed algorithm, a comparative experiment is implemented to segment breast images not only by our proposed algorithm, but also by the improved C-V algorithm, the max-entropy-based PCNN algorithm, the MI-based PCNN algorithm, and the IFMI-based PCNN algorithm. The results show that the breast lesions are well segmented by the proposed algorithm without image preprocessing, with the mean Hausdorff of distance of 5.631±0.822, mean average minimum Euclidean distance of 0.554±0.049, mean Tanimoto coefficient of 0.961±0.019, and mean misclassified error of 0.038±0.004. These values of evaluation indices are better than those of other segmentation algorithms. The results indicate that the proposed algorithm has excellent segmentation accuracy and strong robustness against noise, and it has the potential for breast ultrasound computer-aided diagnosis (CAD).

This paper proposes an efficient approach for unsupervised segmentation of salient objects from color images. A set of Gaussian models are first estimated based on a pre-segmentation result of the input image, and then for each pixel, a set of normalized color likelihood measures to each Gaussian model are calculated. The color saliency and spatial saliency of Gaussian models are exploited to generate the pixel-wise saliency map. By thresholding the saliency map, the pixels are classified into object seed pixels, background seed pixels and uncertain pixels to obtain the trimap. For each pixel, the probability belonging to salient object/background is evaluated using kernel density estimation, and the geodesic distances to salient object and background are calculated based on the object likelihood map. By comparing the two geodesic distances, uncertain pixels are finally classified into salient object or background. Experimental results demonstrate the better segmentation performance of the proposed approach.

Current image representation schemes have limited capability of representing 2D singularities (e.g., edges in an image). Wavelet transform has better performance in representing 1D singularities than Fourier transform. Recently invented ridgelet and curvelet transform achieve better performance in resolving 2D singularities than wavelet transform. To further improve the capability of representing 2D singularities, this paper proposes a new transform called ripplet transform Type II (ripplet-II). The new transform is able to capture 2D singularities along a family of curves in images. In fact, ridgelet transform is a special case of ripplet-II transform with degree 1. Ripplet-II transform can be used for feature extraction due to its efficiency in representing edges and textures. Experiments in texture classification and image retrieval demonstrate that the ripplet-II transform based scheme outperforms wavelet and ridgelet transform based approaches.

This paper exploits different subspace learning methods applied on silhouette based action recognition and evaluates their performance. Our recognition scheme is formed by segmenting action sequence into overlapped sub-clips and using sub-models for action matching. This sub-model matching method shows advantages in processing periodic actions. The experimental results prove that human action silhouettes are very informative for action recognition and subspace analysis can effectively preserve the intrinsic structure of raw data from 3D silhouettes. The subspace learning methods compared in this paper include traditional methods - Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA), and recently reported Orthogonal Local Preserving Projection (OLPP). PCA is observed to perform the best regarding both accuracy and efficiency. We believe our work is helpful for further research in silhouette based action recognition combined with subspace learning methods.

Scale and rotation invariant texture classification is a challenging and significant topic in texture analysis. This paper presents a scale and rotation invariant Gabor Texture Descriptor (GTD) for texture classification. Firstly, Gabor filters with different orientations and scales are carried out on an image. Secondly, to obtain a scale invariant GTD, different scale energies of these Gabor filtered images are calculated, and the optimal matching scales of the GTD are chosen by the sliding window filter with the scale energies. Thirdly, to extract a rotation invariant GTD, a new method using Discrete Fourier Transform is employed on the scale invariant GTD. At last, the distances between the GTDs of a query image and the target images are calculated for texture classification. Five different GTDs are employed for texture classification to demonstrate the effectiveness of the proposed scale and rotation invariant GTD.

In this paper we present a complete framework for scene categorization that builds upon and extends several recent ideas including spatial pyramid representation and a variety of base local descriptors which have different discriminative power and invariance from task to task. Furthermore, we propose two strategies: sum-max and max-max, used to effectively combine diverse source of data in a unified setting way. Our approach shows significantly improved performance on a large, challenging data set of fifteen natural scene categories. Owing to combination of complementary information cues, our approach is expected to equally applicable to a range of tasks.

While the causes and nature of crosstalk, as well as crosstalk reduction techniques have been extensively studied, it is still difficult to eliminate. Perceptually, crosstalk is one of the most annoying distortions in the visualization stage of stereoscopic imaging. Therefore, to understand how users perceive crosstalk is of fundamental importance to improve the quality of 3D presentations. In this paper, we aim at analyzing the impact of crosstalk level, camera baseline and scene content on users' perception of crosstalk. Extensive subjective tests are conducted and the opinion scores are statistically analyzed and discussed. The results indicate that crosstalk level, camera baseline, as well as scene content all have major impacts on the perception of crosstalk. We also show that these three factors correlate with each other in terms of impact on the crosstalk perception. Furthermore, we propose a content descriptor for crosstalk perception (CDCP) and show its effectiveness.

This paper proposes a hardware friendly multi-resolution motion estimation algorithm and VLSI architecture for high definition MPEG-like video encoder hardware implementation. By parallel searching and utilizing the high correlation in multi-resolution reference pixels, huge throughput and computation due to large search window are alleviated considerably. Sixteen way parallel processing element arrays with configurable multiplying technologies achieve fast search with regular data access and efficient data reuse. Also, the parallel arrays can be efficiently reused at three hierarchical levels for sequential motion vector refinement. The modified algorithm reaches a good balance between implementation complexity and search performance. Also, the logic circuit and on-chip SRAM consumption of the VLSI architecture are moderate.

This paper presents a perception-driven, block-based wavelet watermarking algorithm, which merits in both perceptual invisibility and robustness against watermark attacking. First, attention-based region saliency analysis is adopted to select perceptually unconcerned regions for watermark embedding. Second, wavelet transform is carried out to embed secret message into high frequency coefficients within the perceptually unconcerned regions. Especially, we present an optimal block mapping substitution method, which adopts genetic algorithm to reduce PSNR cost. Its main idea is to minimize the degradation of the watermarked image by learning a best mapping function between original image blocks and secret messages at global scope. Extensive experiments are conducted to validate both the perceptual invisibility and robustness against Gaussian Noise and JPEG compression attacks. We also give PSNR comparison with state-of-the-art methods to demonstrate our advantages.

In this paper, we propose a framework for detecting near duplicate copies of a video based on an ordinal method. The framework also incorporates a bitmap indexing structure instead of a typical indexing structure used in the previous published work. Using this method, two levels of indices are constructed. The first level of this process groups each input video (represented by their key frames) into k clusters. These clusters and the associated key frames are then used to construct the first level index. The second level of this process converts ordinal-based video signatures (generated using the technique developed in earlier work) into bitmap vectors. By adopting this two-level indexing scheme, query processing times are significantly reduced. This is because, the system is required to match only videos in the clusters that are relevant to the query and not all the videos in the database. Additionally, the technique implemented utilizes a bitmap structure for indexing, resulting in less storage space. Furthermore, we are able to employ low-cost Boolean operations such as AND, OR, and XOR in the matching process instead of Euclidean distance or other similar matching algorithms. This helps to reduce the computational time for video matching. The system has demonstrated to effectively reduce the space needed to store collections of video signatures in a database, as well as improving the overall system performance. In addition, initial results show that the system is effective and robust to several transformations such as changes in brightness, color, contrast, resolution (reduction) as well as the addition of noise.

3D reconstruction is ambiguous under so-called critical motions or critical surfaces. This paper proposes an algorithm to detect a few critical configurations where Euclidean reconstruction degenerates. Assuming that the focal lengths are the only unknown intrinsic parameters, the following critical configurations are detected: (1) coplanar 3D points, (2) pure rotation; (3) rotation around two camera centers; (4) presence of excessive noise and outliers in the measurements. The configurations in Cases (1), (2) and (4) will affect the rank of the scaled measurement matrix (SMM). The number of camera centers in Case (3) will affect the number of independent rows of the SMM. By examining the rankness and the number of independent rows of the SMM, we are able to detect the above-mentioned critical configurations. Experimental results on both synthetic and real data demonstrate the effectiveness of the proposed algorithm on detecting the critical situations for factorizationbased 3D reconstruction.

In this paper, a method for detecting infringements or modifications of a video in real-time is proposed. The method first segments a video stream into shots, after which it extracts some reference frames as keyframes. This process is performed employing a Singular Value Decomposition (SVD) technique developed in this work. Next, for each input video (represented by its keyframes), ordinal-based signature and SIFT (Scale Invariant Feature Transform) descriptors are generated. The ordinal-based method employs a two-level bitmap indexing scheme to construct the index for each video signature. The first level clusters all input keyframes into k clusters while the second level converts the ordinal-based signatures into bitmap vectors. On the other hand, the SIFT-based method directly uses the descriptors as the index. Given a suspect video (being streamed or transferred on the Internet), we generate the signature (ordinal and SIFT descriptors) then we compute similarity between its signature and those signatures in the database based on ordinal signature and SIFT descriptors separately. For similarity measure, besides the Euclidean distance, Boolean operators are also utilized during the matching process. We have tested our system by performing several experiments on 50 videos (each about 1/2 hour in duration) obtained from the TRECVID 2006 data set. For experiments set up, we refer to the conditions provided by TRECVID 2009 on "Content-based copy detection" task. In addition, we also refer to the requirements issued in the call for proposals by MPEG standard on the similar task. Initial result shows that our framework is effective and robust. As compared to our previous work, on top of the achievement we obtained by reducing the storage space and time taken in the ordinal based method, by introducing the SIFT features, we could achieve an overall accuracy in F1 measure of about 96% (improved about 8%).

Video surveillance has become one of the most important tools for public safety and security. In this paper, we present our new work on developing efficient parallel computing algorithms and schemes for implementing H.264 video encoder engine on Cell Broadband Engine (CBE) blade for high performance large scale video surveillance applications. Extending our previous work on H.264 video encoding on CBE, we have developed new parallel computing schemes in computational load partition, dynamic task scheduling, motion estimation, and mode selection. We partition the intensive H.264 video encoding computation load into four major functional modules, namely, the pre-processing module, the motion estimation module, the mode selection and transform/quantization module, and the Context Adaptive Binary Arithmetic Coding (CABAC) module. The task scheduler dynamically assigns a waiting computing task to a SPE as soon as it becomes available. Our new implementation has achieved more than 5X performance improvement to encode 32 standard-definition (SD 720x480 pixel resolution) H.264 video streams simultaneously at 30 frames per second with one Cell Blade that consists of 16 Synergistic Processor Elements (SPEs) and two control Power Processor Elements (PPEs) or 448 SD channels of H.264 video streams on a single chassis Cell Blade Center with 14 Cell Blades.

We focus on the problem of single image super-resolution in this paper. Given a low-resolution image, we seek to synthesize its underlying high-resolution details using a learning based method. Inspired by recent progress in compressive sensing, we use sparse representation prior to regularize this ill-posed problem. On the other hand, with natural image statistics taken into consideration, we enforce the prior only on those image patches associated with image primitives rather than on arbitrary ones. Specifically, each patch from primitive layer of the lowresolution image, which can be viewed as a low-dimensional projection of a high-resolution primitive patch, is conjectured to have a sparse representation concerning an over-complete dictionary. Under mild conditions, the sparse representation can be correctly restored from the low-dimensional projection according to the theory of compressive sensing. We also construct a dictionary using image primitive patches which works well on generic input images. Experiment results show the efficiency of our method by outperforming other learning-based methods both subjectively and objectively.

We propose a novel image denoising approach, which is based on exploring an underlying (nonlinear) lowdimensional manifold. Using local tangent space alignment (LTSA), we 'learn' such a manifold, which approximates the image content effectively. The denoising is performed by minimizing a newly defined objective function, which is a sum of two terms: (a) the difference between the noisy image and the denoised image, (b) the distance from the image patch to the manifold. We extend the LTSA method from manifold learning to denoising. We introduce the local dimension concept that leads to adaptivity to different kind of image patches, e.g. flat patches having lower dimension. We also plug in a basic denoising stage to estimate the local coordinate more accurately. It is found that the proposed method is competitive: its performance surpasses the K-SVD denoising method.

This paper presents an edge-preserving fourth order partial differential equation (PDE) for image restoration derived from a new surface-based energy functional. The corresponding fourth order PDE can preserve edges and avoid the staircase effect. The proposed model contains a function of gradient norm as an edge detector, which controls the diffusion speed according to the local structure of the image and preserves more details. Denoising results are given and we have also compared our method with some related PDE models.

Complex wavelet structural similarity (CW-SSIM) index has been proposed as a powerful image similarity metric that is robust to translation, scaling and rotation of images, but how to employ it in image classification applications has not been deeply investigated. In this paper, we incorporate CW-SSIM as a kernel function into a random forest learning algorithm. This leads to a novel image classification approach that does not require a feature extraction or dimension reduction stage at the front end. We use hand-written digit recognition as an example to demonstrate our algorithm. We compare the performance of the proposed approach with random forest learning based on other kernels, including the widely adopted Gaussian and the inner product kernels. Empirical evidences show that the proposed method is superior in its classification power. We also compared our proposed approach with the direct random forest method without kernel and the popular kernel-learning method support vector machine. Our test results based on both simulated and realworld data suggest that the proposed approach works superior to traditional methods without the feature selection procedure.

Humans are the ultimate receivers of the visual information contained in an image, so the reasonable method of image quality assessment (IQA) should follow the properties of the human visual system (HVS). In recent years, IQA methods based on HVS-models are slowly replacing classical schemes, such as mean squared error (MSE) and Peak Signal-to-Noise Ratio (PSNR). IQA-structural similarity (SSIM) regarded as one of the most popular HVS-based methods of full reference IQA has apparent improvements in performance compared with traditional metrics in nature, however, it performs not very well when the images' structure is destroyed seriously or masked by noise. In this paper, a new efficient fovea based structure similarity image quality assessment (FSSIM) is proposed. It enlarges the distortions in the concerned positions adaptively and changes the importances of the three components in SSIM. FSSIM predicts the quality of an image through three steps. First, it computes the luminance, contrast and structure comparison terms; second, it computes the saliency map by extracting the fovea information from the reference image with the features of HVS; third, it pools the above three terms according to the processed saliency map. Finally, a commonly experimental database LIVE IQA is used for evaluating the performance of the FSSIM. Experimental results indicate that the consistency and relevance between FSSIM and mean opinion score (MOS) are both better than SSIM and PSNR clearly.

Recently multiview video coding (MVC) standard has been finalized as an extension of H.264/AVC by Joint Video Team (JVT). In the project Joint Multiview Video Model (JMVM) for the standardization, illumination compensation (IC) is adopted as a useful tool. In this paper, a novel illumination compensation algorithm based on template is proposed. The basic idea of the algorithm is that the illumination of the current block has a strong correlation with its adjacent template. Based on this idea, firstly a template based illumination compensation method is presented, and then a template models selection strategy is devised to improve the illumination compensation performance. The experimental results show that the proposed algorithm can improve the coding efficiency significantly.

In this paper, an improved algorithm to generate a smooth and accurate depth map for view synthesis and multiview video coding is developed. For each block in the target view, the algorithm first uses epipolar geometry to find its matched block in the reference view, from which an initial depth is obtained using the triangulation method and depth projection. 3D warping is then applied to refine the depth. In addition, a structural similarity and maximum likelihood-based approach is developed to fuse the depth estimations from multiple references. Finally, the depth map is smoothed via segmentation and plane fitting. Compared to existing 3D warping-based depth estimation, the proposed algorithm can achieve up to 4 dB improvement in view synthesis, while requires much fewer bits to encode the depth map. Experimental results in multiview video coding show that the proposed method can outperform the H.264 JMVC software by more than 1 dB.

In this paper, a new scheme is presented to improve the coding efficiency of sequences captured by stationary cameras (or namely, static cameras) for video surveillance applications. We introduce two novel kinds of frames (namely background frame and difference frame) for input frames to represent the foreground/background without object detection, tracking or segmentation. The background frame is built using a background modeling procedure and periodically updated while encoding. The difference frame is calculated using the input frame and the background frame. A sequence structure is proposed to generate high quality background frames and efficiently code difference frames without delay, and then surveillance videos can be easily compressed by encoding the background frames and difference frames in a traditional manner. In practice, the H.264/AVC encoder JM 16.0 is employed as a build-in coding module to encode those frames. Experimental results on eight in-door and out-door surveillance videos show that the proposed scheme achieves 0.12 dB~1.53 dB gain in PSNR over the JM 16.0 anchor specially configured for surveillance videos.

In this paper, we propose a semi-automatic depth estimation algorithm whereby the user defines object depth boundaries and disparity initialization. Automatic depth estimation methods generally have difficulty to obtain good depth results around object edges and in areas with low texture. The goal of our method is to improve the depth in these areas and reduce view synthesis artifacts in Depth Image Based Rendering. Good view synthesis quality is very important in applications such as 3DTV and Free-viewpoint Television (FTV). In our proposed method, initial disparity values for smooth areas can be input through a so-called manual disparity map, and depth boundaries are defined by a manually created edge map which can be supplied for one or multiple frames. For evaluation we used MPEG multi-view videos and we demonstrate our algorithm can significantly improve the depth maps and reduce view synthesis artifacts.

The impact of using different wavelet packet subband structures in JPEG2000 on the matching accuracy of a fingerprint recognition system is investigated. In particular, we relate rate-distortion performance as measured in PSNR to the matching scores as obtained by the recognition system. Employing wavelet packets instead of the dyadic wavelet transform turns out to be of advantage only in case of low bitrates (i.e. high compression rates). For such settings, the good performance of the WSQ structure is confirmed also in JPEG2000 and in particular we get better recognition accuracy using the WSQ structure as compared to the employment of a rate-distortion optimising subband selection approach

It is known that the 2-D DCT basis is complete and orthogonal in a rectangular region. In this paper, we introduce the way to generate the complete and orthogonal 2-D DCT basis in a trapezoid region or a triangular region without using the complicated Gram-Schmidt method. Moreover, since a polygon can be decomposed several triangular regions, the proposed method is also suitable for the polygonal region. Our algorithm can much generalize the JPEG algorithm. Instead of dividing an image into 8 by 8 blocks, we can divide an image into trapezoid or triangular regions and then transform and code each of them. In addition to the DCT basis, our method can also be used for generating the 2-D complete and orthogonal DFT basis, KLT basis, Legendre basis, Hadamard (Walsh) basis, and polynomial basis in the trapezoid and triangular regions.

This paper investigates how to exploit the artifacts constraint to enhance the quality of side information (SI) in distributed video coding (DVC). The idea originates from the observation that there are usually some regions shown as artifacts in the interpolated SI, which seriously degrade the overall performance of the DVC system, especially when the GOP size is large. Encouraged by the previous work about the backward channel in the literature, we propose a simple and effective DVC scheme based on artifact detection and removal. On one hand, texture analysis via dynamic range and local entropy is utilized to determine artifact regions in spatial domain. On the other hand, motion activity evaluation is used to determine artifact regions in temporal domain, and temporal local entropy difference is further employed to eliminate the effect of other factors like brightness variance. After the artifact blocks are detected, we re-code them in intra mode and replace the co-located ones at the decoder side so that the quality of SI is enhanced. Experimental results demonstrate that the proposed SI quality enhancement technique is effective.

In this paper, adaptive fast matching algorithms are proposed based on the sub-block ordering. Proposed two algorithms for the lossless fast matching only reduce the unnecessary matching complexity of motion estimation in the video coding. By observing the expectation of the sum of absolute differences between the current sub-block and a reference sub-block in the pixel domain, we derive an absolute difference model which indicates the relationship between the sub-block distortion and the intra sub-block complexity. In the same way, we consider the inter complexity among all sub-block with the absolute difference model. From the absolute difference model, we derive the transform coefficient model for measuring the sub-block complexity in the transform domain based on the Parseval's theorem. Experimental results show the proposed fast matching algorithms reduce the unnecessary complexity of the full search algorithm about 20% on average without any loss of image quality.

The three-dimensional subband/wavelet codecs use 5/3 filters rather than Haar filters for the motion compensation temporal filtering (MCTF) to improve the coding gain. In order to curb the increased motion vector rate, an adaptive motion mode driven spatiotemporal motion vector prediction (AMDST-MVP) scheme is proposed. First, by making use of the direction histograms of four motion vector fields resulting from the initial spatial motion vector prediction (SMVP), the motion mode of the current GOP is determined according to whether the fast or complex motion exists in the current GOP. Then the GOP-level MVP scheme is thereby determined by either the S-MVP or the AMDST-MVP, namely, AMDST-MVP is the combination of S-MVP and temporal-MVP (T-MVP). If the latter is adopted, the motion vector difference (MVD) between the neighboring MV fields and the S-MVP resulting MV of the current block is employed to decide whether or not the MV of co-located block in the previous frame is used for prediction the current block. Experimental results show that AMDST-MVP not only can improve the coding efficiency but also reduce the number of computation complexity.

H.264/AVC adopts 9 intra 4×4 prediction modes to improve the intra frame compression efficiency. This makes the intra frame coding and transcoding very complex. In this paper, a fast intra 4×4 mode decision algorithm is proposed to reduce the intra transcoding complexity. Firstly, the surrounding modes in the high bitrate and the modes of the neighboring blocks in the current bitrate are used to form the candidate modes. Secondly, the variance and mean of the intra prediction residual are used to further reduce the size of the candidate mode set. Thirdly, the intra mode context model is built from the high bitrate modes and updated by the low bitrate modes to order the candidate modes. An early skip scheme is used to terminate the mode decision process. Experimental results show that about 50% intra 4×4 modes can be saved with a penalty on coding efficiency less than 0.1dB. The transcoding performance can even gain up to 0.6 dB on some sequences.

Variable block size motion compensation significantly improves the rate-distortion performance of video coding at the cost of high computational complexity. Currently, fast inter-mode decision algorithms only improve the speed of intermode decision (MD) but do not provide a flexible computational complexity control to adapt to different hardware platforms with optimized rate-distortion (R-D) performance. Instead of speeding up inter-mode decision merely, we attempt to propose a complexity adjustable inter-mode decision algorithm to attain optimized coding performance under different computational complexity constraints herein. Our algorithm predicts the Lagrangian cost and complexity slope (J-C slope) of MD for each macroblock (MB) by exploiting their temporal and spatial correlations. MD is applied on the MBs with larger J-C slope to provide the better prediction performance with less computational cost. Adjustable complexity is obtained by preset the number of MBs on which MD is applied for each frame. According to our experiments, the algorithm can both freely adjust the computational complexity and provides an improved R-D performance under different computational constraints.

Intra refresh is well-known as an efficient technique for eliminating temporal error propagation in a predictive video coding system. This paper proposes a novel intra refresh method based on motion affected region tracking (MTIR) for real time video communication systems. For every frame between two successive intra refresh frames, the motion affected regions are statistically analyzed to achieve the error sensibility of macroblocks, and then the error sensitive macroblocks are intra coded. To suppress the potential spatial and temporal error propagation, constrained intra prediction with limited DC prediction (LDC) is used for the refreshed macroblocks, which has also been adopted by the mobile profile of Chinese audio video standard (AVS) in Mar. 2009. In addition, the reference of inter predicted frame which behinds an intra refreshed frame is limited. Experimental results on JM15.1 show that compared with traditional methods, the proposed method can achieve a considerable gain on objective peak signal noise ratio (PSNR) about 0.5 to 1.5 dB, and the subjective visual quality of reconstructed videos can also be obviously improved.

Depth map is expected to be an essential component of upcoming 3D video formats. In a multiview scenario, along with color (texture), amount of depth information will also increase linearly with the number of views. Therefore various techniques are being explored in the research community to efficiently compress the depth data. In this paper, we propose novel methods of depth compression based on MPEG4 Multiview Video Coding standard (MVC) without any substantial increase in computational complexity. Our aim is to improve depth coding gain with minimal modification to the standard. We present experimental results which indicate a considerable coding gain when compared with MVC.

In this paper, a novel block-based motion compensated frame interpolation (MCI) algorithm is proposed to enhance the temporal resolution of video sequences. We formulated motion estimation into MAP framework, and solved it via Bayesian belief propagation. By effectively incorporating a priori knowledge of the motion field and optimizing the whole motion field synchronously, it could derive more accurate motion vectors than traditional methods. Finally, adaptive overlapped block motion compensation (OBMC) is used to reduce blocking artifacts. Experimental results show that the proposed method outperforms other methods in both objective and subjective quality.

This paper presents a motion estimation coprocessor, supporting real-time high definition (HD) video coding. The proposed coprocessor employs a 2D data-reusing processing element array and a multi-standard cost comparator for distortion computation of variable size block. An instruction-set suitable for performing various ME algorithms is developed, providing the flexibility needed for different coding standards and various applications. In 0.13μm CMOS technology, the coprocessor is implemented with 303K gates at operating frequency of 550MHz. A performance comparison to the reference architectures reveals that only the proposed processor can provide the processing capacity for 1920×1080, 60 fps video sequences at 33×33 search window.

Rate control has not been well studied for multi-view video coding (MVC). In this paper, we propose an efficient rate control algorithm for MVC by improving the quadratic rate-distortion (R-D) model, which reasonably allocate bit-rate among views based on correlation analysis. The proposed algorithm consists of four levels for rate bits control more accurately, of which the frame layer allocates bits according to frame complexity and temporal activity. Extensive experiments show that the proposed algorithm can efficiently implement bit allocation and rate control according to coding parameters.

Video adaptation has been proved to be an efficient technique in dealing with various constraints such as bandwidth limitation and user requirement in multimedia applications. However, existing methods including Scalable Video Coding and transcoding cannot get a fine performance when bandwidth constraints exist in various scenarios particularly in realtime applications. In this paper, we propose a novel rate control scheme based on intermediate description. The proposed scheme can provide fast rate control for narrow and time-varying transmission channel in scenarios such as video streaming, video sharing and video on demand. In this scheme, Discrete Cosine Transform (DCT) coefficients distribution is modeled by generalized Gaussian distribution, meanwhile the parameter information of this model is stored as side information for rate control. With the stored parameter information, encoder and transcoder can achieve the target bit-rate with low complexity. Furthermore, an initial Quantization Parameter (QP) determination method is also presented to calculate a proper QP for the Instantaneous Decoding Refresh (IDR) picture. Experimental results show that compared with JVT-G012 in H.264, the proposed rate control scheme can save more than 85% encoding time and obtain the required bit-rate more precisely, meanwhile gains a performance improvement by 0.2dB averagely.

This paper presents an error concealment method for medium grain scalability (MGS) of the scalable extension of H.264/AVC. Instead of reconstructing a lost frame using a typically compute-intensive error concealment method, in this method, a light-weight decoder preprocessor generates a valid bit stream from the available network abstraction layer (NAL) units in case that MGS layers are used. NAL unit header and slice header are parsed to detect the loss of a NAL unit from MGS layers. Modifications are applied to NAL unit header or slice header if some NAL units of MGS layers are lost. Hence, the number of additional packets that a decoder has to discard as a result of a packet loss is minimized. The proposed method requires low computational cost which is suitable for real-time video streaming. The experimental results demonstrate that the proposed method achieves an improvement of 2.16 dB compared to the NAL unit removal method at 5% packet loss rate. The proposed method can work complementary with other error concealment schemes for a whole-frame loss or serve as a preprocessor to a standard decoder.

Adaptive direction lifting-based wavelet transform (ADL) has better performance than conventional lifting both in image compression and de-noising. However, no architecture has been proposed to hardware implement it because of its high computational complexity and huge internal memory requirements. In this paper, we propose a four-stage pipelined architecture for 2 Dimensional (2D) ADL with fast computation and high data throughput. The proposed architecture comprises column direction estimation, column lifting, row direction estimation and row lifting which are performed in parallel in a pipeline mode. Since the column processed data is transposed, the row processor can reuse the column processor which can decrease the design complexity. In the lifting step, predict and update are also performed in parallel. For an 8×8 image sub-block, the proposed architecture can finish the ADL forward transform within 78 clock cycles. The architecture is implemented on Xilinx Virtex5 device on which the frequency can achieve 367 MHz. The processed time is 212.5 ns, which can meet the request of real-time system.

Object tracking plays a very important role in many computer vision applications. However its performance will significantly deteriorate due to some challenges in complex scene, such as pose and illumination changes, clustering background and so on. In this paper, we propose a robust object tracking algorithm which exploits both global color and local scale invariant (SIFT) features in a particle filter framework. Due to the expensive computation cost of SIFT features, the proposed tracker adopts a speed-up variation of SIFT, SURF, to extract local features. Specially, the proposed method first finds matching points between the target model and target candidate, than the weight of the corresponding particle based on scale invariant features is computed as the the proportion of matching points of that particle to matching points of all particles, finally the weight of the particle is obtained by combining weights of color and SURF features with a probabilistic way. The experimental results on a variety of challenging videos verify that the proposed method is robust to pose and illumination changes and is significantly superior to the standard particle filter tracker and the mean shift tracker.

This paper presents a novel approach for automatic pupil segmentation. The proposed algorithm uses local histogram and standard deviation based adaptive thresholding method that looks for the region that has the highest probability of having the pupil. We have tested our proposed algorithm on two public databases namely: CASIA v1.0 and MMU v1.0. Experimental results show that the proposed method has satisfying performance and good robustness against the reflection in the pupil.

This paper presents a novel texture description approach, which is robust to variances in rotation, scale and illumination in images, to classify the texture of images. A limitation with traditional methods is that they are more or less sensitive to the mentioned changes in images. To overcome this problem, we propose a novel Local Haar Binary Pattern (LHBP) based framework to ensure invariance in global rotation, scale, and light change. Our method consists of two components: feature extraction and scale self-adaptive classification. The global rotation invariant LHBP histogram features are extracted against the variances of illumination and global rotation, and the scale self-adaptive strategy is used for optimizing the classification of different scale textures. Evaluation results on Outex and Brodatz databases illustrate the significant advantages of the proposed approach over existing algorithms.

Partial occlusion is one of the most challenging difficulties for object tracking. In this paper, we present an approach to address this problem by using an effective appearance model which has two innovations. First, in contrast to widely used color histogram that models the appearance of an object using only color information, we assert that both color and texture are important cues for tracking, especially in the presence of complex background. We thus propose a novel local descriptor, named local color texture pattern (LCTP), to model the appearance of the object with color and texture information simultaneously. Second, global color histogram completely ignores the spatial layout information of an object and are sensitive to partial occlusion. In this work, we overcome this limitation based on a block-dividing way: 1) divide target into multiple blocks and then represent each block with LCTP histogram, 2) with a selectivity strategy, we select blocks that are not occluded and then combine similarities of those selected blocks to obtain final similarity measure. Experimental results demonstrate that the proposed method is more robust to partial occlusion than two state-of-the-art algorithms.

Finger stroke detection is an important topic in hand based Human Computer Interaction (HCI) system. Few research studies have carried out effective solutions to this problem. In this paper, we present a novel approach for stroke detection based on mono vision. Via analyzing the optical flow field within the finger area, our method is able to detect finger stroke under various camera position and visual angles. We present a thorough evaluation for each component of the algorithm, and show its efficiency and effectiveness on solving difficult stroke detection problems.

It is a challenging problem to robustly track moving objects from image sequences because of occlusions. Previous methods did not exploit depth information sufficiently. Based on multiple camera scenes, we propose a 3D silhouette tracking framework to resolve occlusions and recover the appearances in 3D space, which enhances tracking effectiveness. In the framework, 2D object silhouettes are initially gained by Snake. Then a Voxel Space Carving procedure is introduced to simultaneously generate the occlusion model and visual hull of objects. Next, we adopt Particle Filter to select the valuable parts of occlusion model and combine them with the initial object silhouettes to generate the updated visual hull. Finally, updated visual hull of the objects are re-projected to each view to obtain their final contours. The experiments under the public LAB and SCULPTURE datasets validate the feasibility and effectiveness of our framework.

High Dynamic Range (HDR) images capture the full range of luminance present in real world scenes, and unlike Low Dynamic Range (LDR) images, can simultaneously contain detailed information in the deepest of shadows and the brightest of light sources. In order to render HDR image on LDR displayers, it is often necessary to create LDR depictions of HDR images at the cost of contrast information loss. To reduce the loss, this paper enables to render HDRI (High Dynamic Range Image) with multiple low-bit images periodically. From the viewpoint of a human, the pixel value is fractural. It does not adjust the tones but can reconstruct HDR images.

In this paper we propose a model-assisted binocular stereo algorithm for 3D face reconstruction. First, stereo correspondence is reliably performed between input stereo images by employing a 3D reference face model as a medium. Then, a high-quality dataset of point cloud is generated by triangulation from the stereo correspondence. Finally, an accurate and feature-preserving 3D face surface is reconstructed from the point cloud dataset by a denoising operation of bilateral filtering and surface meshing. We have compared the reconstructed faces with ground truth 3D data. The experimental results show that the proposed algorithm is effective with the average error of 2.6mm.

The commercial success and acceptability of 3D technology will critically depend on the overall visual quality of the rendered images. Therefore Depth Image Based Rendering (DIBR) is a crucial component of the 3D system chain. In this paper, we describe a high quality DIBR system for view synthesis. In particular, a procedure is outlined for the processing of the so-called mixed pixels at object boundaries. A Layered Depth Image (LDI) representation of the scene is obtained given an image and corresponding depth map. In the process, all significant mixed pixels in the image are automatically separated into their local foreground and local background. Our results show superior rendering quality, especially at object edges.

Monte Carlo ray tracing algorithm is widely used by production quality renderers to generate synthesized images in films and TV programs. Noise artifact exists in synthetic images generated by Monte Carlo ray tracing methods. In this paper, a novel noise artifact detection and noise level representation method is proposed. We first apply discrete wavelet transform (DWT) on a synthetic image; the high frequency sub-bands of the DWT result encode the noise information. The sub-bands coefficients are then combined to generate a noise level description of the synthetic image, which is called noise map in the paper. This noise map is then subdivided into blocks for robust noise level metric calculation. Increasing the samples per pixel in Monte Carlo ray tracer can reduce the noise of a synthetic image to visually unnoticeable level. A noise-to-sample number mapping algorithm is thus performed on each block of the noise map, higher noise value is mapped to larger sample number, and lower noise value is mapped to smaller sample number, the result of mapping is called sample map. Each pixel in a sample map can be used by Monte Carlo ray tracer to reduce the noise level in the corresponding block of pixels in a synthetic image. However, this block based scheme produces blocky artifact as appeared in video and image compression algorithms. We use Gaussian filter to smooth the sample map, the result is adaptive sample map (ASP). ASP serves two purposes in rendering process; its statistics information can be used as noise level metric in synthetic image, and it can also be used by a Monte Carlo ray tracer to refine the synthetic image adaptively in order to reduce the noise to unnoticeable level but with less rendering time than the brute force method.

Images encoded at low bit rate may suffer from blocking artifacts, which dramatically degrade the visual quality. In this paper, we propose a postprocessing image restoration scheme for JPEG compressed images based on the maximum a posteriori criterion. A degradation model, represent by additive Gaussian noise model, is proposed to simulate JPEG compression process, while the original image is modeled as a high order Markov random field (MRF) based on the fields of experts framework. Meanwhile, an enhancement algorithm is proposed to restore the high frequency (HF) components. Experiment results have demonstrated that the proposed scheme can reproduce higher-quality images in terms of both objective and subjective quality.

Super-Resolution (SR) is a technique to construct a high-resolution (HR) frame by fusing a group of low-resolution (LR) frames describing the same scene. The effectiveness of the conventional super-resolution techniques, when applied on video sequences, strongly relies on the efficiency of motion alignment achieved by image registration. Unfortunately, such efficiency is limited by the motion complexity in the video and the capability of adopted motion model. In image regions with severe registration errors, annoying artifacts usually appear in the produced super-resolution video. This paper proposes a robust video super-resolution technique that adapts itself to the spatially-varying registration efficiency. The reliability of each reference pixel is measured by the corresponding registration error and incorporated into the optimization objective function of SR reconstruction. This makes the SR reconstruction highly immune to the registration errors, as outliers with higher registration errors are assigned lower weights in the objective function. In particular, we carefully design a mechanism to assign weights according to registration errors. The proposed superresolution scheme has been tested with various video sequences and experimental results clearly demonstrate the effectiveness of the proposed method.

In this paper, we proposed a passive scheme to achieve image tampering detection. To correctly estimate the quantization table, each AC DCT coefficient is classified into different types and then the corresponding quantization stepsize is adaptively measured from its power spectrum density (PSD) and PSD's Fourier transform. Based on the quantization table estimation, the proposed scheme is composed of pre-screening, candidate region selection, and tampering region identification. The pre-screening is developed to decide whether an image had been JPEG compressed. To select candidate regions for estimating quantization table, a seed region is first selected by finding a suitable region by removing suspicious regions. Based on the seed region, the candidate region can be obtained by suitably merging other regions into the seed. To avoid merging the suspect regions, a candidate region refinement is performed in the region growing. After estimating the quantization table from the candidate region, a MLR classifier based on the inconsistency of quantization table is exploited to identify tampered regions block by block. The experimental results demonstrate that our proposed scheme can not only estimate the quantization table but also identify tampered regions well.

This paper presents an effective and reversible watermarking approach for digital copyright protection of 2D-vector maps. To ensure that the embedded watermark is insensitive for human perception, we only select the noise non-sensitive regions for watermark embedding by estimating vertex density within each polyline. To ensure the exact recovery of original 2D-vector map after watermark extraction, we introduce a new reversible watermarking scheme based on reversible high-frequency wavelet coefficients modification. Within the former-selected non-sensitive regions, our watermarking operates on the lower-order vertex coordinate decimals with integer wavelet transform. Such operation further reduces the visual distortion caused by watermark embedding. We have validated the effectiveness of our scheme on our real-world city river/building 2D-vector maps. We give extensive experimental comparisons with state-of-the-art methods, including embedding capability, invisibility, and robustness over watermark attacking.

Mean Shift is popular in object tracking due to its simplicity and efficiency. It finds local maximum of the similarity measure between the target model and target candidate, and works well in many situations. However, it suffers from two aspects. First, Mean Shift tracker ignores background knowledge. As a result, it may fail when the background color is similar to that of the target or the initial target region contains too much background. Second, Mean Shift tracker omits the geometric structure with a global color histogram as the target model. Therefore, it may not work in the case of partial occlusion. To solve the first problem, we introduce background color histogram into a MAP formulation. To address the second problem, we divide the target into hierarchical blocks. These blocks are described with a histogram each but tracked as a whole. The two threads lead to a new algorithm, named MAP spatial pyramid (MAP-SP) Mean Shift. The efficiency of MAP-SP Mean Shift is demonstrated via comparative experiments on both standard and our own video sequences

In this paper, we introduce a new algorithm to register images with rotation and image blurring. The characteristic of this approach is that a set of Legendre moment invariants are used to establish the correspondence of corner points between the reference image and the distorted image, after these points are extracted by Harris corner detector. Transformation parameters can be estimated from those matched points.