Satellite data compression has been an important subject since the beginning of satellites in orbit, and it has become an even more active research topic. Following technological advancements, the trend of new satellites has led to an increase in spatial, spectral, and radiometric resolution, an extension in wavelength range, and a widening of ground swath to better serve the needs of user community and decision makers. Data compression is often used as a sound solution to overcome the challenges of handling a tremendous amount of data. I have been working in this area since I was pursing my Ph. D. thesis almost 30 years ago.
Over the last two decades, I - as a senior research scientist and technical authority with the Canadian Space Agency - have led and carried out research and development of innovative data compression technology for optical satellites in collaboration with my colleagues at the agency, other government departments, my postdoctoral visiting fellows, internship students, and engineers at Canadian space industry. I invented and patented two series of near-lossless satellite data compression techniques and led the Canadian industry teams who implemented the techniques and built the onboard near-lossless compressors. I also led a multidisciplinary user team to assess the impact of the near-lossless compression techniques on ultimate satellite data applications. As the representative of Canada, I am an active member of the CCSDS working group for developing international data-compression standards for satellite data systems. Three international satellite data compression standards have been developed by the working group and published by the International Organization for Standardization (ISO). In collaborating with experts in this area in the world, I have co-chaired an SPIE conference on satellite data compression, communication, and signal processing since 2005. I have published over sixty papers and currently hold six U. S. patents, two European patents, and several pending patents in the subjects of satellite data compression and implementation. I feel that I have acquired sufficient knowledge and accumulated plenty experience in this area, and it is worth the effort to systematically organize them and put them into a book.
This book is my attempt to provide an end-to-end treatment of optical satellite data compression and implementation based on 30 years of firsthand experience and research outcomes. (It is a companion text to my book Optical Satellite Signal Processing and Enhancement, published by SPIE Press.) The contents of the book consist of nine chapters that cover a wide range of topics in this field. It serves as an introduction for readers who are willing to learn the basics and the evolution of data compression, and a guide for those working on onboard and ground satellite data compression, data handling and manipulation, and deployment of data-compression subsystems. The material is written to provide clear definitions and precise descriptions for advanced researchers and expert practitioners as well as for beginners. Chapters open with a brief introduction of the subject matter, followed by a review of previous approaches and their shortcomings, a presentation of recent techniques with improved performance, and finally a report on experimental results in order to assess their effectiveness and to provide conclusions.
Chapter 1 is an introduction to the book that describes the rationale and needs for satellite data compression and introduces a set of image quality metrics for assessing compressed satellite images. Chapter 2 presents a review of satellite lossless-data-compression techniques, considering both prediction-based and transform-based methods. Chapter 3 summarizes three international satellite-data-compression standards developed by CCSDS from a prospect of application of the standards. Chapter 4 describes vector quantization (VQ) based data-compression techniques that I have developed for compressing hyperspectral data. The focus of the research was to significantly reduce the computational complexity of conventional VQ algorithms in order for them to effectively compress hyperspectral datacubes. Many innovative yet practical solutions have been developed, including two of my granted patents: Successive Approximation Multi-stage Vector Quantization (SAMVQ) and Hierarchical Self-Organizing Cluster Vector Quantization (HSOCVQ). Chapter 5 describes how both of these techniques solve the blocking effect when applied to compressing continuous data flow generated aboard satellites and how they restrict the compression error to a level lower than that of the intrinsic noise of the original data to achieve so-called near-lossless compression. Chapter 6 addresses the optimization and implementation aspects of onboard data compression; aspects include the effect of anomalies of input data on compression performance, the location in the onboard data-processing chain where the compressor should be deployed, and the techniques to enhance error resilience in the data downlink transmission channel. Chapter 7 describes the hardware implementation of compression engines and onboard compressors that are based on SAMVQ and HSOCVQ. Chapter 8 reports a multidisciplinary user-acceptance study that assessed the impact of the compression techniques on various hyperspectral data applications to address the users’ concern about possible information loss due to the lossy compression nature of SAMVQ and HSOCVQ. Chapter 9 describes the Hyperspectral Image Browser (HIBR) system, which is capable of remotely displaying large hyperspectral datacubes via the Internet and of quickly processing the datacubes directly on the compressed form for users to identify the interested data, whose richness comes mostly from the spectral information.
There are many people I would like to thank for their contributions to the works included in this book. I would like to thank the Canadian Space Agency, where I have been working for the last 20 years; my colleagues Allan Hollinger, Martin Bergeron, Michael Maszkiewicz, Ian Cunningham, and Davinder Manak for their participation in data compression projects; my postdoctoral visiting fellows Pirouz Zarrinkhat and Charles Serele; and over forty intern students who have each left their mark. I would like to thank Robert Neville (retired), Karl Staenz (now at the University of Lethbridge), and Lixin Sun at the Canada Centre for Remote Sensing for collaborating on the Canadian hyperspectral program; Josée Lévesque and Jean-Pierre Ardouin at the Defence Research and Development Canada for their collaboration on assessing the impact of data compression. I thank David Goodenough at the Pacific Forestry Centre; John Miller and Baoxin Hu at York University for providing datasets and for actively collaborating on the data-compression user acceptability study; and Bormin Huang of the Cooperative Institute for Meteorological Satellite Studies at the University of Wisconsin-Madison for his discussion on satellite data compression.
I would also like to thank the participants in the user acceptability study: Andrew Dyk at the Pacific Forestry Centre; Jing Chen at the University of Toronto; Harold Zwick, Dan Williams, Chris Nadeau, and Gordon Jolly at MacDonald Dettwiler Associates; and Benoit Rivard and Jilu Feng at the University of Alberta. I thank Luc Gagnon, William Harvey, Bob Barrette, and Colin Black at MacDonald Dettwiler Associates (former EMS Technologies) for the development and fabrication of onboard compressor prototypes; and Melanie Dutkiewicz and Herbal Tsang for the development of a hyperspectral browser. I thank Valec Szwarc and Mario Caron at the Communication Research Centre (Canada) for discussions on enhancing resilience to bit errors produced by compression techniques; and Peter Oswald and Ron Buckingham for their discussion on onboard data compression. I would also like to thank Penshu Yeh at the NASA Goddard Space Flight Center, Aaron Kiely at the Jet Propulsion Laboratory, Carole Thiebaut and Gilles Moury at the French Space Agency (CNES), and Raffaele Vitulli at the European Space Agency for the collaboration within the CCSDS in developing international spacecraft-data standards and for their contributions to the CCSDS work included in this book.
I would also like to thank the three anonymous manuscript reviewers for their tireless work and strong endorsement of this book, their careful and meticulous chapter-by-chapter review on behalf of SPIE Press, and their detailed comments leading to the improvement and final results of the book in its current form. Many thanks as well to Tim Lamkins, Scott McNeill, and Dara Burrows at SPIE Press for turning my manuscript into this book.
Finally, I would like to thank my wife Nancy and daughter Cynthia for their help and support. They provided great encouragement and assistance during the period I wrote this book. The credit of this book should go to them.
Shen-En Qian
Senior Scientist, Canadian Space Agency
Montreal, Canada
September 2013
