This thesis investigates the issues of implementing multi-sensor imaging system for surveillance applications. Three topics for the fusion of infrared and electro-optic images are studied, i.e. registration, fusion, and evaluation. The first topic is the image registration or alignment, which is to associate corresponding pixels in multiple images to the same physical point in the scene. A trajectory-based method for registering infrared and electro-optic video sequences is proposed in this study. The initial registration parameters are derived from matching the trajectories across the consecutive video frames. Further refinement can be carried out by applying a maximum mutual information approach. The frame difference, from which the feature point is detected, is found with an image structural similarity measurement. The second topic is the implementation of pixel-level fusion. Two applications are considered in this study. Motivated by the adaptive enhancement, a modified pixel-level fusion scheme is proposed to implement the context enhancement. A visual image is first enhanced with the corresponding infrared image. Then, the enhanced image is fused with the visual image again to highlight the background features. This achieves a context enhancement most suitable for human perception. As the application of multi-sensor concealed weapon detection (CWD) is concerned, this thesis clarifies the requirements and concepts for CWD. How the CWD application can benefit from multi-sensor fusion is identified and a framework of multi-sensor CWD is proposed. A solution to synthesize a composite image from infrared and visual image is presented with experimental results. The synthesized image, on one hand provides both the information of personal identification and the suspicious region of concealed weapons; on the other hand implements the privacy protection, which appears to be an important aspect of the CWD process. The third topic is about the fusion performance assessment. In this study, the evaluation metrics are developed for reference-based assessment and blind assessment respectively. An absolute measurement of image features, namely phase congruency, is employed. Future work should include the reliability and optimization study of multiple image sensors from applications' and human perception-related perspectives. This thesis is a contribution to such research.

Taking another lesson from nature, the latest advances in image processing technology seek to combine image data from several diverse types of sensors in order to obtain a more accurate view of the scene: very much the same as we rely on our five senses. Multi-Sensor Image Fusion and Its Applications is the first text dedicated to the theory and practice of the registration and fusion of image data, covering such approaches as statistical methods, color-related techniques, model-based methods, and visual information display strategies. After a review of state-of-the-art image fusion techniques, the book provides an overview of fusion algorithms and fusion performance evaluation. The following chapters explore recent progress and practical applications of the proposed techniques to solving problems in such areas as medical diagnosis, surveillance and biometric systems, remote sensing, nondestructive evaluation, blurred image restoration, and image quality assessment. Recognized leaders from industry and academia contribute the chapters, reflecting the latest research trends and providing useful algorithms to aid implementation. Supplying a 28-page full-color insert, Multi-Sensor Image Fusion and Its Applications clearly demonstrates the benefits and possibilities of this revolutionary development. It provides a solid knowledge base for applying these cutting-edge techniques to new challenges and creating future advances.

The objective of this book is to advance the current knowledge of sensor research particularly highlighting recent advances, current work, and future needs. The goal is to share current technologies and steer future efforts in directions that will benefit the majority of researchers and practitioners working in this broad field of study.

The recent development of intelligent surveillance systems has captured the interest of both academic research labs and industry. Automated Multi-Camera Surveillance addresses monitoring of people and vehicles, and detection of threatening objects and events in a variety of scenarios. In this book, techniques for development of an automated multi-camera surveillance system are discussed and proposed. The state-of-the-art in the automated surveillance systems is reviewed as well. Detailed explanation of sub-components of surveillance systems are provided, and enhancements to each of these components are proposed. The authors identify important challenges that such a system must address, and propose solutions. Development of a specific surveillance system called “KNIGHT” is described, along with the authors’ experience using it. This book enables the reader to understand the mathematical models and algorithms underlying automated surveillance as well as the benefits and limitations of using such methods.

Large-scale video networks are of increasing importance in a wide range of applications. However, the development of automated techniques for aggregating and interpreting information from multiple video streams in real-life scenarios is a challenging area of research. Collecting the work of leading researchers from a broad range of disciplines, this timely text/reference offers an in-depth survey of the state of the art in distributed camera networks. The book addresses a broad spectrum of critical issues in this highly interdisciplinary field: current challenges and future directions; video processing and video understanding; simulation, graphics, cognition and video networks; wireless video sensor networks, communications and control; embedded cameras and real-time video analysis; applications of distributed video networks; and educational opportunities and curriculum-development. Topics and features: presents an overview of research in areas of motion analysis, invariants, multiple cameras for detection, object tracking and recognition, and activities in video networks; provides real-world applications of distributed video networks, including force protection, wide area activities, port security, and recognition in night-time environments; describes the challenges in graphics and simulation, covering virtual vision, network security, human activities, cognitive architecture, and displays; examines issues of multimedia networks, registration, control of cameras (in simulations and real networks), localization and bounds on tracking; discusses system aspects of video networks, with chapters on providing testbed environments, data collection on activities, new integrated sensors for airborne sensors, face recognition, and building sentient spaces; investigates educational opportunities and curriculum development from the perspective of computer science and electrical engineering. This unique text will be of great interest to researchers and graduate students of computer vision and pattern recognition, computer graphics and simulation, image processing and embedded systems, and communications, networks and controls. The large number of example applications will also appeal to application engineers.

Image fusion is a process of combining complementary information from multiple images to generate a single image that is more suitable for the purpose of human visual perception or further image processing tasks. The source images can come from different types of sensors of the same basic type or they may come from different types of sensors. In our research, we focus on fusing images from different sensors. The application we mainly considered is the concealed weapon detection application (CWD) which utilizes multiple sensors to improve the identification of the person having weapons hidden underneath his clothing. CWD appears to be a critical technology for dealing with terrorism which is the most significant law enforcement problem for the next decade. In this dissertation, we focused on investigating three aspects of image fusion which have impact on the CWD application, with the hope of shedding more light on this promising area. First, we tested and compared the performance of a large set of typical image fusion algorithms for the CWD application using some typical quality measures. Based on the experimental results, it is clear that there is a need to develop new image fusion algorithms that are more effective for this specific application. In addition, most of the current image fusion work has been limited to monochrome images although the human vision system is very sensitive to colors. Thus, we proposed a new color fusion algorithm. Using the proposed method the fused image will maintain the high resolution and the natural color of the visual image while incorporating any useful information detected by the nonvisual sensor. The utility of the proposed method was demonstrated in the experimental tests. For multisensor image fusion, blind estimation of the image quality is of great interest and of great difficulty. Several objective image quality measures for image fusion have appeared, but there has been a lack of theoretical analysis on the performance of the quality measures. Thus, in this dissertation, we took a first step towards theoretically analyzing one of the popular quality measures, a mutual information based quality measure which had only been tested experimentally for a limited number of cases. We augmented these experimental tests with analysis.

Information fusion resulting from multi-source processing, often called multisensor data fusion when sensors are the main sources of information, is a relatively young (less than 20 years) technology domain. It provides techniques and methods for: Integrating data from multiple sources and using the complementarity of this data to derive maximum information about the phenomenon being observed; Analyzing and deriving the meaning of these observations; Selecting the best course of action; and Controlling the actions. Various sensors have been designed to detect some specific phenomena, but not others. Data fusion applications can combine synergically information from many sensors, including data provided by satellites and contextual and encyclopedic knowledge, to provide enhanced ability to detect and recognize anomalies in the environment, compared with conventional means. Data fusion is an integral part of multisensor processing, but it can also be applied to fuse non-sensor information (geopolitical, intelligence, etc.) to provide decision support for a timely and effective situation and threat assessment. One special field of application for data fusion is satellite imagery, which can provide extensive information over a wide area of the electromagnetic spectrum using several types of sensors (Visible, Infra-Red (IR), Thermal IR, Radar, Synthetic Aperture Radar (SAR), Polarimetric SAR (PolSAR), Hyperspectral...). Satellite imagery provides the coverage rate needed to identify and monitor human activities from agricultural practices (land use, crop types identification...) to defence-related surveillance (land/sea target detection and classification). By acquiring remotely sensed imagery over earth regions that land sensors cannot access, valuable information can be gathered for the defence against terrorism. This books deals with the following research areas: Target recognition/classification and tracking; Sensor systems; Image processing; Remote sensing and remote control; Belief functions theory; and Situation assessment.

There is a growing interest in the development and deployment of intelligent surveillance systems in public and private locations. This book consists of a selection of extended versions of presentations made in two symposia on intelligent distributed surveillance systems (IDSS) and brings together the latest developments in the field.