Monitoring of public and private sites is increasingly becoming a very important and critical issue, especially after the recent flurry of terrorist attacks including the one on the Word Trade Center in September 2001. It is, therefore, imperative that effective multisensor surveillance systems be developed to protect the society from similar attacks in the future. The new generation of surveillance systems to be developed have a specific requirement: they must be able to automatically identify criminal and terrorist activity without sacrificing individual privacy to the extent possible. Privacy laws concerning monitoring and surveillance systems vary from country to country but, in general, they try to protect the privacy of their citizens. Monitoring and visual surveillance has numerous other applications. It can be employed to help invalids or handicapped and to monitor the activities of elderly people. It can be used to monitor large events such as sporting events, as well. Nowadays, monitoring is employ~d in several different contexts including transport applications, such as monitoring of railway stations and airports, dangerous environments like nuclear facilities or traffic flows on roads and bridges. The latest generation of surveillance systems mainly rely on hybrid analog-digital, or completely digital video communications and processing methods and take advantage of the greater of flexibility offered by video processing algorithms that are capable focusing a human operator's attention on a set of interesting situations.

Over the years a great deal of multi-sensor fusion research has been conducted. While the potential military value of such research is considered high, there have been few fielded examples in the tactical arena. The objective of the Multi-Spectral Sensor Surveillance System (M4S) program is to transition high-payoff, low risk multi-sensor fusion technology from the research and development phase into practical user-oriented systems applications. The primary application focus of M4S is toward a robust and highly integrated autonomous littoral surveillance capability for either manned or unmanned surveillance aircraft. Sensor data from various organic, i.e. on-board, sensors as well as off-board sources is used to provide superior battlespace awareness. Specific operational payoffs from M4S include reduced manning requirements, improved detection, classification, and geo-location accuracy, and reduced prosecution timelines against timecritical targets. This objective and focus has led to the selection of a suite of notional sensors and a data fusion architecture consistent with the limited space and computational capabilities of airborne platforms. The four non-developmental organic sensors chosen for M4S include the PLAID ESM system, AN/APS-137B (V) 5 synthetic aperture radar, AN/AAQ-49 passive mid-wave FLIR, and a coherent CO2 LIDAR vibration sensor. The distributed data fusion architecture, which includes closed loop control of the above on-board sensors, is implemented using Pentium class microprocessors and the Windows NT operating system. All internal communications between system components is provided by a Multipoint Messaging System (MMS), which provides information flow control via an ask-answer messaging protocol. Use of MMS has aided system integration and provides a plug-and-play capability for new or additional sensors.

The recent development of intelligent surveillance systems has captured the interest of both academic research labs and industry. Automated Multi-Camera Surveillance addresses monitoring of people and vehicles, and detection of threatening objects and events in a variety of scenarios. In this book, techniques for development of an automated multi-camera surveillance system are discussed and proposed. The state-of-the-art in the automated surveillance systems is reviewed as well. Detailed explanation of sub-components of surveillance systems are provided, and enhancements to each of these components are proposed. The authors identify important challenges that such a system must address, and propose solutions. Development of a specific surveillance system called “KNIGHT” is described, along with the authors’ experience using it. This book enables the reader to understand the mathematical models and algorithms underlying automated surveillance as well as the benefits and limitations of using such methods.

This book discusses the design of multi-camera systems and their application to fields such as the virtual reality, gaming, film industry, medicine, automotive industry, drones, etc. The authors cover the basics of image formation, algorithms for stitching a panoramic image from multiple cameras, and multiple real-time hardware system architectures, in order to have panoramic videos. Several specific applications of multi-camera systems are presented, such as depth estimation, high dynamic range imaging, and medical imaging.

There is a growing interest in the development and deployment of intelligent surveillance systems in public and private locations. This book consists of a selection of extended versions of presentations made in two symposia on intelligent distributed surveillance systems (IDSS) and brings together the latest developments in the field.

This thesis investigates the issues of implementing multi-sensor imaging system for surveillance applications. Three topics for the fusion of infrared and electro-optic images are studied, i.e. registration, fusion, and evaluation. The first topic is the image registration or alignment, which is to associate corresponding pixels in multiple images to the same physical point in the scene. A trajectory-based method for registering infrared and electro-optic video sequences is proposed in this study. The initial registration parameters are derived from matching the trajectories across the consecutive video frames. Further refinement can be carried out by applying a maximum mutual information approach. The frame difference, from which the feature point is detected, is found with an image structural similarity measurement. The second topic is the implementation of pixel-level fusion. Two applications are considered in this study. Motivated by the adaptive enhancement, a modified pixel-level fusion scheme is proposed to implement the context enhancement. A visual image is first enhanced with the corresponding infrared image. Then, the enhanced image is fused with the visual image again to highlight the background features. This achieves a context enhancement most suitable for human perception. As the application of multi-sensor concealed weapon detection (CWD) is concerned, this thesis clarifies the requirements and concepts for CWD. How the CWD application can benefit from multi-sensor fusion is identified and a framework of multi-sensor CWD is proposed. A solution to synthesize a composite image from infrared and visual image is presented with experimental results. The synthesized image, on one hand provides both the information of personal identification and the suspicious region of concealed weapons; on the other hand implements the privacy protection, which appears to be an important aspect of the CWD process. The third topic is about the fusion performance assessment. In this study, the evaluation metrics are developed for reference-based assessment and blind assessment respectively. An absolute measurement of image features, namely phase congruency, is employed. Future work should include the reliability and optimization study of multiple image sensors from applications' and human perception-related perspectives. This thesis is a contribution to such research.

Using MATLAB examples wherever possible, Multi-Sensor Data Fusion with MATLAB explores the three levels of multi-sensor data fusion (MSDF): kinematic-level fusion, including the theory of DF; fuzzy logic and decision fusion; and pixel- and feature-level image fusion. The authors elucidate DF strategies, algorithms, and performance evaluation mainly

The active surveillance of maneuvering targets with multiple dynamic sensors requires a planning strategy to dynamically select and position the groups of sensors for optimal performance. This Thesis presents such a generic sensor-planning strategy that can be used for the real-time reconfiguration of a multi-sensor system for multi-object dynamic environments. The environment may include multiple Objects-of-Interest (OoI) as well as static and/or mobile objects that are not of interest (i.e., obstacles) but may act as occlusions. There should be no restrictions imposed on the number of sensors or their mobility as well as no requirement about knowing the objects' trajectories ahead of time. It is proposed herein to handle the sensor-planning problem using two complementary strategies: A coordination strategy to determine how many and specifically which sensors should be used at each demand instant in order to optimize the performance of the surveillance system over the span of several data-acquisition instants; and, a positioning strategy to determine the optimal pose of each sensor for any data-acquisition instant being serviced. In order to demonstrate the generality of the proposed methodology, it has been implemented in active-sensing for object localization, facial recognition, and object recognition via Shape-From-Shading (SFS). The applications cover surveillance of both time-variant and invariant parameters. Furthermore, they demonstrate the advantages of surveillance systems that utilizes multiple mobile sensors coupled with an effective sensor-planning strategy over static or single-sensor systems. The improvements in surveillance performance are primarily due to (i) increased robustness of the system (i.e., its ability to cope with a priori unknown target trajectories and presence of obstacles), (ii) decreased uncertainty associated with estimating the target's pose through sensor fusion, and (iii) increased reliability through sensory fault tolerance. The coordination strategy is proposed to be accomplished in this Thesis through an agent-based system consisting of multiple sensor agents, a referee agent, and a judge agent. Each sensor agent tries to maximize its own performance over the span of the rolling horizon. Although not directly controlled by a centralized controller, the sensor agents must abide the external rules of the environment monitored by the referee agent and enforced by the judge agent. The rules are set to ensure that the collective behaviour of the sensor agents exhibits the desired system behaviour. The positioning strategy is accomplished by determining regions of a sensor's workspace that are both unoccluded and achievable given the sensor's current pose, motion capabilities, and remaining time to data acquisition. The acceptable area is, then, searched for an optimal sensor pose.

This book includes papers from the section “Multisensor Information Fusion”, from Sensors between 2018 to 2019. It focuses on the latest research results of current multi-sensor fusion technologies and represents the latest research trends, including traditional information fusion technologies, estimation and filtering, and the latest research, artificial intelligence involving deep learning.