Audiovisual tracker using particle filter

This project is relevant to an audio-visual tracker system oriented to security and video surveillance applications, using at the same time audio and video information.
The tracking approaches using visual only or audio only data, in many real-world scenarios where both audiovisual information are readily available, can be considered a limitation in terms of tracking performance. The basic idea is to merge all the available information, in this case audio and video, to perform better localization and tracking.
The principal problem in performing audio only tracking is related to the fact that audio data is not continuous over the time and furthermore, influent reverberation components are added to the data, which introduce non-negligible detection errors.
The way of video only tracking is well reliable and constant when the targets are in the camera field of view but limitations are introduced when the targets are video occluded or when they disappear from the camera field of view.
The idea of audio-visual data fusion possibly improves the tracking performance in situations when one of the two information is unavailable or when both information are disturbed.
A valid approach to obtain audio-visual information fusion and to perform targets tracking is by means of particle filter algorithm. Particle filter is based on Bayes’s rule and allows to estimate hidden variables of interest approximating the detections distributions, using a sampling-based approaches.

The developed audiovisual tracker works on the 2D image plane and is better performing over single target scenario. This thesis is organized in three main parts: state of the art, proposed solution and results, furthermore in the appendix is presented the audiovisual data acquisition system developed and used in this project.
State of the Art chapter is divided into three sections which present the existing technology respectively of audio analysis, video analysis, and data fusion techniques oriented to tracking.
• Audio Analysis section discusses a robust cross-correlation estimation method working on real audio dataset; this method is used to get information about audio sources location.
• Video Analysis section presents a background subtraction method and two other image processing steps: image soothing and morphological operations respectively. They are used to segment out objects of interest from a video sequence.
• Tracker and Data Fusion Algorithm section presents particle filter framework as technique to perform tracking and data fusion.

The Proposed Solution chapter is also divided in three main sections and presents specific implementations used to perform and overcome problems related to this project.
• Audio Analysis section presents an innovative method to compute the crosscorrelation function effective to reverberation and correlated noise; and a specific algorithm to extract interesting audio data for data fusion algorithm.
• Video Analysis section presents a specific algorithm based on background subtraction, to extract interesting video data for data fusion algorithm.
• Tracker and Data Fusion Algorithm section presents innovative specific particle filter application to track objects using audio video data together.

Results chapter presents the analysis of the obtained results, where the performances are evaluated processing different real dataset using the tracker developed in this project.