Various research video demos with links to available open access manuscripts, open source software and datasets.
Region-based Anomaly Detection in Thermal Imagery
Issue: the key challenge of Anomaly detection is namely the determination of the normal from the abnormal when operational data availability is highly biased towards one class (normal) due to both insufficient sample size, and inadequate distribution coverage for the other class (abnormal).
Approach: we propose the dual use of both visual appearance and localized motion characteristics, derived from optic flow, applied on a per-region basis to facilitate object-wise anomaly detection within this context.
Application: by leveraging established object localization techniques, optic flow is extracted from each object region and combined with appearance in the far infrared (thermal) band to give a 3-channel spatiotemporal tensor representation for each object (1 × thermal - spatial appearance; 2 × op- tic flow magnitude as x and y components - temporal motion).
This formulation is used as the basis for training contemporary semi-supervised anomaly detection approaches in a region-based manner such that anomalous objects can be detected as a combination of appearance and/or motion within the scene.
1 result2023 | ||
| [gaus23region] |  | Region-based Appearance and Flow Characteristics for Anomaly Detection in Infrared Surveillance Imagery , In Proc. Conf. Computer Vision and Pattern Recognition Workshops, IEEE/CVF, pp. 2995-3005, 2023.Keywords: anomaly detection, optic flow, thermal imagery, visual surveillance, R-CNN, convolutional neural network, deep learning. |
Cross-Spectral Transfer Learning for Object Detection and Classification in Thermal Imagery
Issue: there is a lack of prior work that specifically targets the use of deep learning based object detection in infrared-band images, owing to limited datasets availability that stems from more limited availability and access to infrared-band imagery and associated hardware in general.
Approach: transfer learning can enable the use of such deep learning techniques within infrared-band (thermal) imagery, by leveraging prior training on visible-band (RGB) image datasets, and then subsequently only requiring a secondary, smaller volume of infrared-band (thermal) imagery for final model fine-tuning.
Application: we demonstrate this paradigm of cross-spectral transfer learning over two state-of-art object detectors in order to provide a benchmark for future work in this area, the Single Shot Detector (SSD) and You-Only-Look-Once (YOLO,v3).
Exemplar results reported over the FLIR Thermal and MultispectralFIR benchmark datasets show that significant improvements in statistical detection performance with the use of cross-spectral transfer learning from visible-band (RGB) image datasets to infrared-band (thermal) imagery.
1 result2020 | ||
| [gaus20transfer] |  | Visible to Infrared Transfer Learning as a Paradigm for Accessible Real-time Object Detection and Classification in Infrared Imagery , In Proc. Conf. Counterterrorism, Crime Fighting, Forensics, and Surveillance Technologies, SPIE, Volume 11542, pp. 13-27, 2020.Keywords: far infrared, transfer learning, thermal imaging, people detection, vehicle detection. |
Reducing Racial Bias in Facial Recognition
Issue: No facial analysis system is perfectly accurate and a recent study by the US National Institute of Standards and Technology identified significant differences in how accurately face recognition software tools identify people of varied sex, age and racial background.
Approach: Our work focuses on bridging the gap between facial recognition systems that perform almost perfectly for white faces, but perform less well for faces of people belonging to other racial and ethnic groups.
We do this by making multiple face images, based on a real a person’s face, with different racial characteristics while keeping identifying features that are then used to train face recognition algorithms to verify these identifying features, instead of racial features, with the aim of making them less dependent upon race..
Application: The research has led to a small improvement in reducing racial bias and has increased the accuracy of facial recognition across all ethnicities.
1 result2020 | ||
| [yucer20racialbias] |  | Exploring Racial Bias within Face Recognition via per-subject Adversarially-Enabled Data Augmentation , In Proc. Computer Vision and Pattern Recognition Workshops, IEEE/CVF, pp. 1-8, 2020. (Workshop on Fair, Data Efficient and Trusted Computer Vision)Keywords: face recognition, bias in AI, racial bias, GAN, cycleGAN, style transfer, surveillance. |
Combined Colour and Thermal Imagery for Adaptive Background Modelling
Issue: contrary to other visible-band (colour, RGB) and infrared-band (T) cross-modal work in the field, we present a practical approach to parallax-free RGB-T image formation using a combination of optical engineering (beam-splitter) and visual geometry.
Approach: we evaluate the complementary nature of visible and far infrared (thermal, long-wave) information through three fusion schemes which physically combine visible-band (colour, RGB) and infrared-band (T) imagery into a co-registered, parallax free RGB-T image model.
Application: the use of combined colour and infrared within adaptive background modelling provides superior results under conditions when either visible or infrared band performance is notably degraded.
1 result2018 | ||
| [loveday18parallax] |  | On the Impact of Parallax Free Colour and Infrared Image Co-Registration to Fused Illumination Invariant Adaptive Background Modelling , In Proc. Computer Vision and Pattern Recognition Workshops, IEEE/CVF, pp. 1267-1276, 2018.Keywords: thermal, infrared, beamsplitter, registration, background modelling, thermal visible fusion, far infrared, LWIR. |
Real-time Vehicle Detection and Tracking in Thermal Imagery
Issue: Real-time classification of vehicles into sub-category types poses a significant challenge within infra-red imagery due to the high levels of intra-class variation in thermal vehicle signatures caused by aspects of design, current operating duration and ambient thermal conditions.
Approach: We investigate the accuracy of a number of real-time object classification approaches for this task within the wider context of an existing initial object detection and tracking frame-work.
Application: Based on photogrammetric estimation of target position, we then illustrate the use of regular Kalman filter based tracking operating on actual 3D vehicle trajectories.
Results are presented using a conventional thermal-band infra-red (IR) sensor arrangement where targets are tracked over a range of evaluation scenarios.
1 result2016 | ||
| [kundegorski16vehicle] |  | Real-time Classification of Vehicle Types within Infra-red Imagery , In Proc. SPIE Optics and Photonics for Counterterrorism, Crime Fighting and Defence, SPIE, Volume 9995, pp. 1-16, 2016.Keywords: vehicle sub-category classification, thermal target tracking, bag of visual words, histogram of oriented gradient, convolutional neural network, sensor networks, passive target positioning, vehicle localization. |
Sensing for Asset Protection with Integrated Electronic Networked Technology (SAPIENT)
Issue: we address a concept of Autonomous Sensor Modules (ASM) for wide area surveillance, where these modules have the ability to make low-level decisions on their own in order to fulfill a higher-level objective, and plug in, with the minimum of pre-configuration, to a broader distributed multi-model sensor network.
Approach: a demonstration system, known as Sensing for Asset Protection with Integrated Electronic Networked Technology (SAPIENT), which is shown in realistic base protection scenarios with live sensors and targets. The SAPIENT system performed sensor cuing, intelligent fusion, sensor tasking, target hand-off and compensation for compromised sensors, without human control.
Application: Potential benefits include rapid interoperability for coalition operations, situation understanding with low operator cognitive burden and autonomous sensor management in heterogeneous sensor systems
SAPIENT demonstrates "an autonomous sensor system designed to reduce the workload of perimeter protection and security personnel" - gov.uk/sapient.
1 result2016 | ||
| [thomas16sapient] |  | Towards Sensor Modular Autonomy for Persistent Land Intelligence Surveillance and Reconnaissance , In Proc. SPIE Ground/Air Multisensor Interoperability, Integration, and Networking for Persistent Intelligence Surveillance and Reconnaissance VII, SPIE, Volume 9831, No. VII, pp. 1-18, 2016.Keywords: information fusion, tracking, multiple sensor, sensor networks, wide area surveillance, sapient, ISR, multi-modal, fused tracking, sapient. |
Pedestrian Detection & Tracking in Thermal Imagery
Issue: investigates the accuracy of classical photogrammetry, within the context of current target detection and classification techniques, as a means of recovering the true 3D position of pedestrian targets within a scene under thermal video surveillance.
Approach: we leverage the key advantages of thermal-band infrared (IR) imagery for the pedestrian localization to show that robust localization and foreground target separation, afforded via such imagery, facilities accurate 3D position estimation to within the error bounds of conventional Global Position System (GPS) positioning.
Application: Target tracking within conventional video imagery poses a significant challenge that is increasingly being addressed via complex algorithmic solutions. The complexity of this problem can be fundamentally attributed to the ambiguity associated with actual 3D scene position of a given tracked object in relation to its observed position in 2D image space. We propose an approach that challenges the current trend in com-plex tracking solutions by addressing this fundamental ambiguity head-on.
Based on photogrammetric estimation of target position, we illustrate the efficiency of regular Kalman filter based tracking operating on actual 3D pedestrian scene trajectories with a secondary approach to correct for the potential impact of pedestrian posture variation.
2 results2015 | ||
| [kundegorski15posture] |  | Posture Estimation for Improved Photogrammetric Localization of Pedestrians in Monocular Infrared Imagery , In Proc. SPIE Optics and Photonics for Counterterrorism, Crime Fighting and Defence, SPIE, Volume 9652, No. XI, pp. 1-12, 2015.Keywords: thermal people detection, infrared, tracking, photogrammetry, tracking, posture estimation, pedestrian height regression, regressive height estimation. |
2014 | ||
| [kundegorski14photogrammetric] |  | A Photogrammetric Approach for Real-time 3D Localization and Tracking of Pedestrians in Monocular Infrared Imagery , In Proc. SPIE Optics and Photonics for Counterterrorism, Crime Fighting and Defence, SPIE, Volume 9253, No. 01, pp. 1-16, 2014.Keywords: thermal people detection, infrared, tracking, photogrammetry, 3D tracking. |
Super-Resolution for Real-time Target Tracking
Issue: the ability to obtain a high quality, high resolution appearance model of a tracked target is often highly desirable but the reality of operational deployment often means that imaging systems deployed for this task suffer from limitations reducing effective image quality.
Approach: to improve the visual quality of tracked targets, we propose a simple yet effective solution that integrates a super-resolution imaging approach based on combination of the Sum of the Absolute Differences (SAD) and gradient-descent motion estimation techniques into a novel tracking approach.
Application: the proposed approach demonstrates robustness in improved target appearance modeling that assists the overall tracking system.
Results demonstrate a significant improvement in visual target representation whilst tracking over high dynamic scenes.
1 result2013 | ||
| [mise13superresolution] |  | Image Super-Resolution Applied to Moving Targets in High Dynamics Scenes , In Proc. SPIE Emerging Technologies in Security and Defence: Unmanned Sensor Systems, SPIE, Volume 8899, No. 17, pp. 1-12, 2013.Keywords: super-resolution, tracking. |
Multi-Modal Target Detection for Autonomous Wide Area Search and Surveillance
Issue: the realization of a real-time methodology for the automated detection of people and vehicles using combined visible-band (EO), thermal-band (IR) and radar sensing from a deployed network of multiple autonomous platforms (ground and aerial).
Approach: A range of automatic classification approaches are proposed, driven by underlying machine learning techniques, that facilitate the automatic detection of either target type with cross-modal target confirmation.
Application: Generalised wide are search and surveillance is a common-place tasking for multi-sensory equipped autonomous systems. Here we present on a key supporting topic to this task - the automatic interpretation,fusion and detected target reporting from multi-modal sensor information received from multiple autonomous platforms deployed for wide-area environment search.
This facilities real-time target detection, reported with varying levels of confidence, using information from both multiple sensors and multiple sensor platforms to provide environment-wide situational awareness.
Extended results present both people and vehicle detection under varying conditions in both isolated rural and cluttered urban environments with minimal false positive detection. Episodic target detection, evaluated over a number of wide-area environment search and reporting tasks, generally exceeds 90%+ for the targets considered here.
1 result2013 | ||
| [breckon13autonomous] |  | Multi-Modal Target Detection for Autonomous Wide Area Search and Surveillance , In Proc. SPIE Emerging Technologies in Security and Defence: Unmanned Sensor Systems, SPIE, Volume 8899, No. 01, pp. 1-19, 2013.Keywords: autonomous robots, grand challenge, wide area search, search and rescue, UAV, infrared, thermal. |
Real-time Multi-modal Object Detection
Issue: Automated reporting of detected targets from an unattended multi-modal sensor pod deployed as part of a wide area sensor network.
Approach: : An automated classification approach is employed to detect vehicle and people targets. A strategy is introduced that integrates this detection over modality (visible/thermal) and time to provide robust target reporting.
We illustrate the effectiveness of simple temporal filtering strategies for the integration of automated detections over time and sensor modality for reducing false positive target reports and target re-reporting.
Application: Localised (at sensor) processing facilitates the minimal onward reporting of detected targets making effective use of network bandwidth, negating the need for full motion video transmission and minimizing the communications and power footprint of sensors covertly deployed as part of a wider sensor network.
1 result2012 | ||
| [breckon12multimodal] |  | Consistency in Muti-modal Automated Target Detection using Temporally Filtered Reporting , In Proc. SPIE Electro-Optical Remote Sensing, Photonic Technologies, and Applications VI, Volume 8542, No. 85420L-1, pp. 23:1-23:12, 2012.Keywords: thermal people detection, infrared, multimodal detection, temporal filtering, temporal detection strategy. |
Automated Visual Surveillance
Issue: CCTV surveillance is ever increasing in UK (and European) society. With this increase comes and increased burden of monitoring the vast amounts of visual information for abnormal or irregular events that highlight a particular security problem or related incident.
Approach: This research is aimed towards part of the problem of automatically monitoring CCTV video feeds for detecting pedestrian/vehicle presence together with tracking to detect abnormal or irregular events that may be of interest.
Application: We present a novel pipeline for automated visual surveillance system based on utilising conventional adaptive background modeling in-conjunction with optic flow to provide motion sensitive foreground/background segmentation.
A conjunction of background modeling based foreground detection and optical flow within the image is used to provide a robust foreground object segmentation. Feature-based tracking then recovers scene tracks for these isolated foreground objects.
1 result2007 | ||
| [li07motion] |  | Combining Motion Segmentation and Feature Based Tracking for Object Classification and Anomaly Detection , In Proc. 4th European Conference on Visual Media Production, IET, pp. I-6, 2007.Keywords: visual surveillance, optical flow, person tracking, pedestrian tracking. |