Recently, Sense Aeronautics has been approached with questions regarding the distinctions between its Automatic Target Recognition (ATR) system for identifying people, vehicles, and drones and the open-source alternatives found in public repositories.
The primary distinction is that Sense Aeronautics’ ATR is not merely an advanced AI model with improved detection capabilities and fewer false alerts; it represents a thorough video processing framework, notably suited for drone fleets and swarm operations.
This ATR model supports a wide range of formats, frame rates, and resolutions, all accessible through a single REST API, which can be implemented on various hardware setups or through cloud solutions.
The engineering team at Sense Aeronautics offers extensive support throughout every stage of a project, including custom developments tailored to meet specific requirements.
To focus on model efficacy,the company recently performed a benchmark against leading models in the field. The results are outlined below.
The Importance of a Curated and Representative Dataset
A typical method for selecting data for benchmarking involves randomly sampling videos that were not utilized during model training. However, Sense Aeronautics opted for a more systematic approach to ensure that the selected videos accurately reflect specific scenarios or particular client needs.
The company utilized a meticulously curated dataset from its partners, which accurately represents real-world conditions. Each video was carefully labeled by hand prior to generating performance metrics.
Benchmarking Methodology
The benchmark dataset was evaluated using two models: sense aeronautics’ proprietary ATR 1.0 and a pretrained model based on the YOLO architecture. The following performance metrics were utilized:
- Detection Probability (PD) = TP / (TP + FN), where TP stands for True Positives and FN for False Negatives. This metric assesses the model’s accuracy in detecting targets against the ground truth.
- False Alerts Per Second (FA/s) = FP / time (s), where FP denotes False Positives.This measures the frequency of false alerts.
These metrics were computed for each model: PD (ATR 1.0), PD (SoA), FA/s (ATR 1.0), and FA/s (soa). Improvements in detection and false alert rates were than calculated as follows:
targets were classified based on their pixel area to derive Detection Probability per size category:
- Tiny: fewer than 576 square pixels.
- Small: fewer than 1024 square pixels.
- Medium: fewer than 9216 square pixels.
- Large: greater than 9216 square pixels.
Discover more about Sense Aeronautics’ Automatic Target Recognition (ATR) capability >>
Findings
The benchmark results revealed substantial enhancements in the Sense Aeronautics ATR 1.0 model compared to the leading model. Key insights include:
- Detection probability for large targets remained approximately the same (+1%).
- An increase of +18% in detection probability for medium targets.
- A remarkable +285% increase in detection probability for small targets.
- An astounding +1430% increase in detection probability for tiny targets.
- The ATR 1.0 model produced half the number of false positives compared to the pretrained model.
Beyond the quantitative data, the models were qualitatively assessed by visualizing the bounding boxes around detected targets.In the accompanying image,targets identified by both models are highlighted in orange,while those detected exclusively by the ATR 1.0 model are marked in blue.
It is evident that while both models successfully detect larger (closer) targets, the ATR 1.0 model excels in identifying smaller (more distant) targets. This enhanced capability to detect smaller objects also indicates a greater proficiency in recognizing targets at extended distances.
Final Thoughts
The custom detection model from Sense Aeronautics not only improves the identification of smaller objects but also significantly minimizes false alerts, providing a more dependable solution compared to standard pretrained models.
Discover more about Sense Aeronautics’ Automatic Target Recognition (ATR) capability >>
