- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper approaches the problem of detect-and-avoid (DAA) in air traffic. Using adversarial failure search, the goal is to find near mid-air colission (NMACs) states, which are often too rare and provides sparse learning signals. The paper uses the concept of heterogeneous criticality for the coliiding agents and uses per-agent values to represent how the criticality differs across time and different players in the simulation. The paper also replaces a hard mask with a soft mask calculated by GAE reconnection and long-tail weighting, which provides much better performance than the hard limit. The authors claims to uncover three previously unknown failure modes through this approach. The algorithm was proven and verified in the DAABench, a benchmark built around NASA DAIDALUS, a rule-based simulation of DAA standards. The authors mark DAABench as one of the contributions of the paper.
Critical strengths
The challenge of detect-and-avoid with near mid-air-collisions is a well-formulated and well-motivated problem. The approach is straight-forward and has a clear insight. The empirical study is extensive and ablation study is informative.
Critical weaknesses
While the work is impressive, novelty is still quite uncertain and it would be better to see how the algorithm is placed around and performs against the following existing works that addresses the issue of adversarial learning in multi-agent setup affected by environment.
-
Multi-agent Failure search : Failure-Scenario maker for rule-based agent using multi0agent adverarial RL (Wachi et al, IJCAI2019) already covered multiple adversarial agents to find failure cases of a rule-based system.
-
Works by Song&Schneider (ICRA2022 and CORL2024) uses genetic algorithm to find failure cases. The algorithm has been verified in multi-agent setup with heterogeneous opponent policy and under environment affect, which would be closely related to the work in HAD2RL. Furthermore, these works used adversarial failure search not just for the sake of searching for failure, but how to make the ego agent's policy more robust, connecting to one of the downstream uses of failure searches.
As for writing, since the members of the CS community is often not familiar with physical systems, it would be better to explain the domain-specific concepts more clearly. For example, perturber and intruder are concepts most people would be unfamiliar with unless working in the NMAC domain.
The authors claim that their benchmark is a wrapper around the NASA DAIDALUS. If this is the case, then this is not much of an contribution since most people can easily make a wrapper around an existing package, especially with the advent of coding agent. It would be better to highlight the engineering decisions, expert domain knowledge, and the insights the authors has put in to make the DAABench to be useful, something that would highlight the benchmark more than a a wrapper around an existing package.
While the paper claims to have discovered N-number of failures, such claims could be undermined if the failure mode are too similar to each other and therefore should be counted as one. The claims would be stronger if the authors could quantify how different are each of the situations are, either by some diversity metric, etc.
While this is understandable given the page limit, the experimentation lacks too much detail to understand what kind of environment the algorithm was tested in? For example, a simple summary about the benchmark being simulated/real, realsitic/simplified, fixed-wing/rotorcraft/multicoptor drone could give the some details about what how rigorous the experimentation was performed.
Questions for authors
In addition to addressing the weakness mentioned above, I have some questions related to experimentation details as listed in the last paragraph of the critical weakness.
- What kind of flight model was used? and how realistic?
- What kind of aircraft was it modeled on?
- What kind of pilot model was used and how realistic was it?
Also, would it be possible to share insights on why the failure modes you've discovered was difficult and challenging for existing works to detect? This could give better idea on strengths and weakness of the proposed algorithm, and what kind of failure scenarios it might discover in the future with a different benchmark or a different set of problems.