acceptodds
Under review as a conference paper at ICLR 2027

HAD²RL: Heterogeneous-Agent Dense Reinforcement Learning for Adversarial Failure Search of Detect-and-Avoid Systems

Abstract

Low-altitude unmanned aircraft are being deployed at scale for logistics and urban air mobility, and their detect-and-avoid (DAA) systems are designed to keep them separated from other aircraft and prevent near mid-air collisions (NMACs). Validating DAA safety relies on simulation, but NMACs are extremely rare, so conventional validation requires expert-designed test scenarios and large-scale simulation before failures are observed, which is costly. Recent work uses adversarial reinforcement learning to search for critical failures, but the sparse learning signal is often drowned out by gradient noise from the many non-critical states. Moreover, DAA failures typically arise from heterogeneous factors acting jointly, such as other aircraft and the environment, whose critical moments differ in time, so the search space grows and failures become sparser still. We therefore propose Heterogeneous-Agent Dense Reinforcement Learning (HADRL). HAD2RL estimates, for each agent, how much each state matters from the value of the failure event. It then replaces the hard mask with long-tailed weighting and GAE reconnection, so that credit follows this criticality and misjudged states are down-weighted rather than discarded. We prove that soft densification introduces only a bounded bias into the expected policy gradient while reducing gradient noise by one to two orders of magnitude. To evaluate HAD2RL, we build DAABench, the first benchmark for adversarial failure search on rule-based DAA systems. On its 11 encounter families, HAD2RL raises the NMAC discovery rate of three base algorithms by 15.4% on average and by up to in rare families, and uncovers three previously unknown failure modes. abstract

Primary area:
reinforcement learning
Keywords:
Multi-agent reinforcement learning, Safety Validation, Detect-and-Avoid, Adversarial failure search, Dense reinforcement learning, Credit assignment

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_XsrYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper approaches the problem of detect-and-avoid (DAA) in air traffic. Using adversarial failure search, the goal is to find near mid-air colission (NMACs) states, which are often too rare and provides sparse learning signals. The paper uses the concept of heterogeneous criticality for the coliiding agents and uses per-agent values to represent how the criticality differs across time and different players in the simulation. The paper also replaces a hard mask with a soft mask calculated by GAE reconnection and long-tail weighting, which provides much better performance than the hard limit. The authors claims to uncover three previously unknown failure modes through this approach. The algorithm was proven and verified in the DAABench, a benchmark built around NASA DAIDALUS, a rule-based simulation of DAA standards. The authors mark DAABench as one of the contributions of the paper.

Critical strengths

The challenge of detect-and-avoid with near mid-air-collisions is a well-formulated and well-motivated problem. The approach is straight-forward and has a clear insight. The empirical study is extensive and ablation study is informative.

Critical weaknesses

While the work is impressive, novelty is still quite uncertain and it would be better to see how the algorithm is placed around and performs against the following existing works that addresses the issue of adversarial learning in multi-agent setup affected by environment.

  • Multi-agent Failure search : Failure-Scenario maker for rule-based agent using multi0agent adverarial RL (Wachi et al, IJCAI2019) already covered multiple adversarial agents to find failure cases of a rule-based system.

  • Works by Song&Schneider (ICRA2022 and CORL2024) uses genetic algorithm to find failure cases. The algorithm has been verified in multi-agent setup with heterogeneous opponent policy and under environment affect, which would be closely related to the work in HAD2RL. Furthermore, these works used adversarial failure search not just for the sake of searching for failure, but how to make the ego agent's policy more robust, connecting to one of the downstream uses of failure searches.

As for writing, since the members of the CS community is often not familiar with physical systems, it would be better to explain the domain-specific concepts more clearly. For example, perturber and intruder are concepts most people would be unfamiliar with unless working in the NMAC domain.

The authors claim that their benchmark is a wrapper around the NASA DAIDALUS. If this is the case, then this is not much of an contribution since most people can easily make a wrapper around an existing package, especially with the advent of coding agent. It would be better to highlight the engineering decisions, expert domain knowledge, and the insights the authors has put in to make the DAABench to be useful, something that would highlight the benchmark more than a a wrapper around an existing package.

While the paper claims to have discovered N-number of failures, such claims could be undermined if the failure mode are too similar to each other and therefore should be counted as one. The claims would be stronger if the authors could quantify how different are each of the situations are, either by some diversity metric, etc.

While this is understandable given the page limit, the experimentation lacks too much detail to understand what kind of environment the algorithm was tested in? For example, a simple summary about the benchmark being simulated/real, realsitic/simplified, fixed-wing/rotorcraft/multicoptor drone could give the some details about what how rigorous the experimentation was performed.

Questions for authors

In addition to addressing the weakness mentioned above, I have some questions related to experimentation details as listed in the last paragraph of the critical weakness.

  • What kind of flight model was used? and how realistic?
  • What kind of aircraft was it modeled on?
  • What kind of pilot model was used and how realistic was it?

Also, would it be possible to share insights on why the failure modes you've discovered was difficult and challenging for existing works to detect? This could give better idea on strengths and weakness of the proposed algorithm, and what kind of failure scenarios it might discover in the future with a different benchmark or a different set of problems.

Reviewer_cThQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HAD$^2$RL, a reinforcement learning framework for efficiently discovering safety-critical failures in detect-and-avoid (DAA) systems by addressing the challenges of sparse learning signals and heterogeneous factors. The authors also introduce a dedicated benchmark and demonstrate the effectiveness of the proposed method through extensive experiments.

Critical strengths
  1. The paper introduces a benchmark that can be widely used to evaluate methods for discovering safety-critical failures in DAA systems.

  2. The paper provides both algorithmic contributions and theoretical results.

  3. The main experimental results are comprehensive.

Critical weaknesses
  1. The generalizability of the proposed algorithm needs further validation.

  2. Validation in real-world scenarios is lacking.

Please refer to the questions section for details.

Questions for authors
  1. Besides the benchmark proposed in this paper, are there other benchmarks that could be used to validate the effectiveness of the proposed algorithm?

  2. The proposed method is designed specifically for the UAV scenario. Does it have broader applicability? For example, can it achieve good performance in the SMAC environment?

  3. The figures need further refinement. For example, in Fig. 1, the text “GAE” and “update” overlaps with the borders.

  4. Can classical policy-gradient-based MARL algorithms, such as MADDPG and MAAC, be effective in this scenario?

  5. Can the proposed method be validated in real-world UAV scenarios?

Reviewer_Ek5COpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a multi-agent adversarial failure search of the near mid-air collisions (NMAC), a failure event that is important for the detect-and-avoid (DAA) systems for the low-altitude unmanned aircrafts. The paper uses two distinct agents to do the adversary, to make the failure happen more often. The motivation comes from the fact that the failure cases are rare to happen in the random simulations, thus finding one needs a lot of iterations, while the authors propose the adversary to make it happen more often. The previous work address this issue by classifying the actions into "critical" and "non-critical", however this is not enough, because the failure itself may arise from heterogeneous sources. This paper considered this heterogeneity and proposes a benchmark for studying this problem.

Critical strengths

The paper's main strength is that its extension from single-agent to multi-agent dense reinforcement learning is motivated by the structure of the problem. DAA failures arise from two causally different pathways, the intruder's kinematics and the perturber's corruption of perception and wind, and their critical moments occur at different times. The authors identify precisely what breaks when D2RL is carried over (a single shared critical set and a binary mask) and propose a targeted fix: per-agent criticality with soft, floor-bounded weighting.

A second strength is the generality of the method. HAD²RL modifies only the advantage estimate and leaves the actor objective untouched, so it plugs into existing heterogeneous MARL algorithms. It is applied to HAPPO, MAPPO, and HATRPO and improves the aggregate failure rate on all three. Because setting λ = 0 recovers the base algorithm exactly, the comparisons are clean and the gains can be attributed to densification.

Third, the theory makes a testable prediction that the experiments then check. The propositions imply that the benefit of densification should shrink as failures become common, and the results follow this pattern, with large gains in rare encounter families and almost none in saturated ones. The mechanism check in Fig. 3, which tests whether the learned weights land on steps whose actions actually flip the outcome, supports this reading further.

Finally, DAABench is a valuable contribution in its own right. It provides a closed-loop, DO-365-based benchmark with a dual attack surface, rarity-stratified encounter families, fixed held-out sets, and nine baselines, giving the community a fair basis for comparing failure-search methods regardless of whether HAD²RL itself prevails.

Critical weaknesses

First. the two central motivations are not tested directly. The paper argues that D2RL fails for heterogeneous agents because criticality is shared rather than per-agent, and because the mask is hard rather than soft. Neither claim is isolated experimentally: the ablation only removes components of the full method, and the shared-set and hard-mask comparisons are deferred to future work. The results show that the package helps, not that heterogeneous criticality or soft weighting is the cause. Notably, GAE reconnection, adapted from D2RL, appears to be the dominant ingredient.

Second, headline gains are overstated. The "up to 28×" comes from two rare families with near-zero baselines.

Third, limited generality. All evidence comes from one system under test, one simulator, and one reward design. The shared reward already includes ratchet and potential shaping, which sits awkwardly with the introduction's criticism of dense reward shaping, and it is not shown that HAD²RL helps without it. Applicability to other collision-avoidance logics is claimed but untested.

Last, hand-crafted components. The inert-step rule and the λ schedule are manually specified, and settings differ by backbone (e.g., MAPPO uses a higher floor). This weakens the plug-in claim and raises the question of how much per-backbone tuning contributed to the results.

Questions for authors
  1. Per-agent vs. shared criticality. Can you run a variant of HAD²RL where both agents use the same weight (for example, computed from the joint event advantage)? This is the direct test of your first motivating limitation, and without it the "heterogeneous" part of the contribution is unsupported.

  2. Soft vs. hard masking. What happens if w is replaced by a 0/1 mask derived from the same event critic (equivalent to D2RL applied per agent)? Does the floor w_min actually matter, and how does performance change as w_min goes from 0 to 1?

  3. How sensitive are results to α, w_min, w_max, and the λ schedule?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.