acceptodds
Under review as a conference paper at ICLR 2027

Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control

Abstract

Multi-agent systems are widely used in safety-critical applications that require coordinated behavior under strict safety constraints. Existing approaches face a fundamental trade-off: learning-based methods achieve strong empirical performance but lack theoretical safety guarantees, while control-theoretic methods enforce safety but often lead to overly conservative and inefficient behaviors. We propose a hierarchical multi-agent reinforcement learning framework that enforces hard safety constraints under mild assumptions at low level via a constraint manifold, while enabling effective coordination through high-level policy learning. Our approach provides theoretical safety guarantees in the multi-agent setting and yields stationary learning dynamics, thereby enabling stable and efficient training. Empirically, our method achieves competitive performance while maintaining nearly perfect safety rates, and generalizes effectively to varying numbers of agents and obstacles.

Primary area:
reinforcement learning
Keywords:
Safety, Multi-agent RL

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KhYSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed HMM, a hierarchical MARL framework based on a constraint manifold formulation, enabling coordinated multi-agent control while providing formal safety guarantees at every timestep during both training and execution. A high-level MAPPO policy uses a graph neural network to generate subgoals that help agents coordinate, while a fixed low-level constraint manifold controller modifies their actions to avoid collisions. The approach provides theoretical safety guarantees in the multi-agent setting and induces a stationary learning process with stable convergence behaviour. Also, the provided framework empirically outperforms other safety MARL baselines on lidar benchmarks, achieving state-of-the-art performance. It also demonstrates strong generalisation across task scales.

Critical strengths
  1. The paper addresses the challenge of achieving both effective cooperation and safety in MARL, particularly in navigation tasks where agents must avoid collisions.

  2. The paper provides empirical results by presenting a compositional safety theorem and a stationarity proposition. The mathematical framework provides an opportunity to understand the assumptions under which safety can be established. Strong theoratical analysis is provided.

  3. The paper clearly Separates learned high-level coordination from fixed low-level safety control is an intuitive approach. It allows the policy to focus on reaching goals while safety is handled by a dedicated controller.

  4. The paper provides evaluation across different environments.

  5. Useful component ablations is done in the paper.

  6. The paper also provides generalisation to larger systems.

Critical weaknesses
  1. The safety theorem (1st theorem) is established for continuous-time dynamics under specific assumptions, but the experiments report residual safety violations. The paper could more clearly distinguish its theoretical guarantees from the safety achieved in the actual implementation.

  2. The global safety guarantee relies on strong assumptions. It remains unclear how frequently these assumptions (exact local observations, feasible control inputs, and full-row-rank constraint Jacobians) hold in crowded multi-agent environments.

  3. Assumption 4 (Appendix) restricts the number of simultaneously active independent constraints to the physical control dimension. For 2D agents, this permits at most two. It should be explained how the controller behaves when several agents or obstacles simultaneously impose active constraints.

  4. The paper does not fully establish how the safety controller requires a drift term that includes neighbouring agents' dynamics. can be computed accurately from local observations, particularly with velocity-dependent safety constraints and unknown neighbouring control actions.

Questions for authors

See the weaknesses section

6 comments

We thank the reviewer for their constructive feedback and support. Below, we clarify our approach and provide additional implementation details.

  1. Weakness 1 (Theoretical guarantees vs. empirical safety violations):

    W: The safety theorem (1st theorem) is established for continuous-time dynamics under specific assumptions, but the experiments report residual safety violations. The paper could more clearly distinguish its theoretical guarantees from the safety achieved in the actual implementation.

(a) Discrete-time safety bound Continuous-time constraint preservation does not by itself guarantee discrete-time safety. We therefore characterize the gap as follows.

Holding the control input constant between updates can introduce a small constraint violation. We show that this violation is at most O(Δt)O(\Delta t) at the sampling instants, where Δt\Delta t is the control interval (0.030.03 s in our experiments), and that tightening the constraint by this bound restores h(sk)≤0h(s_k) \le 0.

(1) The control input computed at tkt_k is held constant over [tk,tk+Δt)[t_k, t_k+\Delta t). By Assumption 4, JuJ_u has full row rank throughout a compact neighbourhood N\mathcal{N} of M\mathcal{M}, on which the control inputs and state velocities are bounded and, since our distance-based constraints are smooth, ψ\psi and JuJ_u are Lipschitz continuous. At the start of each step, the controller enforces c˙(tk)=−λck\dot{c}(t_k) = -\lambda c_k, and within the step c˙\dot{c} changes by at most MΔtM\Delta t for some constant MM. Integrating over one step gives

ck+1=(1−λΔt)ck+rk,∥rk∥≤MΔt2.c_{k+1} = (1-\lambda\Delta t)c_k + r_k, \qquad \lVert r_k \rVert \le M\Delta t^2.

(2) Each step therefore adds an error of at most MΔt2M\Delta t^2 and scales the existing residual by 1−λΔt1-\lambda\Delta t, so the error does not accumulate. With c0=0c_0=0, 0<λΔt≤10<\lambda\Delta t\le1 and μk≥0\mu_k\ge0, summing the geometric series gives, for all kk,

h(sk)≤∥ck∥≤MλΔt=ε(Δt),h(s_k) \le \lVert c_k \rVert \le \frac{M}{\lambda}\Delta t = \varepsilon(\Delta t),

provided Δt\Delta t is small enough that the tube of radius ε(Δt)\varepsilon(\Delta t) around M\mathcal{M} stays within N\mathcal{N}.

(3) Applying the controller to the tightened constraint h+ε(Δt)h+\varepsilon(\Delta t), with the initial state satisfying it, therefore gives h(sk)≤0h(s_k)\le0 at every sampling instant. For the distance constraint h=(rsafe+δ)2−∥pi−pj∥2h=(r_{\text{safe}}+\delta)^2-\lVert p_i-p_j\rVert^2, it suffices to increase the safety radius by Δr≥MΔt/(2λrsafe)\Delta r\ge M\Delta t/(2\lambda r_{\text{safe}}). In our implementation, the additional safety margin plays this role and absorbs the discretization error.

(b) Remaining violations in crowded scenes We observed that the remaining safety violations occurred mainly in crowded scenarios where the top-kk restriction appears to be insufficient. Increasing kk reduced the number of violations, but some persisted in densely crowded configurations. One possible explanation is that, in these configurations, no feasible action satisfies all constraints once the agents are that close.

  1. Weakness 2 (Strong assumptions behind the global safety guarantee):

    W: The global safety guarantee relies on strong assumptions. It remains unclear how frequently these assumptions (exact local observations, feasible control inputs, and full-row-rank constraint Jacobians) hold in crowded multi-agent environments.

We ran 10,000 additional evaluation episodes in the Spread environment to examine when the safety filter's assumptions hold.

Local observations are exact by construction in our simulator. In uncrowded scenarios, we observed no violations of the feasibility or rank assumptions. In crowded scenarios such as the 21-agent setting, no feasible control input existed in approximately 0.5% of evaluation episodes, and the constraint Jacobian was rank-deficient at approximately 8.6% of all time steps.

Notably, a rank deficiency does not immediately cause a collision. Recall the pairwise constraint

hij=(rijsafe+δij)2−∥pi−pj∥2.h_{ij} = \left(r_{ij}^{\mathrm{safe}} + \delta_{ij}\right)^2 - \left\lVert \mathbf{p}_i - \mathbf{p}_j \right\rVert^2.

The filter still enforces the most urgent constraint, while the remaining ones only temporarily consume part of the safety distance rijsafe+δijr_{ij}^{\mathrm{safe}} + \delta_{ij}, and neighboring agents also take avoiding action. The safe rate therefore remains at 99.14% in the 21-agent setting.

  1. Weakness 3 (Limit on simultaneously active constraints, Assumption 4):

    W: Assumption 4 (Appendix) restricts the number of simultaneously active independent constraints to the physical control dimension. For 2D agents, this permits at most two. It should be explained how the controller behaves when several agents or obstacles simultaneously impose active constraints.

When several agents or obstacles simultaneously activate constraints, each constraint has its own slack variable, which keeps the constraint manifold feasible.

If the physical control input cannot satisfy all active constraints simultaneously, the controller prioritizes the most urgent constraint. Lower priority constraints may temporarily use part of the clearance reserved by the safety margin. As the corresponding agents or obstacles move closer, their constraints become more urgent and automatically receive higher priority.

  1. Weakness 4 (Computing the drift term from local observations):

    W: The paper does not fully establish how the safety controller requires a drift term that includes neighbouring agents' dynamics. can be computed accurately from local observations, particularly with velocity-dependent safety constraints and unknown neighbouring control actions.

Neighbor jj's position and velocity are observed. Expanding the inter-agent drift term of Eq. (34), ∂hij∂sjs˙j=∂hij∂pjvj+∂hij∂vjaj,\frac{\partial h_{ij}}{\partial s_j}\dot{s}_j = \frac{\partial h_{ij}}{\partial p_j} v_j + \frac{\partial h_{ij}}{\partial v_j} a_j, so aja_j enters agent ii's constraint dynamics only through the velocity-dependent margin δij\delta_{ij}. Theorem 1 implicitly assumes this term is known. We will state this explicitly in the revision.

In our discrete-time implementation, all agents compute their constraint manifolds from the states measured at each step kk, and neighbors are modeled as moving at constant velocity within the step, so their current control actions are never required. Under explicit integration, aj(k)a_j(k) first affects velocities at step k+1k+1, by at most amax⁡Δta_{\max}\Delta t, and positions at step k+2k+2, by at most amax⁡Δt2a_{\max}\Delta t^2. For agent pairs, the velocity change shifts the margin δij\delta_{ij} by at most vapproach amax⁡Δt/aij=vapproachΔt/2v_{\text{approach}}\,a_{\max}\Delta t / a_{ij} = v_{\text{approach}}\Delta t / 2. With the control interval Δt=0.03\Delta t = 0.03 s, these deviations are small and are accounted for by the additional safety margin.

I thank the authors for their detailed rebuttal and additional experiments. The clarifications are helpful, but I have three remaining questions:

  1. The authors report rank deficiency at 8.6% of timesteps and infeasibility in 0.5% of evaluation episodes. What safety guarantees, if any, remain valid when these assumptions are violated?

  2. When multiple constraints are active, the controller prioritizes the most urgent one. Can lower-priority safety constraints be violated, and if so, how does this affect the global safety guarantee?

  3. Regarding the discrete-time safety bound, could the authors clarify whether the safety margin used in the experiments is sufficient to satisfy the derived bound, including errors introduced by the constant-velocity approximation of neighboring agents?

Reviewer_asxpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a hierarchical approach to safe multi-agent control. The low-level controller is implemented using a constraint manifold, and the high-level policy is learned under the CTDE scheme using MAPPO. The constraint manifold guarantees safety under the stated assumptions. The fixed low-level controller induces stationary high-level transition dynamics, requiring only the high-level policy to be learned. The empirical results validate the effectiveness of the proposed approach.

Critical strengths

Please find the strengths of the paper below:

i. The paper is well written and easy to follow. ii. As the authors considers pointwise safety in time this problem setting encapsulates safety more strictly than CMDP setting. iii. The idea of constraint maniform for safe control has been well studied in the control literature which justifies the motivation behind its choice for the low-level controller. iv. The choice of this low-level controller provides theoretical safety guarantee. v. The authors provides extensive empirical results to validate the merit of the proposed approach.

Critical weaknesses

Please find the weaknesses below:

i. The paper closely matches HMARL-CBF [1] with replacing the low level CBF controller with constrained manifold based controller. Hence the novelty of the approach is somewhat limited. ii. The model requires knowledge of the system dynamics making it limited to wide range of robotic application. iii. The experiments are primarily limited to locomotion tasks. There are ample literature tackling safe multi-agent control for navigation tasks. Hence, it raises question about the significance of the paper. iv. It is assumed that low level controller will always have a feasible input, however in reality, actuators have bounds which are not considered in the formulation. v. Proposition 1 concludes fixing the low-level controller removes nonstationarity caused as a result of simultaneously updating it with the high-level policy. However, this does not guarantee the convergence of MAPPO. vi. The method assumes access to explicitly defined safety constraint functions, which limits its applicability to settings where such constraints are unknown or difficult to specify analytically. vii. The low-level controller and its safety guarantees are derived in continuous time, whereas the problem formulation and implementation operate in discrete time. Continuous-time constraint preservation does not automatically guarantee safety under discrete-time execution, as finite integration steps can introduce constraint violations.

[1] H. M. Sabbir Ahmad, Ehsan Sabouni, Alexander Wasilkoff, Param Budhraja, Zijian Guo, Songyuan Zhang, Chuchu Fan, Christos Cassandras, and Wenchao Li. “HMARL-CBF—Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems.” Advances in Neural Information Processing Systems (NeurIPS), 2025.

Questions for authors

Please find my questions below:

i. Can you provide empirical comparison of the method with HMARL-CBF [1] as it is very similar? ii. How would the method apply to settings where the system dynamics are unknown or cannot be described analytically? iii. Could you discuss how the method could be extended to other robotic settings, such as manipulation or humanoid control, particularly where contact dynamics and complex safety constraints challenge the current formulation? iv. How does the low-level controller consider actuator bounds? v. How would the method apply to settings where safety constraints cannot be specified analytically? vi. Why does the low-level controller formulation use continuous time when original problem is defined in discrete time?

[1] H. M. Sabbir Ahmad, Ehsan Sabouni, Alexander Wasilkoff, Param Budhraja, Zijian Guo, Songyuan Zhang, Chuchu Fan, Christos Cassandras, and Wenchao Li. “HMARL-CBF—Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems.” Advances in Neural Information Processing Systems (NeurIPS), 2025.

5 comments

We thank the reviewer for their thoughtful feedback and suggestions. We have added HMARL-CBF as a baseline and provide additional clarifications about their approach below.

  1. Weakness i & Question i (Novelty and empirical comparison with HMARL-CBF):

    W: The paper closely matches HMARL-CBF [1] with replacing the low level CBF controller with constrained manifold based controller. Hence the novelty of the approach is somewhat limited. Q: Can you provide empirical comparison of the method with HMARL-CBF [1] as it is very similar?

We would like to emphasize that our main contribution is a multi-agent constraint manifold formulation for safe MARL that handles coupled safety constraints across agents, with per-agent guarantees composing into global safety. The hierarchical architecture supports this formulation by assigning constraint enforcement to a low-level controller and coordination to high-level policies, which resolves the deadlocks that the safety filter alone suffers from. To the best of our knowledge, this is the first work to do so, providing theoretical safety guarantees together with higher computational efficiency and stronger empirical performance than existing methods.

Because the HMARL-CBF implementation is not publicly available, we re-implemented it and evaluated it in the same environments; some implementation details may differ from the original. HMM achieves higher safety and success rates than HMARL-CBF in almost all cases.

EnvHMM safety rateHMM success rateHMARL-CBF safety rateHMARL-CBF success rate
LidarTarget99.67%71.93%99.07%70.82%
LidarSpread99.83%87.29%99.37%33.53%
LidarLine99.73%85.75%98.24%35.83%
LidarBicycleTarget98.58%64.67%96.55%66.30%
LinearDrone99.80%93.67%96.30%93.38%
CrazyFlie99.87%99.22%81.81%78.25%

HMM also achieves a 12–21× end-to-end training speedup over HMARL-CBF.

EnvHMM (ours) it/sHMARL-CBF it/sSpeedup
LidarTarget6.020.351~17×
LidarSpread5.800.337~17×
LidarLine5.840.343~17×
LidarBicycleTarget5.890.279~21×
LinearDrone (6 obstacles)1.990.166~12×
CrazyFlie (6 obstacles)1.980.134~15×

As an independent reference, Figure 7 of the HMARL-CBF paper [1] shows that DGPPO outperforms HMARL-CBF in the authors' own evaluation, and our method outperforms DGPPO in the same environments. Taken together, these results support our claim that our method achieves the strongest performance among the methods considered.

  1. Weakness ii & Question ii (Requires known system dynamics):

    W: The model requires knowledge of the system dynamics making it limited to wide range of robotic application.

    Q: How would the method apply to settings where the system dynamics are unknown or cannot be described analytically?

The known dynamics assumption is a principled design trade off, rather than a limitation. It is the assumption central to hard safety rather than soft safety. Hard safety guarantees generally require some knowledge of the system dynamics, since safety violations are governed by the true physical dynamics. This assumption is standard in safe control, including CBF, Hamilton–Jacobi reachability, and MPC based safety. In contrast, RL based methods such as DGPPO and Lagrangian MARL do not assume known dynamics and typically provide soft safety guarantees.

The framework could also be extended to learned or uncertain dynamics. The dynamics can be estimated using learning based or model based methods. In that case, however, the guarantee would generally weaken from theoretical safety guarantees to probabilistic or soft safety guarantees, depending on the accuracy and uncertainty of the learned dynamics model. We will add these discussions in the revision.

  1. Weakness iii & Question iii (Significance of the paper and experiments limited to locomotion):

    W: The experiments are primarily limited to locomotion tasks. There are ample literature tackling safe multi-agent control for navigation tasks. Hence, it raises question about the significance of the paper. Q: Could you discuss how the method could be extended to other robotic settings...

(1) Significance of the paper In our setting, agents act from local LiDAR observations only, with no access to a global map or to other agents' plans. All evaluations use randomly generated maps unseen during training, so the reported results measure generalization to new environments rather than performance on the training maps.

Under the environments and evaluation protocols we follow, our method provides theoretical safety guarantees, achieves the best overall performance among the evaluated methods, and maintains high safety rates on unseen maps. It also demonstrates strong generalization by maintaining high safety rates across different numbers of agents and obstacles on unseen maps. Therefore, our method is relevant for real world deployment, where maps are not known in advance and both robust safety and generalization are required.

(2) Other robotic settings Our method can also be applied to manipulation. It requires that the safety constraints be differentiable with respect to the system state and that each robot's control input affect only its own dynamics. Under these conditions, the method can handle joint limits, collision avoidance, and workspace boundaries. Prior work has used constraint manifolds in single agent manipulation [2]. We believe that HMM also has potential for extension to multi-robot manipulation tasks designed to satisfy the method's assumptions.

  1. Weakness iv & Question iv (Actuator bounds / feasibility):

    W: It is assumed that low level controller will always have a feasible input, however in reality, actuators have bounds which are not considered in the formulation.

    Q: How does the low-level controller consider actuator bounds?

In our experimental setup, agents operate under the same velocity and acceleration limits as in the DGPPO and HMARL-CBF runs. For example, in the 2D Target, Spread, and Line environments, velocity is capped at 0.5 along each axis, and acceleration is capped at 10. In practice, these bounds are rarely reached, and clipping caused no collisions in the training setting (3 agents and 3 obstacles).

  1. Weakness v (Proposition 1 does not guarantee MAPPO convergence):

    W: Proposition 1 concludes fixing the low-level controller removes nonstationarity caused as a result of simultaneously updating it with the high-level policy. However, this does not guarantee the convergence of MAPPO.

In hierarchical reinforcement learning, training both levels simultaneously causes the transition dynamics seen by the high level policy to change as the low level policy evolves. This can bias value estimates and destabilize training. By our method, we remove this source of nonstationarity and formulate the high level learning problem as a standard MARL problem with stationary transition dynamics.

We acknowledge that MAPPO has no theoretical convergence guarantee. The purpose of fixing the low-level controller is to improve training stability. We will clarify this in the revision.

  1. Weakness vi & Question v (Requires analytical safety constraints):

    W: The method assumes access to explicitly defined safety constraint functions, which limits its applicability to settings where such constraints are unknown or difficult to specify analytically.

    Q: How would the method apply to settings where safety constraints cannot be specified analytically?

We use explicitly defined constraints because they are what make hard safety guarantees possible. This requirement is standard across safe control: control barrier functions, Hamilton–Jacobi reachability, and MPC-based safety filters all require the safety constraints to be specified explicitly.

When constraints cannot be specified analytically, they could be learned from data and used in place of hh, so HMM could potentially be extended to such settings. The guarantee would then weaken from a hard guarantee to a probabilistic or soft one, depending on the accuracy of the learned constraints, as in constraint manifold methods with stochastic constraints [3].

  1. Weakness vii & Question vi (Continuous-time guarantees vs. discrete-time execution):

    W: The low-level controller and its safety guarantees are derived in continuous time, whereas the problem formulation and implementation operate in discrete time... Q: Why does the low-level controller formulation use continuous time when original problem is defined in discrete time?

In the real world, a robot’s physical state evolves continuously. It is therefore natural to formulate the constraint manifold in continuous time. However, the practical implementation is limited by the simulator’s discrete time steps and the finite control update rate. As a result, we evaluate the continuous-time control theory and update the control commands at discrete control instants.

The guarantee can extend to discrete-time control. The violation at sampling instants is at most O(Δt)O(\Delta t), where Δt\Delta t is the control interval (Δt=0.03\Delta t = 0.03 s in our experiments), and tightening the constraints by this amount restores h≤0h \le 0. We first bound the error introduced within one step, then show that it does not accumulate across steps, and finally remove it by tightening.

The control input computed at tkt_k is held constant over [tk,tk+Δt)[t_k, t_k+\Delta t). By Assumption 4, JuJ_u has full row rank throughout a compact neighbourhood N\mathcal{N} of M\mathcal{M}, so the control inputs and state velocities are bounded on N\mathcal{N}. The smoothness of the distance-based constraints further ensures that ψ\psi and JuJ_u are Lipschitz continuous on N\mathcal{N}.

(a) At tkt_k the controller gives c˙(tk)=−λck\dot{c}(t_k) = -\lambda c_k exactly, and within the step c˙\dot{c} drifts by at most MΔtM\Delta t for a constant MM. Integrating over the step gives

ck+1=(1−λΔt)ck+rk,∥rk∥≤MΔt2.c_{k+1} = (1 - \lambda\Delta t) c_k + r_k, \qquad \Vert r_k \Vert \le M\Delta t^2.

(b) This error does not accumulate, because the factor (1−λΔt)(1 - \lambda\Delta t) contracts the residual at every step. With c0=0c_0 = 0 and λΔt≤1\lambda\Delta t \le 1, summing the geometric series and using μk≥0\mu_k \ge 0 gives, for all kk,

h(sk)≤∥ck∥≤MλΔt,h(s_k) \le \Vert c_k \Vert \le \frac{M}{\lambda}\Delta t,

provided Δt\Delta t is small enough that the tube of this radius around M\mathcal{M} stays in N\mathcal{N}. We denote this bound by ε(Δt)\varepsilon(\Delta t).

(c) Running the controller on the tightened constraint h+ε(Δt)h + \varepsilon(\Delta t) therefore gives h(sk)≤0h(s_k) \le 0 for all kk. For h=(rsafe+δ)2−∥pi−pj∥2h = (r_{\text{safe}} + \delta)^2 - \Vert p_i - p_j \Vert^2, it suffices to enlarge the radius by Δr≥MΔt/(2λ rsafe)\Delta r \ge M\Delta t / (2\lambda\, r_{\text{safe}}). In our implementation, the additional safety margin plays this role and absorbs the discretization error.


[1] H. M. Sabbir Ahmad, Ehsan Sabouni, Alexander Wasilkoff, Param Budhraja, Zijian Guo, Songyuan Zhang, Chuchu Fan, Christos Cassandras, and Wenchao Li. "HMARL-CBF—Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems." NeurIPS, 2025.

[2] Liu, Tateo, Bou Ammar, Peters, Robot Reinforcement Learning on the Constraint Manifold,CoRL 2021

[3] Gu, Liu, Kshirsagar, Chen, Peters, Knoll, RoSCom: Robust Safe Reinforcement Learning on Stochastic Constraint Manifolds,IEEE Transactions on Automation Science and Engineering

Reviewer_GhhROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hierarchical Manifold Multi-Agent PPO (HMM), a safe multi-agent reinforcement learning framework that combines high-level cooperative subgoal learning with low-level constraint manifold control. The high-level policy employs graph-based MAPPO to coordinate agents, while the low-level controller projects nominal actions onto a constraint manifold to enforce safety without solving quadratic programs.

The authors provide theoretical results on compositional safety and the stationarity of the induced high-level learning problem. Experiments across six multi-agent navigation environments demonstrate high safety rates, competitive task performance, improved computational efficiency, and generalization to larger numbers of agents and obstacles.

Critical strengths
  1. The proposed framework combines learned cooperative subgoals with a fixed constraint manifold controller, avoiding repeated QP optimization. The reported 14.4× training-throughput improvement over the evaluated QP-based controller highlights its computational potential.
  2. Experiments across six environments demonstrate consistently high safety rates and competitive task success. The generalization experiments, which extend evaluation from three agents to substantially larger configurations, provide useful evidence of scalability.
Critical weaknesses
  1. The safety theorem relies on continuous-time dynamics, complete constraint enforcement, and full-rank conditions. However, the implementation uses discrete integration, top-k neighbor selection, and partial observations, resulting in observed safety violations. The extent to which the claimed hard safety guarantees apply to the actual system is therefore unclear.
  2. Although HMARL-CBF is a highly relevant prior approach, it is not directly included in the task-performance comparisons. The reported QP-controller speed comparison does not establish an end-to-end advantage over this hierarchical baseline. Moreover, the stationarity result alone does not substantiate the claimed convergence guarantees.
Questions for authors
  1. How can the theoretical safety guarantees be extended or qualified for discrete-time control, truncated neighbor sets, and partial observations?

  2. Can the authors provide direct comparisons with HMARL-CBF under matched environments and computational settings, including safety, success rate, and runtime?

  3. How are neighboring agents' dynamics handled in the drift-compensation term, particularly when velocity-dependent safety margins require information about neighboring accelerations?

4 comments

We sincerely thank the reviewers for their insightful comments and suggestions. Below, we clarify our theoretical safety guarantees and present a comparison with HMARL-CBF.

  1. Weakness 1 and Question 1 (Extending or qualifying the theoretical safety guarantees for discrete-time control)

1. The safety theorem relies on continuous-time dynamics... 1. How can the theoretical safety guarantees be extended or qualified for discrete-time control, truncated neighbor sets, and partial observations?

(a) Discrete-time control. The guarantee extends to discrete-time control with an explicit O(Δt)O(\Delta t) violation bound, where Δt\Delta t is the control interval (Δt=0.03\Delta t = 0.03 s in our experiments).

We hold the control input computed at tkt_k constant over [tk,tk+Δt)[t_k, t_k+\Delta t). By Assumption 4, JuJ_u has full row rank throughout a compact neighborhood N\mathcal{N} of M\mathcal{M}, so the control inputs and state velocities are bounded on N\mathcal{N}. The smoothness of the distance-based constraints further ensures that ψ\psi and JuJ_u are Lipschitz continuous on N\mathcal{N}. At tkt_k, the controller gives c˙(tk)=−λck\dot{c}(t_k) = -\lambda c_k exactly, and within the step c˙\dot{c} drifts by at most KΔtK\Delta t for a constant KK. Integrating gives

ck+1=(1−λΔt) ck+rk,∥rk∥≤KΔt2.c_{k+1} = (1 - \lambda\Delta t)\ c_k + r_k, \qquad \lVert r_k \rVert \le K\Delta t^2.

With c0=0c_0 = 0 and λΔt≤1\lambda\Delta t \le 1, summing the geometric series and using μk≥0\mu_k \ge 0 gives, for all kk,

h(sk)≤∥ck∥≤KλΔt=:ε(Δt),h(s_k) \le \lVert c_k \rVert \le \frac{K}{\lambda}\Delta t =: \varepsilon(\Delta t),

for any Δt\Delta t small enough that the ε\varepsilon-tube around M\mathcal{M} stays in N\mathcal{N}.

Running the controller on the tightened constraint h+ε(Δt)h + \varepsilon(\Delta t) therefore gives h(sk)≤0h(s_k) \le 0 for all kk. For h=(rsafe+δ)2−∥pi−pj∥2h = (r_{\text{safe}} + \delta)^2 - \lVert p_i - p_j \rVert^2, it suffices to enlarge the radius by Δr≥KΔt/(2λrsafe)\Delta r \ge K\Delta t / (2\lambda r_{\text{safe}}). In our implementation, the safety margin plays this role.

(b) Truncated neighbor sets. The guarantee extends whenever the number of entities within the activation range does not exceed kk. Theorem 1 holds for all selected neighbors. We initialize the slack for each newly selected entity as μ=−h\mu = -h, so its constraint starts on the manifold without a jump. Because pairwise constraints are symmetric, the controller enforces an inter-agent constraint as long as either agent selects the other.

(c) Partial observations. Agents observe the positions and velocities of entities within their sensing radius. The guarantee extends whenever the sensing radius exceeds the activation range plus the maximum distance by which two entities can close the gap between them in one control step. This ensures that agents observe each entity before its constraint becomes active. We discuss unobserved neighbor accelerations in Q3.

Empirically, the remaining unsafe cases arise mainly when the top-kk condition is violated. Increasing kk reduced the number of unsafe cases, though some remained. These cases occurred mainly in crowded scenes with 21 agents on a map of the same size as in the 3-agent setting. In such scenes, a safe action may not exist.

  1. Weakness 2 and Question 2 (Reimplementing HMARL-CBF as a baseline)

Although HMARL-CBF is a highly relevant prior approach... Can the authors provide direct comparisons with HMARL-CBF...

We initially omitted HMARL-CBF because its implementation is not publicly available, which makes it difficult to reproduce every detail. Following the reviewer's suggestion, we reimplemented HMARL-CBF and evaluated its safety, success rate, and computational efficiency in our experimental setting. The results show that our HMM achieves higher safety and success rates than HMARL-CBF in nearly all cases.

EnvHMM safety rateHMM success rateHMARL-CBF safety rateHMARL-CBF success rate
LidarTarget99.67%71.93%99.07%70.82%
LidarSpread99.83%87.29%99.37%33.53%
LidarLine99.73%85.75%98.24%35.83%
LidarBicycleTarget98.58%64.67%96.55%66.30%
LinearDrone99.80%93.67%96.30%93.38%
CrazyFlie99.87%99.22%81.81%78.25%

Our method achieves a 12–21× end-to-end speedup over HMARL-CBF.

EnvHMM (ours), it/sHMARL-CBF, it/sSpeedup
LidarTarget6.020.351~17×
LidarSpread5.800.337~17×
LidarLine5.840.343~17×
LidarBicycleTarget5.890.279~21×
LinearDrone obs=61.990.166~12×
CrazyFlie obs=61.980.134~15×

Figure 7 of the HMARL-CBF paper [1] shows that HMARL-CBF performs slightly worse than DGPPO. Our method consistently outperforms DGPPO in the same environments. We thank the reviewer for this suggestion and will include the HMARL-CBF comparison in the revision.

  1. Question 3: How are neighboring agents' dynamics handled in the drift-compensation term

Neighbor jj's position and velocity are observed (assumption (iv)). Expanding the inter-agent drift term of Eq. (34),

∂hij∂sjs˙j=∂hij∂pjvj+∂hij∂vjaj,\frac{\partial h_{ij}}{\partial s_j}\dot{s}_j = \frac{\partial h_{ij}}{\partial p_j} v_j + \frac{\partial h_{ij}}{\partial v_j} a_j, so aja_j enters agent ii's constraint dynamics only through the velocity-dependent margin δij\delta_{ij}. Theorem 1 implicitly assumes this term is known. We will state this explicitly in the revision.

In our discrete-time implementation, all agents compute their constraint manifold from the states measured at each step kk, and neighbors are modeled as moving at constant velocity within the step. Under explicit integration, aj(k)a_j(k) first affects velocities at step k+1k+1, by at most amax⁡Δta_{\max}\Delta t, and positions at step k+2k+2, by at most amax⁡Δt2a_{\max}\Delta t^2. The velocity change shifts the margin δij\delta_{ij} by at most vapproach amax⁡Δt/aij=vapproachΔt/2.v_{\text{approach}}\, a_{\max}\Delta t / a_{ij} = v_{\text{approach}}\Delta t / 2.

With the control interval Δt=0.03\Delta t = 0.03 s, these deviations remain below the additional safety margin, which therefore absorbs them.

The additional safety margin of 0.020.02 accounts for both the O(Δt)O(\Delta t) discretization error discussed in Q1 and the deviation caused by unmodeled neighbor accelerations, which is at most amax⁡Δt2a_{\max}\Delta t^2 in position and vapproachΔt/2v_{\text{approach}}\Delta t/2 in the margin δij\delta_{ij}.

[1] H. M. Sabbir Ahmad et al., "Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems," in Advances in Neural Information Processing Systems (NeurIPS), 2025.

Dear Reviewer,

Thank you so mush for your time and thoughtful review. We would appreciate hearing whether our responses and additional experiments have addressed your concerns. We would be happy to provide further clarification or answer any remaining questions.

Best regards,

The Authors

Dear reviewer, the discussion period is about to close. Please read the authors' response and let us know briefly which of your concerns remain. If you have any further questions for the authors, please post them soon so the authors have time to respond.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.