- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed HMM, a hierarchical MARL framework based on a constraint manifold formulation, enabling coordinated multi-agent control while providing formal safety guarantees at every timestep during both training and execution. A high-level MAPPO policy uses a graph neural network to generate subgoals that help agents coordinate, while a fixed low-level constraint manifold controller modifies their actions to avoid collisions. The approach provides theoretical safety guarantees in the multi-agent setting and induces a stationary learning process with stable convergence behaviour. Also, the provided framework empirically outperforms other safety MARL baselines on lidar benchmarks, achieving state-of-the-art performance. It also demonstrates strong generalisation across task scales.
Critical strengths
-
The paper addresses the challenge of achieving both effective cooperation and safety in MARL, particularly in navigation tasks where agents must avoid collisions.
-
The paper provides empirical results by presenting a compositional safety theorem and a stationarity proposition. The mathematical framework provides an opportunity to understand the assumptions under which safety can be established. Strong theoratical analysis is provided.
-
The paper clearly Separates learned high-level coordination from fixed low-level safety control is an intuitive approach. It allows the policy to focus on reaching goals while safety is handled by a dedicated controller.
-
The paper provides evaluation across different environments.
-
Useful component ablations is done in the paper.
-
The paper also provides generalisation to larger systems.
Critical weaknesses
-
The safety theorem (1st theorem) is established for continuous-time dynamics under specific assumptions, but the experiments report residual safety violations. The paper could more clearly distinguish its theoretical guarantees from the safety achieved in the actual implementation.
-
The global safety guarantee relies on strong assumptions. It remains unclear how frequently these assumptions (exact local observations, feasible control inputs, and full-row-rank constraint Jacobians) hold in crowded multi-agent environments.
-
Assumption 4 (Appendix) restricts the number of simultaneously active independent constraints to the physical control dimension. For 2D agents, this permits at most two. It should be explained how the controller behaves when several agents or obstacles simultaneously impose active constraints.
-
The paper does not fully establish how the safety controller requires a drift term that includes neighbouring agents' dynamics. can be computed accurately from local observations, particularly with velocity-dependent safety constraints and unknown neighbouring control actions.
Questions for authors
See the weaknesses section
6 comments
- Weakness 3 (Limit on simultaneously active constraints, Assumption 4):
W: Assumption 4 (Appendix) restricts the number of simultaneously active independent constraints to the physical control dimension. For 2D agents, this permits at most two. It should be explained how the controller behaves when several agents or obstacles simultaneously impose active constraints.
When several agents or obstacles simultaneously activate constraints, each constraint has its own slack variable, which keeps the constraint manifold feasible.
If the physical control input cannot satisfy all active constraints simultaneously, the controller prioritizes the most urgent constraint. Lower priority constraints may temporarily use part of the clearance reserved by the safety margin. As the corresponding agents or obstacles move closer, their constraints become more urgent and automatically receive higher priority.
- Weakness 4 (Computing the drift term from local observations):
W: The paper does not fully establish how the safety controller requires a drift term that includes neighbouring agents' dynamics. can be computed accurately from local observations, particularly with velocity-dependent safety constraints and unknown neighbouring control actions.
Neighbor 's position and velocity are observed. Expanding the inter-agent drift term of Eq. (34), so enters agent 's constraint dynamics only through the velocity-dependent margin . Theorem 1 implicitly assumes this term is known. We will state this explicitly in the revision.
In our discrete-time implementation, all agents compute their constraint manifolds from the states measured at each step , and neighbors are modeled as moving at constant velocity within the step, so their current control actions are never required. Under explicit integration, first affects velocities at step , by at most , and positions at step , by at most . For agent pairs, the velocity change shifts the margin by at most . With the control interval s, these deviations are small and are accounted for by the additional safety margin.
I thank the authors for their detailed rebuttal and additional experiments. The clarifications are helpful, but I have three remaining questions:
-
The authors report rank deficiency at 8.6% of timesteps and infeasibility in 0.5% of evaluation episodes. What safety guarantees, if any, remain valid when these assumptions are violated?
-
When multiple constraints are active, the controller prioritizes the most urgent one. Can lower-priority safety constraints be violated, and if so, how does this affect the global safety guarantee?
-
Regarding the discrete-time safety bound, could the authors clarify whether the safety margin used in the experiments is sufficient to satisfy the derived bound, including errors introduced by the constant-velocity approximation of neighboring agents?
We thank the reviewer for their constructive feedback and support. Below, we clarify our approach and provide additional implementation details.
(a) Discrete-time safety bound Continuous-time constraint preservation does not by itself guarantee discrete-time safety. We therefore characterize the gap as follows.
Holding the control input constant between updates can introduce a small constraint violation. We show that this violation is at most O(Δt) at the sampling instants, where Δt is the control interval (0.03 s in our experiments), and that tightening the constraint by this bound restores h(sk)≤0.
(1) The control input computed at tk is held constant over [tk,tk+Δt). By Assumption 4, Ju has full row rank throughout a compact neighbourhood N of M, on which the control inputs and state velocities are bounded and, since our distance-based constraints are smooth, ψ and Ju are Lipschitz continuous. At the start of each step, the controller enforces c˙(tk)=−λck, and within the step c˙ changes by at most MΔt for some constant M. Integrating over one step gives
ck+1=(1−λΔt)ck+rk,∥rk∥≤MΔt2.
(2) Each step therefore adds an error of at most MΔt2 and scales the existing residual by 1−λΔt, so the error does not accumulate. With c0=0, 0<λΔt≤1 and μk≥0, summing the geometric series gives, for all k,
h(sk)≤∥ck∥≤λMΔt=ε(Δt),
provided Δt is small enough that the tube of radius ε(Δt) around M stays within N.
(3) Applying the controller to the tightened constraint h+ε(Δt), with the initial state satisfying it, therefore gives h(sk)≤0 at every sampling instant. For the distance constraint h=(rsafe+δ)2−∥pi−pj∥2, it suffices to increase the safety radius by Δr≥MΔt/(2λrsafe). In our implementation, the additional safety margin plays this role and absorbs the discretization error.
(b) Remaining violations in crowded scenes We observed that the remaining safety violations occurred mainly in crowded scenarios where the top-k restriction appears to be insufficient. Increasing k reduced the number of violations, but some persisted in densely crowded configurations. One possible explanation is that, in these configurations, no feasible action satisfies all constraints once the agents are that close.
We ran 10,000 additional evaluation episodes in the Spread environment to examine when the safety filter's assumptions hold.
Local observations are exact by construction in our simulator. In uncrowded scenarios, we observed no violations of the feasibility or rank assumptions. In crowded scenarios such as the 21-agent setting, no feasible control input existed in approximately 0.5% of evaluation episodes, and the constraint Jacobian was rank-deficient at approximately 8.6% of all time steps.
Notably, a rank deficiency does not immediately cause a collision. Recall the pairwise constraint
hij=(rijsafe+δij)2−∥pi−pj∥2.
The filter still enforces the most urgent constraint, while the remaining ones only temporarily consume part of the safety distance rijsafe+δij, and neighboring agents also take avoiding action. The safe rate therefore remains at 99.14% in the 21-agent setting.