- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper addresses the problem that the stacked actor-update field in CTDE multi-agent training can contain asymmetric cross-agent response terms, which may induce cyclic or non-potential interaction dynamics. To mitigate this, the authors propose HPML (Hodge-Projected Multi-Agent Learning), which projects the realized joint update field onto the closest metric-gradient potential flow and uses the orthogonal residual to characterize the non-potential/circulation component. The paper develops continuous, sample-graph, and amortized neural realizations of the projection, and integrates HPML as a plug-in projection layer on top of MAPPO-style actor updates. The theoretical analysis includes projection orthogonality, residual energy, Lyapunov improvement of projected ascent, and VI-gap bounds with explicit residual terms. Experiments include controlled cyclic games, 2D/3D fields with known potential components, and CTDE benchmarks on Melting Pot.
Critical strengths
1.The use of Hodge projection to mitigate cyclic interaction dynamics is a highly novel approach. HPML operates directly on the realized learning field produced during training and returns an executable projected update direction. The combination of a Hodge-type projection, a realized multi-agent update field, and a CTDE plug-in is original and conceptually appealing.
2.Solid theoretical foundation. The paper provides an projection formulation, an orthogonal residual decomposition, a Pythagorean identity, and a non-potentiality measure. The projected-ascent analysis gives Lyapunov-style improvement, and the VI-gap bounds explicitly separate structural non-potentiality from projection approximation error. This separation is a valuable theoretical contribution.
Critical weaknesses
1.In environments with a very large number of steps, the results of some experiments are not sufficiently significant. Particularly in the later stages of the "Melting Pot" experiments, the proposed methods did not achieve optimal results in most cases.
2.Lacks comparisons with existing methods for addressing gradient conflicts in multi-agent RL. The author could consider the following comparison methods. 1) HPS (Fu et al., 2026) HypeMARL: Multi-Agent Reinforcement Learning For High-Dimensional, Parametric, and Distributed Systems, 2) PCGrad (Yu et al., 2020) Gradient Surgery for Multi-Task Learning .
Questions for authors
1.How much has the computational load increased? Please quantify the additional computational cost of HPML relative to the MAPPO backbone, including:1)Wall-clock time per training step and total training time,2) For HPML-Neural: the potential-network parameter count,3) How the projection overhead scales with the number of agents n.
2.Is the advantage of fewer environmental steps attributed to the increased computational complexity of the estimated potential function? Please clarify whether the observed sample-efficiency gain is due to the geometric effect of the Hodge projection or simply to additional per-update computation. Specifically: Under a compute-matched comparison (same total wall-clock or FLOPs), does HPML still require fewer environment steps than MAPPO to reach the same return?
2 comments
Response to training overhead, agent-count scaling, and compute-matched comparisons.
HPML adds computation to the actor-training phase, but its graph projection does not require solving a Laplacian whose dimension equals the total number of actor parameters.
Let be the concatenated actor dimension, the sketch dimension, the buffer size, the graph neighborhood degree, and the refresh interval.
For HPML-Graph, constructing a CountSketch representation of a new actor or update vector requires work. Straightforward pairwise neighbor construction requires per rebuild, and edge-flow construction requires .
The graph Poisson system has unknowns and graph edges. The submitted implementation constructs a dense normal-equation matrix and solves it using a direct linear solver. This requires memory for the matrix and up to time for the dense solve.
A sparse iterative implementation could reduce the matrix-vector cost to per iteration, but this is not the solver used in the submitted code. The graph projection is attempted once every actor-minibatch updates, after initialization of the sample buffer.
The number of agents affects the actor dimension , the cost of field estimation, and potentially the coverage quality of the sampled neighborhoods. It does not directly determine the dimension of the graph Laplacian.
HPML-Neural avoids the graph Poisson linear solve but still constructs a sample graph for edge-difference potential fitting. Its additional cost includes graph construction, neural optimization, and input-gradient evaluation. Its additional cost depends on the potential-network architecture and the number of inner optimization steps; the reported configuration uses two inner steps. The potential-network parameter count must be determined from the actual architecture, rather than inferred from the number of optimization steps.
Isolated HPML-Graph projection-kernel timing on CPU (mean s.e.).
| Agents | ms/projection | Amortized ms/update |
|---|---|---|
| 2 | ||
| 4 | ||
| 8 |
This is a standalone CPU kernel benchmark with 128 actor parameters per agent, sketch dimension 64, buffer size 64, , and ridge coefficient .
The values exclude environment interaction, PPO gradient computation, Adam, and neural potential fitting. They are therefore not end-to-end training-time measurements.
Finally, matched environment interactions and matched computational resources are distinct experimental controls. Our existing results support the former comparison. Without measured wall-clock times or FLOPs, they do not establish compute-matched superiority.
A compute-matched evaluation should compare return against both environment steps and cumulative training time, including a MAPPO baseline allowed to use the same total computation.
Manuscript revision. We make the graph and neural training costs explicit, distinguish projection frequency from per-update complexity, and qualify the sample-efficiency claim. Exact wall-clock overhead and neural-network parameter counts require measurements from the executed configurations and should not be replaced by theoretical complexity estimates.
Response to gradient-conflict methods and the geometric distinction.
The suggested gradient-conflict comparisons concern a different mathematical property from the one HPML targets.
PCGrad tests instantaneous gradient conflicts between objectives, whereas HPML concerns the spatial integrability of a joint update field.
In our implementation, each agent maintains its own actor parameters, while all compared algorithms use the same actor architecture. The actor parameter blocks are disjoint.
Consequently, when each agent's gradient is embedded in the full actor parameter space as
gi=(0,…,∇θiLi,…,0),
we have
\qquad i\neq j.$$ Thus, cross-agent PCGrad conflict tests based on parameter-space inner products vanish by construction in our setting and cannot diagnose the integrability of the joint update field. Nevertheless, the concatenated update field can still be non-integrable. For example, in our two-player quadratic game, $$g_1=(-x+\rho y,0), \qquad g_2=(0,-y-\rho x).$$ These vectors are orthogonal at every point, yet the stacked field satisfies $$\partial_y U_x-\partial_x U_y=2\rho.$$ Thus, absence of instantaneous gradient conflict does not imply absence of non-potential multi-agent dynamics. This demonstrates the fundamental distinction: PCGrad modifies the relative directions of simultaneous task gradients, whereas HPML analyzes the spatial variation and integrability of the joint update field. For clarity, HPS (Fu et al., 2026) studies gradient conflicts in shared parameters through hyperspherical parameter sharing, whereas HypeMARL (Botteghi et al.) uses hypernetwork-based parameterization for distributed control. These are distinct MARL approaches, and neither directly computes a Hodge projection of the joint actor-update field. These methods remain relevant to a broader discussion of MARL optimization, but they do not directly replace the geometric comparison provided by HPML. *Manuscript revision.* We clarify the distinction between instantaneous gradient alignment, parameter sharing, and joint-field integrability. The related-work discussion is extended to distinguish these mechanisms rather than grouping all update corrections under a single gradient-conflict interpretation. --- **Response to late-stage return and the interpretation of sample efficiency.** We acknowledge that late-stage performance is task-dependent and does not always reach the highest observed score. This is consistent with the scope of our empirical claim, which concerns aggregate improvements under matched environment-interaction budgets. We report improvements over a matched MAPPO backbone across the evaluated ten-scenario suite, not unconditional dominance on every task, seed, or training iteration. The reported macro-average normalized returns are $78.1$ for MAPPO, $98.7$ for HPML-Graph, and $98.2$ for HPML-Neural. The performance differences are scenario-dependent. For example, MAPPO already obtains near-ceiling normalized scores on some Collaborative Cooking scenarios, leaving little scope for further improvement. HPML also does not dominate MAPPO in every individual scenario. The relevant conclusion is therefore improvement under the reported matched environment-interaction budget, not convergence to a globally optimal policy or universal late-stage dominance. Importantly, the reported scores are normalized training-population returns. They should not be interpreted as original-game VI gaps, absolute optimality certificates, or directly comparable scores under a different Melting Pot evaluation protocol. ---