acceptodds
Under review as a conference paper at ICLR 2027

Hodge-Projected Multi-Agent Learning: Reducing Cyclic Policy Dynamics via Potential-Flow Projection

Abstract

General-sum multi-agent learning is often governed by a stacked update field in which each agent's policy update changes the optimization landscape faced by the others. This coupling can entangle an integrable component of collective improvement with cyclic interaction dynamics, leading to slow or unstable multi-agent learning. Existing approaches, such as regularization, credit assignment, and consensus methods, stabilize MARL through local or algorithmic modifications; HPML complements them by projecting the joint update field onto a metric-gradient component. We introduce HPML (Hodge-Projected Multi-agent Learning), which views the joint update field of a multi-agent system as an element of an space of vector fields and computes a Hodge-type projection onto the closest metric-gradient potential flow. HPML follows the projected component as the update direction, yielding the closest metric-gradient field under the chosen metric and sampling measure. The projection is defined variationally, characterized by a Poisson-type equation, and implemented through graph-based and amortized neural realizations that recover projected directions from samples. We show that the projected dynamics admit a Lyapunov potential and yield equilibrium-gap bounds with an explicit additive non-potentiality term. Controlled experiments validate circulation suppression and component recovery, and CTDE benchmarks show higher final normalized return for HPML-MAPPO than the matched MAPPO baseline on the evaluated suite.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, General-Sum Games, Hodge Decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_WsSTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses the problem that the stacked actor-update field in CTDE multi-agent training can contain asymmetric cross-agent response terms, which may induce cyclic or non-potential interaction dynamics. To mitigate this, the authors propose HPML (Hodge-Projected Multi-Agent Learning), which projects the realized joint update field onto the closest metric-gradient potential flow and uses the orthogonal residual to characterize the non-potential/circulation component. The paper develops continuous, sample-graph, and amortized neural realizations of the projection, and integrates HPML as a plug-in projection layer on top of MAPPO-style actor updates. The theoretical analysis includes projection orthogonality, residual energy, Lyapunov improvement of projected ascent, and VI-gap bounds with explicit residual terms. Experiments include controlled cyclic games, 2D/3D fields with known potential components, and CTDE benchmarks on Melting Pot.

Critical strengths

1.The use of Hodge projection to mitigate cyclic interaction dynamics is a highly novel approach. HPML operates directly on the realized learning field produced during training and returns an executable projected update direction. The combination of a Hodge-type projection, a realized multi-agent update field, and a CTDE plug-in is original and conceptually appealing.

2.Solid theoretical foundation. The paper provides an L2(M,μ)L^2(M, \mu) projection formulation, an orthogonal residual decomposition, a Pythagorean identity, and a non-potentiality measure. The projected-ascent analysis gives Lyapunov-style improvement, and the VI-gap bounds explicitly separate structural non-potentiality from projection approximation error. This separation is a valuable theoretical contribution.

Critical weaknesses

1.In environments with a very large number of steps, the results of some experiments are not sufficiently significant. Particularly in the later stages of the "Melting Pot" experiments, the proposed methods did not achieve optimal results in most cases.

2.Lacks comparisons with existing methods for addressing gradient conflicts in multi-agent RL. The author could consider the following comparison methods. 1) HPS (Fu et al., 2026) HypeMARL: Multi-Agent Reinforcement Learning For High-Dimensional, Parametric, and Distributed Systems, 2) PCGrad (Yu et al., 2020) Gradient Surgery for Multi-Task Learning .

Questions for authors

1.How much has the computational load increased? Please quantify the additional computational cost of HPML relative to the MAPPO backbone, including:1)Wall-clock time per training step and total training time,2) For HPML-Neural: the potential-network parameter count,3) How the projection overhead scales with the number of agents n.

2.Is the advantage of fewer environmental steps attributed to the increased computational complexity of the estimated potential function? Please clarify whether the observed sample-efficiency gain is due to the geometric effect of the Hodge projection or simply to additional per-update computation. Specifically: Under a compute-matched comparison (same total wall-clock or FLOPs), does HPML still require fewer environment steps than MAPPO to reach the same return?

2 comments

Response to gradient-conflict methods and the geometric distinction.

The suggested gradient-conflict comparisons concern a different mathematical property from the one HPML targets.

PCGrad tests instantaneous gradient conflicts between objectives, whereas HPML concerns the spatial integrability of a joint update field.

In our implementation, each agent maintains its own actor parameters, while all compared algorithms use the same actor architecture. The actor parameter blocks are disjoint.

Consequently, when each agent's gradient is embedded in the full actor parameter space as

gi=(0,…,∇θiLi,…,0),g_i=(0,\ldots,\nabla_{\theta_i}L_i,\ldots,0),

we have

\qquad i\neq j.$$ Thus, cross-agent PCGrad conflict tests based on parameter-space inner products vanish by construction in our setting and cannot diagnose the integrability of the joint update field. Nevertheless, the concatenated update field can still be non-integrable. For example, in our two-player quadratic game, $$g_1=(-x+\rho y,0), \qquad g_2=(0,-y-\rho x).$$ These vectors are orthogonal at every point, yet the stacked field satisfies $$\partial_y U_x-\partial_x U_y=2\rho.$$ Thus, absence of instantaneous gradient conflict does not imply absence of non-potential multi-agent dynamics. This demonstrates the fundamental distinction: PCGrad modifies the relative directions of simultaneous task gradients, whereas HPML analyzes the spatial variation and integrability of the joint update field. For clarity, HPS (Fu et al., 2026) studies gradient conflicts in shared parameters through hyperspherical parameter sharing, whereas HypeMARL (Botteghi et al.) uses hypernetwork-based parameterization for distributed control. These are distinct MARL approaches, and neither directly computes a Hodge projection of the joint actor-update field. These methods remain relevant to a broader discussion of MARL optimization, but they do not directly replace the geometric comparison provided by HPML. *Manuscript revision.* We clarify the distinction between instantaneous gradient alignment, parameter sharing, and joint-field integrability. The related-work discussion is extended to distinguish these mechanisms rather than grouping all update corrections under a single gradient-conflict interpretation. --- **Response to late-stage return and the interpretation of sample efficiency.** We acknowledge that late-stage performance is task-dependent and does not always reach the highest observed score. This is consistent with the scope of our empirical claim, which concerns aggregate improvements under matched environment-interaction budgets. We report improvements over a matched MAPPO backbone across the evaluated ten-scenario suite, not unconditional dominance on every task, seed, or training iteration. The reported macro-average normalized returns are $78.1$ for MAPPO, $98.7$ for HPML-Graph, and $98.2$ for HPML-Neural. The performance differences are scenario-dependent. For example, MAPPO already obtains near-ceiling normalized scores on some Collaborative Cooking scenarios, leaving little scope for further improvement. HPML also does not dominate MAPPO in every individual scenario. The relevant conclusion is therefore improvement under the reported matched environment-interaction budget, not convergence to a globally optimal policy or universal late-stage dominance. Importantly, the reported scores are normalized training-population returns. They should not be interpreted as original-game VI gaps, absolute optimality certificates, or directly comparable scores under a different Melting Pot evaluation protocol. ---

Response to training overhead, agent-count scaling, and compute-matched comparisons.

HPML adds computation to the actor-training phase, but its graph projection does not require solving a Laplacian whose dimension equals the total number of actor parameters.

Let d=∑idid=\sum_i d_i be the concatenated actor dimension, ss the sketch dimension, NN the buffer size, kk the graph neighborhood degree, and HH the refresh interval.

For HPML-Graph, constructing a CountSketch representation of a new actor or update vector requires O(d)O(d) work. Straightforward pairwise neighbor construction requires O(N2s)O(N^2s) per rebuild, and edge-flow construction requires O(Nks)O(Nks).

The graph Poisson system has NN unknowns and O(Nk)O(Nk) graph edges. The submitted implementation constructs a dense N×NN\times N normal-equation matrix and solves it using a direct linear solver. This requires O(N2)O(N^2) memory for the matrix and up to O(N3)O(N^3) time for the dense solve.

A sparse iterative implementation could reduce the matrix-vector cost to O(Nk)O(Nk) per iteration, but this is not the solver used in the submitted code. The graph projection is attempted once every H=8H=8 actor-minibatch updates, after initialization of the sample buffer.

The number of agents affects the actor dimension dd, the cost of field estimation, and potentially the coverage quality of the sampled neighborhoods. It does not directly determine the dimension of the graph Laplacian.

HPML-Neural avoids the graph Poisson linear solve but still constructs a sample graph for edge-difference potential fitting. Its additional cost includes graph construction, neural optimization, and input-gradient evaluation. Its additional cost depends on the potential-network architecture and the number of inner optimization steps; the reported configuration uses two inner steps. The potential-network parameter count must be determined from the actual architecture, rather than inferred from the number of optimization steps.

Isolated HPML-Graph projection-kernel timing on CPU (mean ±\pm s.e.).

Agentsms/projectionAmortized ms/update
21.690±0.0341.690\pm0.0340.211±0.0040.211\pm0.004
41.692±0.0301.692\pm0.0300.211±0.0040.211\pm0.004
81.738±0.0381.738\pm0.0380.217±0.0050.217\pm0.005

This is a standalone CPU kernel benchmark with 128 actor parameters per agent, sketch dimension 64, buffer size 64, k=4k=4, and ridge coefficient 10−410^{-4}.

The values exclude environment interaction, PPO gradient computation, Adam, and neural potential fitting. They are therefore not end-to-end training-time measurements.

Finally, matched environment interactions and matched computational resources are distinct experimental controls. Our existing results support the former comparison. Without measured wall-clock times or FLOPs, they do not establish compute-matched superiority.

A compute-matched evaluation should compare return against both environment steps and cumulative training time, including a MAPPO baseline allowed to use the same total computation.

Manuscript revision. We make the graph and neural training costs explicit, distinguish projection frequency from per-update complexity, and qualify the sample-efficiency claim. Exact wall-clock overhead and neural-network parameter counts require measurements from the executed configurations and should not be replaced by theoretical complexity estimates.

Reviewer_yfhTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies cyclic learning dynamics in general-sum multi-agent reinforcement learning. It proposes HPML, which treats the stacked update field of all agents as a vector field and replaces it with its closest gradient field under an L2 projection. The non-potential residual is discarded. The projection is computed from recent policy or parameter iterates, either by solving a Poisson system on a k-nearest-neighbour graph or by fitting a neural scalar potential. Under smoothness assumptions, the projected dynamics increase the fitted potential monotonically, and the duality gap of the original game is bounded by a stationarity term plus a term proportional to the residual. Experiments include four controlled fields and ten Melting Pot environments, where HPML-MAPPO reports a higher normalised return than IA2C, IPPO, MAPPO, HAPPO and COMA.

Critical strengths

S1. The paper formulates the non-potential part of a multi-agent update field as the residual of an L2 projection onto gradient fields (Definition 3.1 and Corollary 3.3), a global counterpart to local Jacobian diagnostics. The controlled 2D and 3D fields, whose potential components are known, show that the graph projection recovers the intended component. These are the main points in the paper's favour.

Critical weaknesses

W1. The empirical comparison relies on baselines from 2022 or earlier, omits the standard cooperative benchmarks, and does not follow the Melting Pot evaluation protocol. The baselines are IA2C, IPPO, MAPPO, HAPPO and COMA, and COMA scores 0.0 on all ten environments (Table 3). The paper's motivation, instability from simultaneously updated agents, is the subject of the heterogeneous-agent trust-region line. Kuba et al. (ICLR 2022) introduced this line for exactly this reason and proved monotonic improvement of the joint return. Later work such as A2PO (ICLR 2023), HASAC (ICLR 2024) and MAGPO (ICLR 2026) retains such guarantees. Only HAPPO appears, as a baseline. These methods are also evaluated on standard cooperative benchmarks. A2PO and HASAC report results on Multi-Agent MuJoCo, SMAC, Google Research Football and MPE, HASAC also on Bi-DexHands, and MAGPO on SMACv2 tasks and MPE. Although the paper presents HPML as a plug-in for MAPPO-style training, it is evaluated only on Melting Pot and four controlled fields. For general-sum learning dynamics, recent opponent-shaping methods such as LOQA (ICLR 2024) and Advantage Alignment (ICLR 2025) are neither discussed nor compared. Advantage Alignment was evaluated on Melting Pot with the focal per-capita return against background populations and alongside the published Melting Pot baselines. The submission reports no focal or background-population evaluation, and its normalised return is never defined. Each column of Table 3 peaks at exactly 100.0, which suggests normalisation by the best method in each environment, and raw returns are not reported. Without current baselines and the standard protocol, the gains over MAPPO do not place the method against the state of the art.

W2. The released code does not implement the method described in the paper. The supplementary hpml.py computes the projection in a CountSketch compression of the actor parameters and maps the result back, a step the paper never mentions. The projected direction replaces the gradient only on every eighth minibatch update, and the other seven updates use the plain MAPPO gradient (Table 2 lists a refresh rate of 8). Algorithm 4, by contrast, applies the projected direction at every update. The recovered direction is a ridge solution in the span of four displacement vectors between the current parameters and stored past iterates in the sketched space, as the code comment states. This differs substantially from the L2 projection of the joint field analysed in Sections 3 to 6. The supplement contains three Python files whose imported modules (models, utils) are missing, and it has no licence or documentation, contrary to Appendix A.2. The reported gains therefore cannot be attributed to the analysed projection, and the results cannot be reproduced from the submission.

W3. The theory guarantees ascent on a fitted surrogate, and its link to the reported gains is not shown. Theorem 6.4 is the standard sufficient-increase property of projected gradient ascent on an L-smooth function, applied to the fitted potential. This potential is a least-squares fit to the update field and is not the return of any agent. For the original game, Theorem 6.7 adds a term that grows with the residual, and the paper states that a large residual yields no strong equilibrium claim. The guarantee therefore weakens exactly where cycling is strong, which is the case the method targets. The benchmark results do not close this gap. No residual is reported for Melting Pot (Appendix A.5). The gains are therefore not connected to reduced circulation. The largest gain, 100.0 versus 42.1 for MAPPO on Collaborative Cooking: Crowded, comes from a fully cooperative task. There the exact per-agent gradients stack into the gradient of the team return, and Remarks E.3 and E.4 state that a residual arises only when the realised update departs from this gradient.

Questions for authors

Q1 (clarification). Which code produced Tables 1 and 3? Please state whether the projection was applied at every update or at every eighth, and how the CountSketch dimension and the buffer size were set. The missing modules are needed to reproduce the results.

Q2 (requested experiments). How does HPML-MAPPO compare with recent methods such as HASAC, MAGPO and Advantage Alignment? Results on the standard cooperative benchmarks, namely Multi-Agent MuJoCo, SMACv2, Google Research Football, MPE and Bi-DexHands, are needed, together with Melting Pot results under its evaluation protocol with focal per-capita and raw returns.

Q3 (clarification). Can the authors report the residual or NonPot during Melting Pot training for MAPPO and HPML-MAPPO? This would show whether the gains relate to reduced circulation, in particular on the fully cooperative Collaborative Cooking tasks.

4 comments

Response to novelty and comparisons with recent multi-agent learning methods.

We appreciate the comparison with recent MARL policy-improvement and opponent-shaping approaches. Although these methods address related training challenges, their optimization objectives differ from HPML's projection of a supplied joint update field.

HAPPO and related heterogeneous-agent policy optimization methods establish policy-improvement guarantees under their respective assumptions, primarily for cooperative objectives. A2PO, HASAC, and MAGPO develop different mechanisms for policy optimization, maximum-entropy learning, or centralized guidance. LOQA and Advantage Alignment instead explicitly account for, or influence, opponent responses.

HPML addresses a different property: the global integrability of a supplied joint learning-update field. Its defining objective is

\mathbb E_{\mu} \left[ \|U-\nabla_M\Phi\|_M^2 \right].$$ This computes a closest metric-gradient component and an orthogonal non-potential residual. Neither ordered policy improvement, centralized guidance, nor opponent shaping is equivalent to this projection problem. The distinction is substantive, rather than terminological. A method can guarantee improvement of a cooperative return under its update assumptions without characterizing whether an arbitrary supplied joint update field is integrable. Conversely, our projected-potential Lyapunov guarantee is not a monotonic joint-return guarantee. We make this separation explicit rather than claiming the two guarantees are interchangeable. The empirical comparison raises a separate issue. Our experiments were designed primarily to isolate the modification to a matched MAPPO training backbone. They do not constitute a comprehensive state-of-the-art evaluation against HASAC, MAGPO, or Advantage Alignment. Likewise, the focal-agent evaluation against background populations used in the standard Melting Pot protocol differs from our training-population evaluation. Scores from these protocols cannot be directly juxtaposed without controlling the evaluated population, reward aggregation, and normalization. The absence of these additional benchmarks limits the empirical scope, but it does not eliminate the mathematical distinction between potential-flow projection and the optimization objectives of the cited methods. *Manuscript revision.* We clarify the relationship to heterogeneous-agent policy optimization, centralized-guidance methods, opponent shaping, and operator correction. We explicitly restrict the benchmark claim to the evaluated Melting Pot training setup and matched MAPPO backbone, rather than claiming comprehensive state-of-the-art dominance. ---

Response to the CountSketch implementation, projection frequency, and code consistency.

We also clarify that the SharedDiscreteActor class name refers to the common actor architecture used across the compared algorithms, not parameter sharing between agents. Each agent maintains an independent set of actor parameters.

We appreciate the implementation-level distinction raised in the review. The practical implementation retains the graph edge-space projection mechanism in compressed coordinates, but uses an intermittent, inexact realization rather than the ideal full-space projection at every update.

The implementation uses an approximate, intermittent realization of the graph projection. It constructs a graph from CountSketch-compressed actor-parameter and update samples, computes the conservative component of the resulting edge flow, performs local ridge lifting, and maps the direction back into actor parameter space.

The projected direction is used every eighth actor-minibatch update. The other seven minibatches use the ordinary PPO actor direction. Therefore, the method is not an exact original-space Hodge projection applied at every update. It is an intermittent sketched graph-projection layer.

This is a meaningful approximation of the proposed geometric operation, although its approximation error must be distinguished from the ideal projection. In particular, CountSketch distortion, limited neighborhood rank, ridge bias, historical drift, and optimizer state all affect the relationship between the executed update and the continuous field.

The fact that the practical implementation is approximate does not negate the continuous projection formulation or the exact edge-space orthogonality theorem. However, it does prevent the fixed-potential Lyapunov theorem from being interpreted as an unconditional guarantee for the executed Adam updates.

We have therefore revised Algorithm 4 to describe the implemented schedule explicitly, including CountSketch, buffered field samples, local lifting, projection refresh, and ordinary PPO steps between refreshes.

The reproducibility concern is separate from this mathematical distinction. A supplementary release that omits imported modules, dependencies, or execution instructions is incomplete, regardless of the correctness of the proposed projection. These artifacts must be provided to make the reported results independently reproducible.

Manuscript revision. Algorithm 4 and its implementation discussion now distinguish exact continuous projection, graph edge-space projection, sketched lifting, and actual optimizer execution. The stated refresh interval is eight minibatch updates. The supplementary material includes the core implementations of HPML and the corresponding MAPPO components, allowing reviewers to inspect the main algorithmic modifications.

We also provide the experimental settings, hyperparameters, and implementation details in the appendix to facilitate reproducibility. The complete training and evaluation code, including experiment configurations, will be released with the final published version.


Response to the original-game guarantees and fully cooperative tasks.

The fixed-potential Lyapunov result does not, by itself, imply improvement in the original game. Our analysis therefore includes separate VI-gap guarantees that quantify the relationship to the original operator:

Gap⁡F(x)≤Gap⁡Φ(x)+D∥R(x)∥.\operatorname{Gap}_{F}(x) \le \operatorname{Gap}_{\Phi}(x) + D\|R(x)\|.

The residual term is mathematically necessary. An orthogonal projection does not generally preserve every strategic incentive in the original game. Removing the residual from this inequality would incorrectly imply that projected stationarity always yields an equilibrium of the original game.

Crucially, a large global residual bound does not imply a large pointwise residual at the point reached by the projected dynamics.

The revised quadratic-game experiment illustrates that a residual-dependent worst-case bound does not preclude improvement in the original game's VI gap, even under strong rotational coupling. With ρ=4\rho=4, simultaneous ascent has an exact original-game VI gap of 3.90823.9082, compared with 0.10780.1078 for ideal HPML and 0.88750.8875 for norm-matched simultaneous ascent.

This is improvement of the original VI metric, not merely the fitted potential or a circulation statistic. Extragradient performs even better at this setting, so the result establishes a distinct possible benefit of potential projection, not superiority over all game-optimization methods.

The fully cooperative Collaborative Cooking tasks require a different argument. If every agent follows the exact population gradient of the same team return JteamJ_{\mathrm{team}}, then

U(θ)=∇θJteam(θ)U(\theta)=\nabla_\theta J_{\mathrm{team}}(\theta)

is conservative, and its non-potential residual is zero. We do not dispute this fact.

However, the implemented PPO update is not necessarily the exact population gradient. Clipping, stale advantages, critic approximation, adaptive optimizer state, and intermittent projection may produce a different realized training rule. Whether this rule has a substantial non-potential component in Collaborative Cooking must be established from field diagnostics, not inferred from shared rewards or from improved returns alone.

The controlled experiments provide direct evidence of circulation suppression on analytically specified fields. In contrast, the Melting Pot experiments evaluate the effect of the complete training procedure on normalized episodic return.

These are distinct empirical claims. The reported Melting Pot evaluations do not independently measure the circulation of the online update field. Therefore, we do not interpret higher task returns as direct measurements of reduced non-potentiality.

Consequently, neither conclusion suggested by the task label is justified: shared rewards do not guarantee that every realized training update follows the exact team-return gradient, and improved returns do not establish that circulation removal caused the improvement.

Manuscript revision. We retain the exact original-operator VI-gap experiment, clarify the scope of the residual-dependent guarantee, and distinguish theoretical mechanism evidence from benchmark return evidence. We do not attribute the Collaborative Cooking gains uniquely to circulation removal without direct residual and implementation-control measurements.

The reviewer thanks the authors for the detailed reply and for the revised quadratic-game experiment. The reply concedes the substance of each weakness. It then narrows the claims of the paper rather than supplying the evidence the weaknesses asked for. The reviewer's assessment is unchanged.

On W1, the reviewer remains unpersuaded by the argument that HPML addresses a problem distinct from policy improvement and from opponent shaping. That argument already appears in Appendix C under "Relation to the closest decomposition frameworks", and the reply therefore restates the positioning of the submission rather than answering the concern. A baseline is determined by the measurement a paper reports, not by the object a method decomposes. Table 3 reports a normalised return over ten Melting Pot environments, and the omitted methods compete on that measurement. Appendix C also states that two algorithms on the same payoff game have different HPML decompositions as they induce different learning fields. HASAC, MAGPO and A2PO are on that account alternative supplying algorithms rather than alternatives to HPML, and the submission evaluates a single one. The definition of normalised return, the raw returns, the exactly equal column maxima of 100.0 and the focal evaluation all remain outstanding.

On W2, the reply confirms every point of the weakness. The revision records the discrepancy in Algorithm 4 rather than removing it. This leaves Theorem 6.4 addressing the exact projection at every update while the reported experiments run the intermittent sketched variant. Q1 asked which code produced Tables 1 and 3, and that question is still open. One answer is that the supplementary implementation produced them, in which case the reported numbers measure the sketched variant and the fixed-potential result does not apply to them. The other is that it did not, in which case the submission holds nothing that reproduces the results. The CountSketch dimension and the buffer size are not stated. The missing modules are deferred to the published version and stay unavailable for assessment within this review.

On W3, the revised quadratic game is a responsive addition, as it evaluates a gap of the original operator rather than of the fitted surrogate. Its scope is the controlled setting. The residual requested in Q3 was not reported for Melting Pot, and the authors now decline to attribute the Collaborative Cooking gains to reduced circulation. The evidence for the mechanism rests on the synthetic fields alone, and the benchmark results stand without a stated cause. The same table also places extragradient ahead of HPML at rho = 4. The revised experiment therefore does not persuade the reviewer that the projection accounts for the reported advantage.

The reviewer would reconsider the rating on three items within the discussion period. These are the comparisons requested in Q2 under the Melting Pot evaluation protocol, the residual trace requested in Q3, and a complete supplement that runs the method the theory analyses.

Reviewer_LsJ9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Hodge-Projected Multi-Agent Learning (HPML), which replaces a joint learning-update field with its closest metric-gradient field under a chosen sampling measure. It develops a continuous variational formulation, a graph-based projection with local direction reconstruction, and an amortized neural-potential implementation. For a fixed smooth potential, the paper establishes projected-ascent improvement and bounds the original operator’s variational-inequality gap using separate non-potentiality and approximation-error terms.

Validation comprises controlled low-dimensional examples and a ten-scenario Melting Pot evaluation. Using a matched MAPPO backbone, both HPML variants report substantially higher macro-average normalized return across five seeds.

Critical strengths
  1. The analysis explicitly accounts for the consequences of removing part of the original update field. Theorems 6.7–6.8 retain an additive residual term when assessing the original VI operator and distinguish this from projection approximation error. The paper also correctly limits the relationship between first-order VI solutions and Nash equilibria. These qualifications make the theoretical interpretation more useful than an unconditional claim that suppressing cycles solves the original game.

  2. The matched MAPPO comparison reports substantial improvements across several task types. Section 7.2 and Appendix A.2 match actor parameterization, critic, rollout budget, and observation interface between MAPPO and HPML. Table 3 reports macro-average normalized returns of 78.1 ± 3.3 for MAPPO, 98.7 ± 3.0 for HPML-Graph, and 98.2 ± 1.6 for HPML-Neural. Together with the controlled examples, these results provide a promising empirical motivation for further investigation.

Critical weaknesses
  1. The bridge from conservative graph flows to the actual actor-update field is not established. The graph orthogonality results concern edge flows, whereas Eqs. (3)–(4) reconstruct each node’s direction independently through a local ridge fit. For M = I, that direction lies in the span of neighboring parameter displacements. Consequently, the procedure can alter an already integrable field: if sampled points lie on the x-axis and F(x,y) = (0,1) = ∇y, all edge flows and reconstructed directions are zero, although the exact continuous projection leaves F unchanged. This illustrates why zero graph residual does not establish accurate full-vector projection. The issue is especially relevant to the sparse k = 4 graph used with high-dimensional actors. Theorem 6.8 assumes a bound on direction error rather than deriving or measuring it. Furthermore, the online implementations change their fitted potential and sampled region, while the Lyapunov analysis uses a fixed potential. These gaps limit how directly the theory supports the practical method.

  2. The empirical evidence does not sufficiently isolate the effect of potential projection, and key reporting details are missing. There is no control that retains the same graph construction and local lifting while bypassing cycle removal. Such a comparison is important because the implementation also restricts update directions to local subspaces and applies shrinkage. The controlled games compare primarily against the raw field rather than methods directly targeting rotational dynamics, such as SGA or extragradient. The supplied PDFs also do not define the normalized-return formula and reference values, evaluation episode count, or full aggregation protocol. Buffer size, potential-network architecture, and measured training overhead are not reported. The return improvements are promising, but these omissions prevent a sufficiently clear assessment of their mechanism, reproducibility, and computational cost.

Questions for authors
  1. Projection accuracy: What sampling or coverage conditions justify the graph-to-vector lifting in high-dimensional actor space? Can the authors provide full-vector approximation-error measurements and a control using the same local lifting without the Poisson projection? These would directly address whether the gains depend on removing non-potential components.

  2. Implemented update field: What exactly is projected in the MAPPO implementation: the actor gradient or the optimizer-preconditioned update? How are PPO clipping, Adam state, changing critics, buffered field estimates, and the eight-update refresh interval handled? What evidence supports treating the resulting directions as approximations to the fixed projected field used in the analysis?

  3. Evaluation and comparative evidence: What normalization and evaluation protocol produces Tables 1 and 3? Please provide raw returns, the missing projection settings, and measured training overhead. Are comparisons against SGA or extragradient available for the controlled games, reporting equilibrium quality alongside circulation? These results could materially change my assessment.

6 comments

Response to the exact graph projection and the identifiability of lifted directions.

The counterexample correctly illustrates an identifiability limitation of the local lifting step. This limitation is distinct from the exact discrete Hodge decomposition in the measured graph edge space.

Our method contains two mathematically distinct operations. First, the ideal unregularized graph Poisson problem projects the measured edge flow onto the graph cut space. Its residual satisfies

B⊤Wωcyc=0.B^\top W\omega_{\mathrm{cyc}}=0.

The released implementation uses ridge regularization, which introduces a numerical approximation to this exact decomposition.

Second, local ridge lifting reconstructs an ambient actor-update vector from projected potential differences. This inverse problem is identifiable only in directions covered by the sampled neighborhood displacements.

The example F(x,y)=(0,1)=∇yF(x,y)=(0,1)=\nabla y, with all samples lying on the xx-axis, correctly illustrates the second limitation. The available measurements cannot identify the normal component of the gradient. They also cannot be used to disprove the first, edge-space orthogonal decomposition.

We have made this separation explicit in the revised manuscript. The new rank-dependent lifting proposition bounds ambient reconstruction error by the sum of an unobserved-gradient component, a singular-value-dependent ridge bias, and potential-difference error. It also states the degenerate rank-zero case.

Thus, the theoretical conclusion is precise: graph projection is exact for the measured edge flow, whereas full-vector recovery requires directional coverage and sufficiently accurate potential differences. With k=4k=4, we do not assert arbitrary high-dimensional full-rank recovery.

For HPML-Neural, integrability follows from the scalar-potential parameterization, but approximation accuracy must be evaluated separately. A fixed neural potential and a continually refreshed sequence of potential networks also have different dynamical guarantees. The revised appendix supplies a one-step bound that explicitly includes potential drift.

Manuscript revision. We strengthened the existing lifting analysis with a singular-value-dependent error bound and clarified the separate approximation errors of graph and neural realizations. The exact graph decomposition is no longer presented as an unconditional full-vector recovery guarantee.


Response to mechanism isolation and comparison with update regularization.

HPML's defining operation is not generic smoothing, clipping, or update shrinkage. It is the weighted orthogonal decomposition of a supplied edge flow into conservative and residual components.

In particular, the ideal unregularized graph projection satisfies

B⊤W(ω−Bϕ⋆)=0,B^\top W(\omega-B\phi^\star)=0,

whereas generic smoothing or norm reduction does not enforce this condition. The ridge-regularized implementation approximates this projection and may introduce additional bias.

The controlled original-game experiment directly tests one alternative explanation. It compares ideal HPML with raw simultaneous ascent, norm-matched ascent, and extragradient on the same game, with the exact original VI gap as the metric. At rotational strength ρ=4\rho=4, the corresponding gaps are 0.10780.1078, 3.90823.9082, 0.88750.8875, and 0.00160.0016. Thus, norm matching alone does not reproduce the effect of ideal potential projection.

The distinction between the ideal operation and practical graph lifting remains important. Local subspace restriction and ridge shrinkage can affect the implemented update even without removing cycles. We therefore conduct a controlled ablation that bypasses the graph Poisson solve while keeping the CountSketch and local ridge-lifting procedures identical.

Synthetic sketched graph ablation (ambient relative error, mean ±\pm s.e., 30 seeds).

ddkkρ\rhoNo-PoissonHPML-GraphNonPot
2400.262±0.0070.262\pm0.0070.262±0.0070.262\pm0.0070.000±0.0000.000\pm0.000
2422.016±0.0032.016\pm0.0031.482±0.0481.482\pm0.0480.463±0.0280.463\pm0.028
2822.017±0.0032.017\pm0.0030.848±0.0390.848\pm0.0390.666±0.0180.666\pm0.018
16421.335±0.0051.335\pm0.0051.098±0.0051.098\pm0.0050.559±0.0120.559\pm0.012
64400.971±0.0010.971\pm0.0010.971±0.0010.971\pm0.0010.000±0.0000.000\pm0.000
64421.069±0.0031.069\pm0.0031.003±0.0021.003\pm0.0020.495±0.0130.495\pm0.013

Both variants use the same 64 sampled nodes, a 64-dimensional CountSketch, and ridge parameter λ=10−4\lambda=10^{-4}. The no-Poisson control directly lifts measured midpoint edge flows, whereas HPML-Graph first solves the graph Poisson problem and then lifts the resulting potential differences. We evaluate all sampled nodes against the known ideal potential component.

For the two-dimensional rotational field (ρ=2\rho=2), HPML-Graph reduces the reconstruction error from 2.0162.016 to 1.4821.482 with k=4k=4. Increasing the neighborhood size to k=8k=8 further reduces the error to 0.8480.848.

The improvement is also observed at d=16d=16. However, at d=64d=64, both variants retain substantial reconstruction error, reflecting the limitations of sketching and local directional coverage.

These results show that the Poisson operation provides an effect beyond local lifting alone in this controlled synthetic setting. They do not establish that circulation removal is the sole cause of the observed Melting Pot performance improvements.

Manuscript revision. We supplement the original-game VI-gap comparison with a sketched graph-only lifting ablation. The appendix provides the experimental protocol and distinguishes improvements from Poisson projection from the limitations of high-dimensional direction recovery. The causal contribution of individual projection components to benchmark returns remains an open empirical question.


Response to evaluation scope, reproducibility, and computation.

The principal Melting Pot comparison is a controlled modification of the MAPPO update procedure, rather than an uncontrolled comparison between different actor-critic architectures.

MAPPO, HPML-Graph, and HPML-Neural use identical actor architectures, with independent parameters for each agent. They also share the same centralized critic architecture, observation interface, and environment-interaction budget. Across the ten reported scenarios, the macro-average normalized returns are 78.1±3.378.1\pm3.3 for MAPPO, 98.7±3.098.7\pm3.0 for HPML-Graph, and 98.2±1.698.2\pm1.6 for HPML-Neural.

This establishes a substantial difference under the reported matched training setup. It does not establish a universal ranking over MARL algorithms or isolate the geometric mechanism without the additional ablations discussed above.

We also distinguish normalized return from raw episodic return. A scenario-normalized value of 100100 does not establish optimality, equilibrium, or comparability with results produced using different evaluation anchors. Accordingly, the normalization formula, reference values, raw episode returns, evaluation episode count, and aggregation order must be specified explicitly.

The computational comparison has a similar distinction. Equal environment steps do not imply equal training time. For NN buffered samples and neighborhood degree kk, the graph has O(Nk)O(Nk) edges and its Poisson system has NN unknowns. The total actor dimension still affects sketch construction and field estimation, and the actual wall-clock overhead depends on the implementation and refresh frequency. These complexities are not substitutes for measured runtimes.

Manuscript revision. We clarify the scope of the matched-MAPPO evidence and the limits of scenario-relative scores. The implementation appendix distinguishes the graph size, actor dimension, projection frequency, and computational overhead. Exact normalization anchors, raw evaluations, architecture details, and measured runtimes must be supplied from the original experiment records rather than reconstructed from the normalized table.

Reviewer_b9WLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Hodge-Projected Multi-Agent Learning (HPML), which projects a joint multi-agent update field onto the space of metric-gradient fields under a chosen metric and sampling measure, and follows the projected direction. It presents graph-Laplacian and neural-potential implementations, establishes Lyapunov improvement for ideal projected dynamics, and derives VI-gap bounds containing non-potential residual and projection-error terms. Experiments include low-dimensional mechanism tests and ten Melting Pot scenarios, with reported return improvements over MAPPO.

Critical strengths

The paper provides a clear variational formulation of integrability in joint update fields and proposes two implementations compatible with CTDE. The analysis explicitly separates structural residuals from projection approximation errors, making the conditions behind the guarantees reasonably transparent. The MAPPO comparisons match actor parameterization, critic, and environment interaction budgets, and the reported improvements on several coordination tasks provide empirical motivation for further investigation.

Critical weaknesses
  1. The field being projected in the implementation, and its connection to the performance gains, remain insufficiently established. Section 5.3 and Appendix E correctly acknowledge that non-potentiality depends on the realized update rule, but the manuscript does not adequately identify where the targeted non-integrability arises in its MAPPO implementation. With independent actors and fixed data, old policies, and advantage estimates, the concatenated raw PPO surrogate gradients form the gradient of the sum of the individual scalar surrogate objectives. This does not imply that the online expected update field is integrable; it does require distinguishing the conditional surrogate gradient, the policy-dependent expected update field, and the actual parameter increment after Adam. The graph uses recent parameter points and update estimates, yet the manuscript does not specify sufficiently how estimates at different nodes correspond to a common field-evaluation rule, or how changes in historical critic, data, and optimizer states are controlled. Consequently, the graph residual may mix spatial non-integrability, temporal drift, and estimation noise. The low-dimensional experiments verify projection behavior on particular constructed fields, but the Melting Pot experiments do not identify the sources of the measured residual or include controls for gradient smoothing, update-norm matching, and local subspace restriction. The results support the potential usefulness of the update modification, but do not yet establish that recovering the potential component and removing circulation explain its benefits.

  2. The practical implementations lack the error control needed to connect them to the theoretical potential-field projection and its guarantees for the original problem. The critical step in the graph implementation is lifting conservative edge flows into high-dimensional update vectors through Eq. (3). Under the experimental choice (M=I) and a positive ridge coefficient, the recovered vector necessarily lies in the span of the neighborhood displacements. If this span does not cover the target gradient, useful directions are lost, even when the input field is already entirely potential. For example, sampling along a line removes the normal component of a constant potential gradient during lifting. The sparse neighborhood configuration in Table 2 makes this concern particularly relevant, but the paper supplies neither the necessary coverage conditions nor a recovery-error bound. Proposition D.4 only establishes a local approximation of potential differences; it does not establish consistency of the complete projection-and-lifting procedure. Theorem 6.8 instead assumes a bound on direction error. The neural implementation does produce a gradient field for fixed network parameters, but neither its fitting error nor the changes induced by continually refreshing the potential are controlled sufficiently to connect the implementation to the fixed-potential analysis. Furthermore, Theorem 6.7 retains a (D\varepsilon) term that does not automatically vanish with iteration. The authors explicitly acknowledge this limitation; the unresolved issue is whether projection errors and structural residuals are sufficiently small in the actual experiments, and how the implemented parameter-update field relates quantitatively to the original game's payoff pseudo-gradient. Thus, the conditional theorems do not yet establish that circulation suppression in the implementation preserves the improvement directions needed for the original problem.

Questions for authors
  1. What precisely is projected in the experiments: raw surrogate gradients, an expected update field, or parameter increments after Adam? When evaluating different parameter points, which quantities are held fixed and which are re-estimated, and how are spatial non-integrability, temporal drift, and sampling noise distinguished?

  2. What neighborhood rank or coverage conditions ensure accurate recovery of the high-dimensional potential gradient from graph potential differences? Please report direction-preservation errors on known potential fields as dimension, neighborhood size, and buffer size vary, and explain how the corresponding projection error is assessed for the neural implementation.

  3. In controlled games where the original operator's VI gap or exploitability can be computed, does HPML continue to improve the original problem as the non-potential component grows, rather than merely reducing circulation or update magnitude? Comparisons with extragradient/optimistic methods and controls matching smoothing, update norms, and subspace restriction would help isolate the contribution of potential-field projection.

6 comments

Response to the projected update field and the frozen-surrogate distinction.

We appreciate the distinction between the frozen PPO surrogate and the policy-dependent online update field. Our formulation is fully consistent with the former being integrable while allowing the latter to be non-integrable.

For fixed rollout data, old policies, advantage estimates, and critic parameters, the concatenated independent-actor PPO gradients are indeed the gradient of the sum of individual surrogate objectives. Their conditional Jacobian is symmetric wherever the surrogates are twice differentiable. This fact is fully compatible with our formulation.

The distinction is that the online expected update field is

U(θ)=Eξ∼Pθ[u(θ;ξ)],U(\theta) = \mathbb E_{\xi\sim P_\theta} [u(\theta;\xi)],

where u(θ;ξ)u(\theta;\xi) is the frozen-data actor-gradient estimator and PθP_\theta is the policy-dependent data-generation procedure. Under standard differentiation assumptions, since uu is a conditional scalar gradient,

∂jUi−∂iUj=Eξ∼Pθ[ui ∂jlog⁡pθ(ξ)−uj ∂ilog⁡pθ(ξ)].\partial_j U_i-\partial_i U_j = \mathbb E_{\xi\sim P_\theta} \left[ u_i\,\partial_j\log p_\theta(\xi) - u_j\,\partial_i\log p_\theta(\xi) \right].

Thus, symmetry of the conditional Hessian does not imply symmetry of the online field's Jacobian. The distribution-response terms need not cancel in general-sum learning.

This is not merely a distinction between two notations. In the quadratic game used in our revised controlled analysis,

\qquad J_2=-\frac{y^2}{2}-\rho xy,$$ the simultaneous payoff-gradient field is $$U(x,y)=(-x+\rho y,-y-\rho x).$$ Each agent differentiates a scalar payoff, yet $$\partial_y U_x-\partial_x U_y=2\rho.$$ For $\rho\neq0$, the stacked game field is therefore non-integrable. Individual scalar objectives do not imply a common potential. We also clarify precisely what the implementation projects. HPML-Graph operates on stacked actor-gradient estimates computed before Adam, using a CountSketch representation and historical parameter-update samples. The graph projection is applied every eight actor-minibatch updates; intervening minibatches use the ordinary PPO direction. We do not describe post-Adam increments as the field being projected. Importantly, historical field estimates need not be evaluations of a stationary expected operator. Critic drift, changing rollout distributions, and sampling noise can contribute to the observed graph residual. Consequently, the implemented Graph NonPot is a normalized residual of the ridge-regularized edge-flow fit. It may reflect ridge bias, sampling noise, and temporal drift in addition to genuine spatial non-integrability. It should therefore not be interpreted as an unbiased measurement of the stationary online field's circulation. *Manuscript revision.* Section 5 now distinguishes frozen surrogate gradients, policy-dependent expected updates, and optimizer-preconditioned increments. The CTDE algorithm explicitly describes the intermittent sketched implementation. We also clarify the limitations of residual diagnostics constructed from historical samples. ---

Response to the graph lifting guarantee and ambient projection accuracy.

The rank and coverage concerns are important for recovering ambient actor-update directions from graph potential differences. We distinguish this inverse problem from the discrete Hodge projection, which remains an exact weighted orthogonal projection in the measured graph edge space.

For the ideal unregularized discrete problem, the graph Poisson solve computes the exact weighted orthogonal projection of the measured edge flow:

\qquad B^\top W\omega_{\mathrm{cyc}}=0.$$ The practical implementation uses a ridge-regularized graph solve with a fixed gauge. Consequently, its computed residual need not satisfy exact orthogonality. This numerical approximation should be distinguished from the ideal discrete Hodge decomposition. Recovering an ambient vector from the projected potential differences is a separate inverse problem. The two operations should not be identified. Our revised appendix makes this distinction quantitative. With neighborhood displacement matrix $X_i$, the lifted gradient is $$\widehat h_i= (X_i^\top W_iX_i+\lambda I)^{-1} X_i^\top W_i y_i.$$ Writing the target gradient as $h^\star=h^\star_\parallel+h^\star_\perp$, where $h^\star_\perp$ is orthogonal to the local displacement span, Proposition D.3's lifting analysis is strengthened by a rank-dependent error bound: $$\begin{aligned} \|\widehat h_i-h^\star\| \le {}& \|h^\star_\perp\| + \frac{\lambda}{\sigma_r^2+\lambda} \|h^\star_\parallel\|\\ &+ \max_j \frac{\sigma_j}{\sigma_j^2+\lambda} \|W_i^{1/2}e_i\|. \end{aligned}$$ Here $\sigma_r$ is the smallest nonzero singular value of $W_i^{1/2}X_i$, and $e_i$ contains potential-difference estimation and discretization errors. This result explicitly separates missing directional coverage, ridge bias, and measurement error. In particular, with only four neighbors in a high-dimensional actor space, arbitrary full-vector reconstruction cannot be guaranteed. However, failure to recover an unobserved direction does not contradict exact orthogonality in the observed graph edge space. The revised result therefore establishes the precise scope of the graph approximation instead of assuming that zero graph residual implies zero ambient error. The neural realization is likewise distinguished: its scalar-potential parameterization guarantees integrability for fixed network parameters, while its approximation accuracy depends on the fitting error. **Synthetic full-vector graph-lifting diagnostic without CountSketch (mean $\pm$ s.e., 30 seeds).** | $d$ | $k$ | $N$ | Coverage | Graph Error | |---:|---:|---:|---:|---:| | 16 | 4 | 64 | $0.276\pm0.026$ | $0.850\pm0.016$ | | 16 | 12 | 128 | $0.753\pm0.029$ | $0.515\pm0.027$ | | 64 | 4 | 64 | $0.055\pm0.007$ | $0.976\pm0.003$ | | 64 | 12 | 128 | $0.177\pm0.010$ | $0.921\pm0.006$ | | 256 | 4 | 64 | $0.017\pm0.002$ | $0.996\pm0.001$ | | 256 | 16 | 256 | $0.065\pm0.004$ | $0.984\pm0.002$ | This controlled experiment isolates local graph lifting without CountSketch. It demonstrates the effect of directional coverage on ambient reconstruction error. We additionally evaluate the sketched HPML-Graph projection kernel against a no-Poisson lifting control on synthetic fields with known potential and rotational components. This experiment isolates the contribution of the Poisson operation while keeping the sketching and local lifting procedures fixed. It does not evaluate HPML-Neural or establish the mechanism behind the Melting Pot return improvements. *Manuscript revision.* The node-wise lifting subsection and the appendix now state the identifiable subspace and the explicit rank-dependent error bound. Theorem 6.8 is interpreted as a conditional guarantee requiring direction accuracy, not as an unconditional guarantee supplied by graph orthogonality. ---

Response to improvement in the original game and the role of the residual term.

The fixed-potential Lyapunov guarantee and the original-game VI-gap guarantees address different quantities. Theorem 6.4 establishes improvement of the projected potential, whereas Theorems 6.7--6.8 explicitly bound the VI gap of the original operator, retaining separate contributions from non-potentiality and projection approximation error.

The additive DεD\varepsilon term is not a claim that a large global residual prevents improvement in every game. It is a worst-case fidelity bound. The sharper pointwise relation is

Gap⁡F(z)≤Gap⁡Φ(z)+D∥R(z)∥\operatorname{Gap}_{F}(z) \le \operatorname{Gap}_{\Phi}(z) + D\|R(z)\|

A large global residual bound does not force a large residual at the point reached by projected learning.

To demonstrate improvement in the original game, rather than only circulation reduction, we added an exactly computable two-player experiment. For the game defined above, the original VI operator is F=−UF=-U, and on [−1,1]2[-1,1]^2,

Gap⁡F(x,y)=x2+y2+∣x−ρy∣+∣y+ρx∣.\operatorname{Gap}_{F}(x,y) = x^2+y^2+ |x-\rho y|+ |y+\rho x|.

The ideal potential component is −z-z under a rotationally invariant reference measure on the unit disk. All methods start from z0=(0.2,0)z_0=(0.2,0), use step size 0.20.2, and run for ten iterations.

ρ\rhoRawNorm-matchedExtragradientIdeal HPML
00.02190.02190.03620.0219
10.04560.10180.04840.0434
20.19540.30810.02400.0649
43.90820.88750.00160.1078

These are exact original-game VI gaps, not fitted-potential scores or additional Melting Pot measurements.

At ρ=4\rho=4, ideal HPML reduces the gap from 3.90823.9082 to 0.10780.1078. A norm-matched raw update still gives 0.88750.8875, so the difference cannot be explained solely by the update magnitude. At ρ=0\rho=0, HPML exactly preserves the already-potential field.

Extragradient performs better at strong coupling in this example, which is consistent with our claim: HPML provides a distinct geometric mechanism, not universal superiority over all operator-correction methods.

This experiment establishes that eliminating the non-potential component can improve the original game's equilibrium metric. It does not, by itself, establish that high-dimensional sketched lifting reproduces the ideal mechanism.

Manuscript revision. Section 7 includes the exact original-game VI-gap comparison, with the full derivation in the appendix. We distinguish this controlled causal evidence from the Melting Pot return comparison, for which independent mechanism diagnostics remain necessary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.