acceptodds
Under review as a conference paper at ICLR 2027

Coordinated Value Gradients for Decentralized Test-Time Guidance in Offline MARL

Abstract

In offline multi-agent reinforcement learning (MARL), diffusion and flow policies capture multimodal joint behavior, but existing methods couple the generator to critic training, which limits their scalability and adaptability. Offline RL avoids this coupling through test-time critic guidance, but extending it to cooperative MARL is nontrivial: centralized critics require global inputs, whereas locally queryable critics need not improve team value. We propose Coordinated Q-Gradient Flow (CQF), to our knowledge the first test-time guidance framework for offline MARL, whose guidance is both locally queryable and team-aligned: monotonically factorizing a team-level critic's objective yields local utilities whose simultaneous guidance steps form a first-order ascent direction of the learned team value at every joint action. The flow policy is fit by supervised matching alone, and at deployment each agent steers it using only its observation and utility gradient. A one-hot relaxation extends the same pipeline to discrete control. Across MA-MuJoCo, MPE, SMACv1, and goal-conditioned MangoBench, CQF attains the best average scores among compared Gaussian, diffusion, and flow methods.

Primary area:
reinforcement learning
Keywords:
Multi-agent Reinforcement Learning, Offline Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_8HwmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CQF, a test-time guidance framework for offline MARL. CQF decouples behavior modeling from reward optimization. It trains a local flow with supervised flow matching and a team-level IQL critic with a monotonic mixer.

Critical strengths
  1. This paper introduces the test-time guidance mechanism from offline DRL into offline MARL and designs an adapted method for it.
  2. Extensive experiments are conducted to validate the effectiveness of the proposed method.
Critical weaknesses
  1. It is unclear whether the proposed method is effective only with the QGF estimator; whether the local gradient guidance of the factorized team critic still holds when the mechanism is replaced by QFQL or GradStep remains unknown.
  2. The current validation tasks are all small-scale multi-agent tasks, and it remains unclear whether the method can run on large-scale tasks.
Questions for authors
  1. Balancing individual and team interests is a classic topic in MARL. Can existing solutions in the MARL community for balancing individual and team interests be directly applied to this research problem? What new challenges does introducing test-time guidance into MARL bring to resolving the tension between individual and team interests?
  2. Is the proposed method effective only with the QGF estimator? If the mechanism is replaced by QFQL or GradStep, does the local gradient guidance of the factorized team critic still hold? Relevant analysis and experimental validation are suggested.
  3. The paper does not integrate multiple different test-time guidance methods from offline DRL (such as QFQL, GradStep, Best-of-N/IFQL, etc.) as independent baselines directly into offline MARL for comparison.
  4. The experimental tasks in the paper are almost all locomotion/navigation tasks, with no manipulation tasks. Additional validation on manipulation tasks would help demonstrate the generality of the proposed method.
  5. The number of agents in the validation tasks is very small, and there is a lack of validation on larger-scale multi-agent tasks, such as tens or even hundreds of agents.
5 comments
AuthorsOpenReview ↗

Response to reviewer's concerns (1/3)

We sincerely thank the reviewer for the careful assessment and constructive feedback. Your comments help us clarify CQF’s connection to existing MARL methods and its compatibility with alternative guidance mechanisms, while motivating stronger baseline comparisons and broader evaluation across task types and agent scales. We address each concern below.


W1, Q2 & Q3. Compatibility with alternative test-time guidance and direct-transfer baselines

Thank you for suggesting these complementary analyses. We have extended both the theory analysis and experiments to examine whether team-aligned local guidance remains effective beyond QGF and how CQF compares with direct transfers of alternative guidance methods.

Compatibility beyond QGF. The alignment property follows from the monotonic critic factorization and holds at any differentiable joint query action. It therefore applies to both QFQL’s intermediate-action guidance and GradStep’s post-denoising refinement. Our extended analysis establishes first-order alignment of their local guidance directions with the learned team value at the query action. The derivation is provided in the revised Appendix J.2 Team-aligned local guidance across different gradient estimators.

Effectiveness with alternative guidance. We then test whether this compatibility translates into practical benefits by replacing the guidance rule with QFQL or GradStep under fully-decentralized (FD), team-aligned CTDE, and centralized formulations. The results are shown in the Table below. CTDE's advantage over FD on most dataset splits supports the effectiveness of team-aligned local gradients across different test-time guidance rules. The comparison appears in Section 5.8 CQF with different test-time guidance mechanisms, with experimental details and per-dataset results in the corresponding Appendix J Details of the Different-Guidance Experiment.

Different test-time guidance methods under independent, team-aligned, and centralized settings.

MethodMA-MuJoCoMPE (Simple Spread)
FD (GradStep)2464.245.0
CTDE (GradStep)2848.752.5
Cent† (GradStep)2500.051.0
FD (QFQL)2607.948.0
CTDE (QFQL)2463.957.3
Cent† (QFQL)2700.653.9

Comparison with direct-transfer baselines. The same experiment also supplies two independent offline MARL baselines, FD (QFQL) and FD (GradStep), which apply the respective test-time guidance rules independently to each agent. Comparing these results with the original MA-MuJoCo and MPE tables (Table 1 and Table 2 in the paper) shows that CQF achieves higher average performance than both baselines on both benchmarks. Together, the replacement experiments establish compatibility with alternative guidance mechanisms, while the direct-transfer comparisons demonstrate the practical advantage of the complete CQF framework over these independent extensions.


W2 & Q5. Scaling to larger teams

Thank you for highlighting the need to evaluate larger teams. We extend the landmark-covering evaluation to 3, 10, 20, and 40 agents, increasing the joint action dimension from 6 to 80, and compare CQF with FD under fixed data and training budgets and a common guidance weight β=0.3 (see Appendix K Scaling the number of agents for the full protocol and results).

Guidance remains effective with larger teams. CQF achieves higher mean return than FD at every tested team size, while also improving over its own unguided flow. This benefit extends across the guidance sweep: at every tested positive β up to 0.3, CQF produces larger gains over its own unguided checkpoint than FD does, at all four team sizes. These results support the continued usefulness of team-aligned local gradients as the joint action space grows.

Improved coordination throughout execution. The return gains are accompanied by faster landmark occupation. At 40 agents, CQF increases mean coverage from FD’s 81.60% to 83.13% and reaches 95.00% final coverage. Its mean coverage first reaches 90% at step 15, compared with step 17 for FD and 19 for the unguided flow, showing that guidance improves progress throughout the trajectory.

The added experiments extend CQF’s empirical validation to teams of tens of agents, demonstrating gains in return and coordination under controlled training budgets.

AuthorsOpenReview ↗

Response to reviewer's concerns (2/3)

Q1. Existing MARL solutions and the new challenges of test-time guidance

How existing MARL solutions apply. Existing value-decomposition methods are directly relevant, and CQF builds on them to learn local utilities through a shared team-value objective. Since agents share the same reward, the goal is to align local action improvements with team performance. Test-time guidance introduces additional requirements on how this learned value structure is used: agents must refine their actions during execution using only locally available information.

First challenge: making team-informed guidance locally executable. The first additional challenge is that the critic itself becomes part of the decentralized execution procedure in test-time methods. In centralized actor–critic training-time methods [1] [2], the value gradient is evaluated during training, when global state and teammate actions are available; at execution, the local actors produce actions without querying the centralized critics. With test-time guidance, however, the critic gradient is evaluated during action generation itself, so the guidance computation must satisfy the same local-information constraints as the deployed policy. A centralized critic cannot generally be queried unchanged under this local information constraint, while an independently trained local critic provides no assurance that its gradient serves the team objective. CQF addresses this interface requirement by learning local utilities through the factorized team critic and querying only their local action gradients at execution.

Second challenge: aligning simultaneous action refinements. Once guidance is locally computable, the next question is whether the agents’ simultaneous refinements remain consistent with the team objective, beyond ensuring agreement between greedy maximizers. The individual-global-max (IGM) property establishes consistency between local and joint maximization. Test-time gradient guidance, however, performs a limited number of updates from behavior-generated actions, which need not be maximizers. Agreement at the maximizers alone therefore does not establish whether local gradients at intermediate query actions point toward improvement in the team value. CQF addresses this requirement through the gradient structure induced by monotonic value factorization. As shown in Proposition 1 in Section 4.3, the gradient of the learned team value with respect to each agent’s action is a nonnegative multiple of that agent’s local utility gradient. Consequently, the local guidance directions, when applied simultaneously, cannot oppose the learned team-value gradient at first order. Each agent can therefore compute its guidance using only its own observation and candidate action, without evaluating the centralized mixer or accessing teammates’ actions. This gives a concrete role to monotonic factorization in test-time guidance: it provides alignment at each differentiable query action encountered during sampling, beyond consistency between final greedy choices. Our extended analysis establishes the same alignment property for QFQL and GradStep in Section 5.8 and Appendix J; see the revised theoretical analysis, experiment results and our response to W1/Q2.

Third challenge: obtaining useful guidance from an offline critic. This alignment provides a structural basis for coordination, while practical improvement also depends on where and how the critic is queried during generation. Generated actions can move beyond reliable offline support, and locally plausible refinements can produce unfamiliar joint combinations. The query mechanism and guidance strength therefore remain consequential design choices. Our CTDE–FD comparisons, fixed-checkpoint guidance ablations, and QFQL/GradStep replacements examine these effects, connecting locally executable team alignment to measured improvements in return.

AuthorsOpenReview ↗

Response to reviewer's concerns (3/3)

Q4. Generality beyond locomotion and navigation: validation on manipulation tasks

Additional evaluation on manipulation tasks. Thank you for this suggestion. To assess applicability beyond locomotion and navigation, we additionally evaluated goal-conditioned CQF (GCCQF) on two bimanual manipulation tasks from MangoBench [3]: LiftBarrier and PlaceFood. These tasks involve different coordination requirements: LiftBarrier requires two robotic arms to grasp and lift a barrier together, whereas PlaceFood requires sequential cooperation, with one arm opening a pot lid before the other places food inside. They therefore extend our evaluation to both synchronous and asynchronous cooperative manipulation.

Comparison with MangoBench baselines. We use the same offline datasets, observation inputs, evaluation random seeds and final-task success criteria as MangoBench. The IQL expectile for both task is 0.7, the horizon H for LiftBarrier is 5 and H for PlaceFood is 1. The table below compares our results with the baselines reported in Figure 2 in its main text and Table 7 of its supplementary material.

Final task success rate (%; higher is better).

TaskIHIQLGCMBCICRLDPGCCQF (ours)
LiftBarrier82.047.056.058.096.0
PlaceFood23.021.035.020.035.0

On LiftBarrier, GCCQF achieves 96% success, exceeding the strongest reported baseline, IHIQL, by 14 percentage points. On PlaceFood, its 35% success rate matches the best reported point estimate, achieved by ICRL. The results demonstrate successful use of the guided framework in cooperative manipulation. These experiments broaden the empirical validation beyond locomotion and navigation and support the framework’s applicability to different forms of cooperative robotic control.

Reference.

[1] FACMAC: Factored Multi-Agent Centralised Policy Gradients

[2] Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

[3] MangoBench: A Benchmark for Multi-Agent Goal-Conditioned Offline Reinforcement Learning

Reviewer_eScuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CQF addresses decentralized test-time guidance in offline MARL by separating behavior modeling from value learning. Each local flow is fitted by supervised flow matching. Separately, local utilities are combined through a monotonic mixer and trained with team-level IQL: the TD target uses the mean team reward and the next mixed value, while the value function uses expectile regression against the target mixed critic. Reward information therefore enters the sampler at deployment without updating the behavior-flow parameters (pp. 4–5, Eqs. 3–6).

At inference, agent ii forms a one-step clean-action estimate a^i=ai,λ+(1−λ)vi\hat a_i=a_{i,\lambda}+(1-\lambda)v_i, queries gi=∇a^iQi(oi,a^i)g_i=\nabla_{\hat a_i}Q_i(o_i,\hat a_i), and adds βgi\beta g_i to its flow velocity. The mixer and global state are discarded. Monotonicity establishes that the assembled local gradients have a nonnegative directional derivative for the learned mixed critic at the query action. This concerns the added guidance direction; the full solver update, final executed action, and true team return are distinct objects (pp. 5–6, Proposition 1; pp. 16–18, Appendix C).

Experiments cover MA-MuJoCo, MPE, SMACv1, and goal-conditioned MangoBench, with substantial supplementary controls. The core identity is correct and the reported improvements are promising. My main reservations concern selected evaluation maxima, the scope of the coordination/discrete claims, and unresolved details of the implemented algorithm and goal-conditioned comparison (pp. 7–9, 24–32).

Critical strengths
  1. A coherent decentralized execution interface. The behavior-flow objective is disjoint from the critic objectives, and deployment requires each agent's observation/ID, local generative state, frozen flow, and local utility gradient. Discarding the mixer genuinely avoids a global critic query during action generation under the specified interface (pp. 4–6, Eqs. 3–6; p. 15, Algorithm 1).

    The distinction between separately learning behavior and introducing value during sampling is practically useful. Guidance weights can be changed without retraining these models, and the frozen-flow sweeps directly examine that property (pp. 8, 25–26, Figures 2 and 10–11).

  2. A correct local identity with meaningful qualifications. With mi=∂Qtot/∂Qi≥0m_i=\partial Q_{\mathrm{tot}}/\partial Q_i\geq0 and gi=∇aiQig_i=\nabla_{a_i}Q_i, the chain rule gives ∇aiQtot=migi\nabla_{a_i}Q_{\mathrm{tot}}=m_i g_i and ⟨∇aQtot,g⟩=∑imi∣gi∣22≥0\langle\nabla_aQ_{\mathrm{tot}},g\rangle=\sum_i m_i\\|g_i\\|_2^2\geq0 (p. 6, Proposition 1/Eq. 7).

    Appendix C explicitly separates this added-direction identity from the composite v+βgv+\beta g, and states that its finite clean-query probe is not the next actual clean estimate. Appendix J also limits the guarantee to the learned critic rather than true return. These qualifications substantially improve the precision of the theoretical account (pp. 16–18, Table 6; p. 33).

  3. Broad reported guidance gains, including substantial effects. Tables 12–14 report improved guided means in all 28 standard settings. The matched-flow averages rise from approximately 1718.3 to 2952.8 on MA-MuJoCo, 28.2 to 60.7 on MPE, and 13.0 to 16.1 on SMACv1 (pp. 24–25).

    Table 5 reports MangoBench success rates of 95.3%, 89.0%, and 84.0%, exceeding the strongest listed alternatives by 32.0, 58.7, and 46.5 percentage points. Selection and protocol concerns qualify these effects, but the larger improvements deserve separate consideration from the small differences between benchmark leaders (p. 9).

  4. A substantial set of relevant controls. The independent-critic comparisons reuse local behavior flows; the centralized references examine a different information interface; and the unguided local-versus-joint comparison helps separate behavior modeling from guidance. The discrete study compares clipping/projection variants with frozen parameters and measures simplex residuals and illegal mass (pp. 8, 18–21, 26, Tables 4 and 7–10/15).

    The MPE study also uses paired environment/action-sampling conditions and uniformly sampled trajectory illustrations, alongside explicit coverage and assignment metrics. These are useful behavioral checks, although their scope remains limited as discussed below (pp. 22–24, Table 11 and Figures 6–9).

Critical weaknesses
  1. The reported entries are selected maxima on the evaluation episodes themselves. Appendix I explicitly states that the table entry maximizes mean online return on the same episodes the tables report. MA-MuJoCo uses 16 guidance candidates, with a separate selected value for each dataset-quality split. Sharing that value across three training seeds avoids individual seed maximizers, but does not provide independent evaluation after selection (p. 30, lines 1589–1605; p. 32, Table 19).

    This particularly affects the small average leads: MPE is 60.7 versus 58.775 for ICQ, and SMACv1 is 16.125 versus 15.9 for DoF. Standard deviations across the selected policies do not quantify selection uncertainty. Coupled baselines also tune by online return, but comparable search budgets and result provenance are not documented for every listed entry (pp. 7, 30, Tables 2–3).

    The broader MA sweeps do show useful nonzero-weight regions on several splits. Conversely, including β=0\beta=0 among candidates means that improvements over that endpoint alone are not independent evidence against selection effects. The concern is the reliability of comparative estimates; it does not establish that the much larger matched-flow or MangoBench gains are artifacts (pp. 8, 24–26).

  2. The coordination guarantee is conditional on a restrictive representation. A monotonic mixer cannot reverse one local scalar action preference as a teammate's action changes. With fixed uninformative observations and Q∗(a1,a2)=−(a1−a2)2Q^*(a_1,a_2)=-(a_1-a_2)^2, the derivative at a1=0a_1=0 is 2a22a_2. Its sign reverses between a2=−1a_2=-1 and a2=1a_2=1, which no nonnegative multiplier of a fixed local g1(0)g_1(0) can express. This limits the claim about retaining joint-action interaction information; it does not refute Eq. 7 (pp. 4–6).

    Moreover, 26 of 28 standard configurations use additive VDN mean, whose cross-action derivatives vanish at fixed observations; only two use QMIX. The statement that interaction is represented by the critic needs this qualification for the architecture most often used (pp. 5, 30, 32, Table 19).

    Under independent per-agent Gaussian initialization and local-only updates, the conditional policy also factorizes. For example, independent equally weighted local modes cannot preserve a joint mixture containing only (+1,+1)(+1,+1) and (−1,−1)(-1,-1): they additionally generate mismatched pairs. Informative observations/IDs can support coordination, and a policy can select one common mode, but correlated multimodal behavior requires further information or shared randomness. Appendix J partially acknowledges this limitation (pp. 15, 27, 33).

  3. Hard-action training does not identify the relaxed discrete gradient. The discrete critic is trained at one-hot vertices but queried at continuous actions. Clipping to [0,1]A[0,1]^A does not match that finite support; even simplex projection permits interior points absent from critic training (p. 6, Section 4.4; pp. 18–20).

    Concretely, replacing a local utility by Q(x)+cx1x2Q(x)+c x_1x_2 leaves every one-hot value unchanged. At the simplex point (1/4,3/4)(1/4,3/4), however, the added directional derivative along (1,−1)(1,-1) is c/2c/2. Thus equally fitting hard-action values can produce different relaxed guidance directions. Monotonic mixing does not remove this ambiguity, and continuous ascent does not imply improvement after argmax.

    Appendix D usefully demonstrates legal execution and reports geometry/return diagnostics, but simplex distance and illegal mass do not establish gradient calibration. The projection alternatives also reuse the weight selected for BoxClip. Table 7 therefore supports BoxClip at that operating point rather than a fully tuned comparison of geometries. The discrete extension can be a useful empirical heuristic; its support explanation should say so precisely (pp. 18–20, Tables 7–9).

  4. The effective MangoBench relabeling protocol is unresolved. Appendices H/I state that the methods share hindsight-relabeling distributions. Table 18 includes random dataset goals, notably a 50% random component for Stitch actor goals. Section H.3 instead says that GCMADiff uses the sampled window's last observation as its training goal, without explaining how the random-goal component is realized (pp. 27, 29–31).

    Endpoint inpainting is a legitimate adaptation, but its effective training-goal distribution and treatment of unrelated goals must be stated before the common-protocol claim can be assessed. Tables 20–21 also list selected baseline settings or broad ranges rather than complete candidate grids. This ambiguity matters to the large goal-conditioned comparison; it does not establish invalid results, and the separate large advantage over GCMAC-Flow remains relevant (pp. 9, 32, Tables 5/20–21).

  5. The implemented chunk/ensemble algorithm cannot be reconstructed precisely. Table 19 uses action chunks H=5H=5 in multiple settings and refers to Section 4.1, but Eq. 4 and Algorithm 1 define a single action. Chunk construction, critic inputs, actions executed before replanning, and reward/discount handling are not specified. Table 18 separately gives MangoBench horizon h=5h=5 with action chunking disabled (pp. 5, 15, 28, 31–32).

    Tables 17–18 also specify two critics and minimum aggregation without locating the minimum before/after mixing, in targets, or in guidance. Selecting a different minimum critic independently for each utility need not select the same branch as minimizing the whole mixed critics. The implementation might be consistent, but the derivation cannot be connected to it from the present description. This is a specification concern, not evidence of a faulty implementation (pp. 6, 31–32).

  6. The coordination diagnostics support a narrower conclusion than general coordinated ascent. Testing CQF gradients against CQF's own factorized critic checks architectural consistency: nonnegative infinitesimal alignment is built into the model. Independent gradients have no analogous constraint against that particular critic. The comparison does not independently establish which critic better approximates true joint value (pp. 16–18, Table 6).

    The finite clean-query probes increase learned value on 53.5%/54.0% of VDN/QMIX steps, and the appendix correctly distinguishes those probes from actual solver estimates. These rates should neither be presented as an executed-step guarantee nor treated as a contradiction of Proposition 1 (pp. 17–18).

    The paired MPE study adds more direct behavioral evidence, but uses one training seed. Duplicate targeting increases on Medium and Random, while Medium-Replay joint completion does not improve. Improved coverage and assignment distances remain useful findings; they establish improvement in these observed behaviors rather than all coordination dimensions (pp. 22–24, Table 11).

Questions for authors
  1. Can the comparative returns be evaluated after fixing the selected weight? Please retain the disclosed selection procedure, freeze its chosen β\beta, and evaluate on fresh simulator episodes not used to maximize the grid. Existing checkpoints can be reused. Please identify which baseline entries were rerun or reused and document their relevant selection budgets (p. 30; Weakness 1).

    The most useful result would distinguish persistence of the small MPE/SMAC leader gaps from persistence of the larger matched-flow effects. A new benchmark family or retraining every flow candidate is unnecessary.

  2. What coordination scope is intended? Please state the limitation imposed by monotonic/additive utilities and independent local sampling. For MA-QGF, specify whether rewards are individual or the same team-mean reward, and how its TD/expectile targets differ from CQF's. This would make Tables 4/10 easier to interpret as evidence about team supervision (pp. 5–8, 21, 32; Weaknesses 2 and 6).

    Please also distinguish agreement with the learned factorized objective from evidence about actual coordinated behavior; the existing MPE metrics can support the latter within their stated scope.

  3. Is the discrete extension intended as an empirical heuristic? Please correct the one-hot-support explanation and distinguish legality, simplex proximity, and identification of relaxed gradients. Describe Table 7 as a common-weight operating-point comparison unless geometry-specific tuning is available (pp. 6, 18–20; Weakness 3).

    No general discrete improvement theorem is required. A precise statement of what the existing experiments establish would resolve much of this concern.

  4. How does GCMADiff realize the stated common goal distribution? Please explain how last-window-state goals implement the future/random mixture in Table 18, especially for Stitch, or explicitly describe the effective method-specific difference. Include the searched and selected controls needed to interpret Table 5 (pp. 27, 29–32; Weakness 4).

    If the effective distributions differ, please explain whether that changes the comparison's conclusion; an endpoint-inpainting adaptation alone does not settle this point.

  5. What is the exact algorithm when chunks and critic ensembles are used? Please provide concise pseudocode defining the H=5H=5 action object, critic/target, and execution/replanning rule; clarify MangoBench's separate horizon notation; and locate every two-critic minimum. State which factorized branch supplies the executed local gradients (pp. 5, 15, 28, 31–32; Weakness 5).

    This clarification should connect the tested implementation to the single-action derivation. It need not introduce a new method or additional benchmark.

1 comment
AuthorsOpenReview ↗

Response to reviewer's concerns (6/6)

W6 & Q2. Scope of alignment diagnostics (Table 6 / revised Table 8) and multi-seed behavioral coordination on MPE

Mechanism verification vs. environment effectiveness (Appendix C, Table 6 / revised Table 8). As already noted in Appendices C.1–C.2, Table 6 (revised Table 8) is explicitly designed to verify internal mechanism consistency with the learned factorized critic QtotQ_{\mathrm{tot}}, rather than claiming true-Q∗Q^{\ast} accuracy or per-step execution guarantees. Specifically, B=⟨∇Qtot,g⟩≥0B=\langle\nabla Q_{\mathrm{tot}}, g\rangle \ge 0 (100% empirically) confirms Proposition 1 for our deployed gradients, while the matched independent-critic negative control drops to 76.6% (VDN, cos⁡=0.32\cos=0.32) and 60.3% (QMIX, cos⁡=0.15\cos=0.15), showing that local queryability without team supervision frequently opposes the learned team objective on identical states. Likewise, Proposition 2 (Appendix C.2, Eqs. 8–10) explicitly states that CQF does not guarantee C>0C > 0 or ΔQtot>0\Delta Q_{\mathrm{tot}} > 0 on every step (as β\beta balances value ascent against staying in-support and curvature bounds finite steps); rather, it tests whether team guidance shifts v+βgv+\beta g toward higher learned value more often than independent gradients (raising ΔQtot>0\Delta Q_{\mathrm{tot}} > 0 from 33.7%/46.6% to 53.5%/54.0%). Environment-level effectiveness and coordination are established separately by closed-loop returns (Section 5) and behavioral diagnostics (Appendix E.3).

Behavioral coordination scope and multi-seed validation on MPE. In our 100-paired-episode CQF vs. MA-QGF diagnostic (Appendix E.3, Table 11 / revised Table 13), CQF consistently improves spatial task allocation across all four splits (reducing DassignD_{\mathrm{assign}} and DcoverD_{\mathrm{cover}}, increasing coverage C0.1C_{0.1}), with selective trade-offs in secondary metrics (RdupR_{\mathrm{dup}} +0.063+0.063/+0.066+0.066 on Medium/Random; JtailJ_{\mathrm{tail}} 0.0000.000 on Medium-Replay). To address the single-seed limitation, we recorded the full metric suite (including collisions) across multiple independently trained checkpoints during our held-out fixed-β\beta evaluation (10 fresh paired episodes per seed, fixed historical shared β\beta vs. β=0\beta=0 over the final 25% window; mean ± SD across seeds):

DatasetΔDcover\Delta D_{\mathrm{cover}} ↓ΔDassign\Delta D_{\mathrm{assign}} ↓ΔC0.1\Delta C_{0.1} ↑ΔRdup\Delta R_{\mathrm{dup}} ↓ΔJtail\Delta J_{\mathrm{tail}} ↑Δ\Delta Collisions ↓
Expert−0.013 ± 0.009−0.020 ± 0.0190.000 ± 0.000−0.031 ± 0.010+0.014 ± 0.000−0.043 ± 0.202
Medium−0.080 ± 0.020−0.099 ± 0.061+0.102 ± 0.017+0.079 ± 0.0520.000 ± 0.000+0.086 ± 0.065
Medium-Replay−0.113 ± 0.024−0.190 ± 0.086−0.002 ± 0.023−0.063 ± 0.0410.000 ± 0.000+0.105 ± 0.146
Random−0.310 ± 0.026−0.458 ± 0.011+0.133 ± 0.055−0.041 ± 0.056+0.014 ± 0.025+0.095 ± 0.059

Across 3 training seeds, guidance consistently reduces DcoverD_{\mathrm{cover}} and DassignD_{\mathrm{assign}} on all four splits (and RdupR_{\mathrm{dup}} on three of four), linking higher returns directly to better spatial division of labor—further corroborated at 10–40 agents in Appendix K. Moving more actively toward landmarks on non-expert splits entails slightly more close-range contacts (Δ\Delta Collisions +0.086+0.086 to +0.105+0.105), a transparent trade-off outweighed by coverage gains. We have added this multi-seed table to Appendix E.3 and scope our behavioral claim specifically to spatial landmark allocation and coverage.


References

[1] Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization

[2] Multi-Agent Coordination via Flow Matching

[3] Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification

[4] MADiff: Offline Multi-Agent Learning with Diffusion Models

[5] Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learning.

[6] CoFlow: Coordinated Flow Matching for Offline Multi-Agent Reinforcement Learning

[7] Planning with Diffusion for Flexible Behavior Synthesis

Reviewer_Kox5OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In offline MARL, a centralized joint critic is team-aligned but not locally queryable at deployment, while independently trained local critics are deployable but may point in directions that do not improve the team objective. To address this problem, this paper proposes Coordinated Q-Gradient Flow (CQF), a test-time guidance framework for offline cooperative multi-agent reinforcement learning (MARL) under decentralized execution.

The paper evaluates CQF on MA-MuJoCo, MPE, SMACv1, and MangoBench, comparing against Gaussian, diffusion, and flow baselines. CQF reports the best average performance, with the largest gains on mixed and suboptimal data, and BC-like training efficiency because behavior fitting and critic learning are independent. A goal-conditioned variant, GCCQF, is also presented.

Critical strengths
  1. In offline MARL centralized critics are not deployable, while local critics are deployable but not necessarily team-aligned. CQF addresses this directly by carrying coordination in a locally queryable value gradient.

  2. CQF seems to be the first test-time gradient-guidance method for cooperative offline MARL. Reward optimization is moved entirely to inference, while training remains supervised behavior fitting plus critic learning.

  3. Proposition 1 shows that monotonic mixing makes each local utility gradient a positively scaled block of the joint-value gradient. Simultaneous local guidance steps therefore cannot be first-order opposed to the learned QtotQ_{\rm tot}. This is a useful and non-trivial justification for decentralized test-time guidance.

Critical weaknesses
  1. The paper shows return rises and then falls with β\beta. The guidance weight is tuned per dataset and domain, and large β\beta can severely degrade performance. There is no automatic or principled selection rule for β\beta.

  2. The method inherits errors and distribution-shift problems from the offline critic. Although IQL reduces OOD action maximization, the critic is still queried at policy-generated clean-action estimates, which may leave the offline support. Monotonic factorization also restricts the representable joint-value class

  3. The paper mentions that a weakly fit flow leaves less headroom for guidance. On some tasks, the centralized joint flow also underperforms, so a stronger locally executable reference remains a problem to be addressed.

Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Response to reviewer's concerns (1/2)

We sincerely thank the reviewer for the time and care devoted to reviewing our work and for the constructive feedback. Your comments are valuable for clarifying the scope, evaluation protocol, and limitations of CQF. We address each concern below.


W1. Guidance strength β and reliable selection

Current selection protocol. Thank you for raising this concern. Related single-agent methods tune guidance empirically: QGF [1] selects hyperparameters per domain, while QFQL [2] tunes β per task. Following this practice, we select β through simulator-return grid search using the same frozen flow and critic, avoiding retraining. Selection remains empirical, and excessive β can degrade performance. We acknowledge these limitations in Appendix N "Limitations" and add an independent evaluation to test whether the selected weights remain effective on fresh episodes.

Independent evaluation at fixed β. We fix the historical β, share it across runs within each task–dataset setting, and evaluate the frozen models on fresh episodes against matched β=0 configurations (see the Appendix L "Independent Evaluation with Fixed Guidance Weights" for the full protocol and results).

Guidance yields higher mean performance in all 28 standard-task settings, with domain-level gains of 67.3% in MA-MuJoCo return, 28.88 normalized-score points in MPE, and 17.9% in SMACv1 return. Benefits also persist across all three MangoBench tasks, with success-rate gains of 37.33–75.67 percentage points.

Consistency with the original results. These results are broadly consistent with Tables 1–3, supporting the stability of the guidance benefits on fresh episodes after β is fixed. Automatic offline selection, potentially informed by critic uncertainty, remains a useful direction for reducing simulator validation requirements.


W2(a). Critic error and distribution shift

We thank the reviewer for highlighting the important issue of critic reliability under distribution shift.

Clean-action guidance and data coverage. Following QGF [1], CQF evaluates critic gradients at estimated clean actions rather than intermediate noisy actions. This addresses the mismatch between noisy sampler states and the clean actions used for critic learning. We agree, however, that generated clean actions can still lie outside well-covered regions of the offline data, making critic reliability important for effective guidance.

Improving guidance through alternative critic learning. Fortunately, CQF's team-aligned guidance mechanism is compatible with different critic-learning objectives. This flexibility allows us to investigate whether incorporating policy-generated next actions into critic training improves guidance. Specifically, we replace IQL-based critic learning with TD learning using next actions sampled from an auxiliary QAM policy [2], while retaining CQF’s original guidance sampler. This tests whether an alternative critic-learning procedure can improve guidance while preserving decentralized coordination.

On 6-HalfCheetah, we obtain:

MethodMediumExpert
CQF (IQL critic)3721.2 ± 26.24461.9 ± 22.0
CQF (QAM-based critic)4475.43 ± 64.545465.21 ± 97.37
Fully decentralized (QAM-based critic)3781.56 ± 60.834765.71 ± 17.75
Centralized (QAM-based critic)4289.01 ± 154.164800.58 ± 213.22

Mean ± standard deviation over three seeds.

The QAM-based CQF variant improves mean return by 20.3% on Medium and 22.5% on Expert relative to the reported IQL-based results. Under the same QAM-critic training settings, CQF also achieves higher mean returns than both fully decentralized and centralized guidance. CQF and the fully decentralized variant share the same frozen local BC reference, supporting the benefit of team-aligned critic learning in this comparison.

These results demonstrate the practical value of alternative critic learning within CQF and the continued effectiveness of its coordination mechanism. Full results appear in Section 5.7 “CQF with different critics”, with implementation and evaluation details in Appendix I “Details of the Different-Critic Experiment.” in the revised paper.

AuthorsOpenReview ↗

Response to reviewer's concerns (2/2)

W2(b). Representational limits of monotonic factorization

Thank you for highlighting this limitation. We agree that monotonic factorization restricts the representable joint-value class, particularly when an agent’s action preference depends on its teammates’ actions, as discussed in Weighted QMIX [3].

Scope of Proposition 1. Monotonicity provides the structural basis for CQF’s locally computable, team-aligned guidance. Under the stated assumptions, Proposition 1 establishes first-order alignment with the learned factorized team-value objective. It does not establish that this objective can represent an arbitrary team value function. This is a trade-off between representational flexibility and decentralized guidance with an explicit alignment guarantee.

Illustrative XOR limitation. To make this representational limitation concrete, consider a two-agent XOR payoff, where each agent’s preferred action reverses with its teammate’s action. A fixed local utility ordering combined with a monotone mixer cannot represent both reversals. Nevertheless, agents with distinct roles can select one optimal joint action deterministically. Selecting an optimal action combination is distinct from representing the complete joint-value function.

Related evidence and remaining limitation. CQF and MAC-Flow [4]both rely on monotonic value factorization to support decentralized decision-making. This shared structural choice makes the expressiveness of the factorized representation a relevant limitation for both methods. MAC-Flow’s XOR stress test [4, Appendix H.6] illustrates a related failure at the policy-distribution level: its decentralized policies fail to preserve the correlated modes captured by the joint flow. Although CQF uses test-time local guidance rather than joint-to-local policy distillation, its guidance still operates through a monotonically factorized critic and therefore retains the value-representation restriction illustrated by the XOR example above. We have clarified this structural limitation and the scope of Proposition 1 in the limitations discussion Appendix N. Extending CQF to more expressive joint-value structures while retaining locally computable, team-aligned guidance remains an important direction for future work.


W3. Quality and coverage of the reference flow

Thank you for highlighting the importance of the reference flow. We agree that its quality affects the performance achievable through guidance. CQF’s contribution is to enable team-aligned policy improvement at test time while preserving supervised behavior modeling and decentralized execution.

Evidence from the existing comparisons. Appendix G (Table 15) compares unguided local and centralized flows, while the matched-reference experiments evaluate the gains from guidance using the same frozen reference. These controlled comparisons demonstrate the value of CQF’s guidance without attributing the improvement to a stronger behavior model. Moreover, the centralized variant’s weaker performance on 6-HalfCheetah and 3-Hopper shows that centralization alone does not consistently improve performance in our evaluated settings. Together, these results support the effectiveness of combining a locally executable reference with team-aligned guidance.

Remaining limitation and opportunities. As acknowledged in our Appendix N "Limitations", a reference that inadequately captures useful behaviors can constrain the gains achievable through guidance. A similar dependence on reference-model quality is discussed in QGF [1, Appendix I]. Improving the reference therefore complements CQF’s contribution. In particular, CQF’s separation of behavior modeling from critic learning provides a practical route to incorporate stronger local observation-history representations and improved supervised flow-training procedures while retaining decentralized execution. Developing and evaluating such references remains an important direction for future work.

Reference.

[1] Test-time gradient guidance of flow policies in reinforcement learning

[2] Q-guided flow q-learning

[3] Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

[4] Multi-agent Coordination via Flow Matching

I read the comments and responses of the authors. The additional explanations and analyses do help me to better appreciate the results. I thank the authors for their efforts.

We truly thank you for your thoughtful feedback and for carefully considering our responses! We are glad the additional analyses were helpful, and we sincerely appreciate your time and constructive engagement!

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.