- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CQF, a test-time guidance framework for offline MARL. CQF decouples behavior modeling from reward optimization. It trains a local flow with supervised flow matching and a team-level IQL critic with a monotonic mixer.
Critical strengths
- This paper introduces the test-time guidance mechanism from offline DRL into offline MARL and designs an adapted method for it.
- Extensive experiments are conducted to validate the effectiveness of the proposed method.
Critical weaknesses
- It is unclear whether the proposed method is effective only with the QGF estimator; whether the local gradient guidance of the factorized team critic still holds when the mechanism is replaced by QFQL or GradStep remains unknown.
- The current validation tasks are all small-scale multi-agent tasks, and it remains unclear whether the method can run on large-scale tasks.
Questions for authors
- Balancing individual and team interests is a classic topic in MARL. Can existing solutions in the MARL community for balancing individual and team interests be directly applied to this research problem? What new challenges does introducing test-time guidance into MARL bring to resolving the tension between individual and team interests?
- Is the proposed method effective only with the QGF estimator? If the mechanism is replaced by QFQL or GradStep, does the local gradient guidance of the factorized team critic still hold? Relevant analysis and experimental validation are suggested.
- The paper does not integrate multiple different test-time guidance methods from offline DRL (such as QFQL, GradStep, Best-of-N/IFQL, etc.) as independent baselines directly into offline MARL for comparison.
- The experimental tasks in the paper are almost all locomotion/navigation tasks, with no manipulation tasks. Additional validation on manipulation tasks would help demonstrate the generality of the proposed method.
- The number of agents in the validation tasks is very small, and there is a lack of validation on larger-scale multi-agent tasks, such as tens or even hundreds of agents.
5 comments
Response to reviewer's concerns (2/3)
Q1. Existing MARL solutions and the new challenges of test-time guidance
How existing MARL solutions apply. Existing value-decomposition methods are directly relevant, and CQF builds on them to learn local utilities through a shared team-value objective. Since agents share the same reward, the goal is to align local action improvements with team performance. Test-time guidance introduces additional requirements on how this learned value structure is used: agents must refine their actions during execution using only locally available information.
First challenge: making team-informed guidance locally executable.
The first additional challenge is that the critic itself becomes part of the decentralized execution procedure in test-time methods. In centralized actor–critic training-time methods [1] [2], the value gradient is evaluated during training, when global state and teammate actions are available; at execution, the local actors produce actions without querying the centralized critics. With test-time guidance, however, the critic gradient is evaluated during action generation itself, so the guidance computation must satisfy the same local-information constraints as the deployed policy. A centralized critic cannot generally be queried unchanged under this local information constraint, while an independently trained local critic provides no assurance that its gradient serves the team objective. CQF addresses this interface requirement by learning local utilities through the factorized team critic and querying only their local action gradients at execution.
Second challenge: aligning simultaneous action refinements. Once guidance is locally computable, the next question is whether the agents’ simultaneous refinements remain consistent with the team objective, beyond ensuring agreement between greedy maximizers.
The individual-global-max (IGM) property establishes consistency between local and joint maximization. Test-time gradient guidance, however, performs a limited number of updates from behavior-generated actions, which need not be maximizers. Agreement at the maximizers alone therefore does not establish whether local gradients at intermediate query actions point toward improvement in the team value.
CQF addresses this requirement through the gradient structure induced by monotonic value factorization. As shown in Proposition 1 in Section 4.3, the gradient of the learned team value with respect to each agent’s action is a nonnegative multiple of that agent’s local utility gradient. Consequently, the local guidance directions, when applied simultaneously, cannot oppose the learned team-value gradient at first order. Each agent can therefore compute its guidance using only its own observation and candidate action, without evaluating the centralized mixer or accessing teammates’ actions. This gives a concrete role to monotonic factorization in test-time guidance: it provides alignment at each differentiable query action encountered during sampling, beyond consistency between final greedy choices.
Our extended analysis establishes the same alignment property for QFQL and GradStep in Section 5.8 and Appendix J; see the revised theoretical analysis, experiment results and our response to W1/Q2.
Third challenge: obtaining useful guidance from an offline critic. This alignment provides a structural basis for coordination, while practical improvement also depends on where and how the critic is queried during generation. Generated actions can move beyond reliable offline support, and locally plausible refinements can produce unfamiliar joint combinations. The query mechanism and guidance strength therefore remain consequential design choices. Our CTDE–FD comparisons, fixed-checkpoint guidance ablations, and QFQL/GradStep replacements examine these effects, connecting locally executable team alignment to measured improvements in return.
Response to reviewer's concerns (3/3)
Q4. Generality beyond locomotion and navigation: validation on manipulation tasks
Additional evaluation on manipulation tasks. Thank you for this suggestion. To assess applicability beyond locomotion and navigation, we additionally evaluated goal-conditioned CQF (GCCQF) on two bimanual manipulation tasks from MangoBench [3]: LiftBarrier and PlaceFood. These tasks involve different coordination requirements: LiftBarrier requires two robotic arms to grasp and lift a barrier together, whereas PlaceFood requires sequential cooperation, with one arm opening a pot lid before the other places food inside. They therefore extend our evaluation to both synchronous and asynchronous cooperative manipulation.
Comparison with MangoBench baselines. We use the same offline datasets, observation inputs, evaluation random seeds and final-task success criteria as MangoBench. The IQL expectile for both task is 0.7, the horizon H for LiftBarrier is 5 and H for PlaceFood is 1. The table below compares our results with the baselines reported in Figure 2 in its main text and Table 7 of its supplementary material.
Final task success rate (%; higher is better).
| Task | IHIQL | GCMBC | ICRL | DP | GCCQF (ours) |
|---|---|---|---|---|---|
| LiftBarrier | 82.0 | 47.0 | 56.0 | 58.0 | 96.0 |
| PlaceFood | 23.0 | 21.0 | 35.0 | 20.0 | 35.0 |
On LiftBarrier, GCCQF achieves 96% success, exceeding the strongest reported baseline, IHIQL, by 14 percentage points. On PlaceFood, its 35% success rate matches the best reported point estimate, achieved by ICRL. The results demonstrate successful use of the guided framework in cooperative manipulation. These experiments broaden the empirical validation beyond locomotion and navigation and support the framework’s applicability to different forms of cooperative robotic control.
Reference.
[1] FACMAC: Factored Multi-Agent Centralised Policy Gradients
[2] Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
[3] MangoBench: A Benchmark for Multi-Agent Goal-Conditioned Offline Reinforcement Learning
Response to reviewer's concerns (1/3)
We sincerely thank the reviewer for the careful assessment and constructive feedback. Your comments help us clarify CQF’s connection to existing MARL methods and its compatibility with alternative guidance mechanisms, while motivating stronger baseline comparisons and broader evaluation across task types and agent scales. We address each concern below.
Thank you for suggesting these complementary analyses. We have extended both the theory analysis and experiments to examine whether team-aligned local guidance remains effective beyond QGF and how CQF compares with direct transfers of alternative guidance methods.
Compatibility beyond QGF.The alignment property follows from the monotonic critic factorization and holds at any differentiable joint query action. It therefore applies to both QFQL’s intermediate-action guidance and GradStep’s post-denoising refinement. Our extended analysis establishes first-order alignment of their local guidance directions with the learned team value at the query action. The derivation is provided in the revised Appendix J.2 Team-aligned local guidance across different gradient estimators.Effectiveness with alternative guidance.We then test whether this compatibility translates into practical benefits by replacing the guidance rule with QFQL or GradStep under fully-decentralized (FD), team-aligned CTDE, and centralized formulations. The results are shown in the Table below. CTDE's advantage over FD on most dataset splits supports the effectiveness of team-aligned local gradients across different test-time guidance rules. The comparison appears in Section 5.8 CQF with different test-time guidance mechanisms, with experimental details and per-dataset results in the corresponding Appendix J Details of the Different-Guidance Experiment.Different test-time guidance methods under independent, team-aligned, and centralized settings.
Comparison with direct-transfer baselines.The same experiment also supplies two independent offline MARL baselines, FD (QFQL) and FD (GradStep), which apply the respective test-time guidance rules independently to each agent. Comparing these results with the original MA-MuJoCo and MPE tables (Table 1 and Table 2 in the paper) shows that CQF achieves higher average performance than both baselines on both benchmarks. Together, the replacement experiments establish compatibility with alternative guidance mechanisms, while the direct-transfer comparisons demonstrate the practical advantage of the complete CQF framework over these independent extensions.Thank you for highlighting the need to evaluate larger teams. We extend the landmark-covering evaluation to 3, 10, 20, and 40 agents, increasing the joint action dimension from 6 to 80, and compare CQF with FD under fixed data and training budgets and a common guidance weight β=0.3 (see Appendix K Scaling the number of agents for the full protocol and results).
Guidance remains effective with larger teams.CQF achieves higher mean return than FD at every tested team size, while also improving over its own unguided flow. This benefit extends across the guidance sweep: at every tested positive β up to 0.3, CQF produces larger gains over its own unguided checkpoint than FD does, at all four team sizes. These results support the continued usefulness of team-aligned local gradients as the joint action space grows.Improved coordination throughout execution.The return gains are accompanied by faster landmark occupation. At 40 agents, CQF increases mean coverage from FD’s 81.60% to 83.13% and reaches 95.00% final coverage. Its mean coverage first reaches 90% at step 15, compared with step 17 for FD and 19 for the unguided flow, showing that guidance improves progress throughout the trajectory.The added experiments extend CQF’s empirical validation to teams of tens of agents, demonstrating gains in return and coordination under controlled training budgets.