acceptodds
Withdrawn from ICLR 2027

MeanMAC: One-Step MeanFlow Policies with Team Q-Learning for Multi-Agent Coordination

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Offline multi-agent reinforcement learning (MARL) requires policies that can represent diverse cooperative behaviors from fixed datasets while remaining efficient enough for multi-agent decision making. Although diffusion- and flow-based policies provide substantially greater expressivity than conventional unimodal policies, existing generative MARL methods either rely on iterative sampling at execution time or adopt a two-stage pipeline that first learns a multi-step joint generator and subsequently distills it into one-step executable policies, thereby introducing additional inference or training overhead and separating generative modeling from the final policy. To address this challenge, we propose MeanMAC: One-Step MeanFlow Policies with Team Q-Learning for Multi-Agent Coordination, which directly learns a shared one-step MeanFlow policy for multi-agent action generation and couples policy improvement through a centralized team-value objective. The shared actor maps each agent's local information and noise directly to an action in a single forward pass, while the resulting joint action is optimized through team-level Q-learning, allowing generative distribution modeling and cooperative policy improvement within a unified training procedure. Consequently, MeanMAC avoids iterative denoising or ODE sampling and separate teacher–student distillation while retaining efficient factorized action generation and team-level coordination. Extensive experiments across three offline MARL benchmark suites, covering 10 environments and 33 task–dataset configurations on MA-MuJoCo, MPE, and SMACv1, demonstrate strong performance across both continuous and discrete control.

Primary area:
reinforcement learning
Keywords:
Offline reinforcement learning, Multi-agent reinforcement learning, Flow matching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_M3kJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MeanMAC, which combines one-step MeanFlow policies with team Q-learning for offline multi-agent coordination. Each agent generates actions from its local observation, identity, and noise, while additive team-value learning and joint-action candidate selection guide policy optimization without iterative sampling or teacher–student distillation. Experiments are conducted on MA-MuJoCo, MPE, and SMACv1.

Critical strengths
  1. The method is clearly structured.
  2. The experiments are extensive and cover both continuous and discrete action tasks.
Critical weaknesses
  1. No code is provided.
  2. The novelty is relatively limited. MeanFlowQL[1] and OM2P[2] have already combined MeanFlow with Q-learning. The main additions are additive team-value learning and joint-action candidate selection.
Questions for authors
  1. MeanMAC uses VDN-style additive value decomposition. Have the authors considered alternatives, such as QMIX-style nonlinear mixing or directly modeling a joint Q-function? Does the w/o IGM ablation include either approach?
  2. OM2P[2] already combines MeanFlow with Q-value guidance for offline MARL. Could the authors clarify the differences between MeanMAC and OM2P and provide experimental comparisons to demonstrate the benefit of these differences?
  3. Have the authors considered other one-step generators, such as Shortcut models[3]? Please explain why MeanFlow was chosen, how it differs from Shortcut models in these tasks, and ideally provide a comparison under the same settings.
  4. Each agent uses independent noise, so the base joint policy is a product of local policies. When multiple cooperative modes require agents to make matching choices under the same observation conditions, how does the method avoid mismatched modes across agents? Could the authors provide further analysis or experiments?

If the authors address my concerns, I will increase my score.

References [1] One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow [2] OM2P: Offline Multi-Agent Mean-Flow Policy [3] One Step Diffusion via Shortcut Models

Reviewer_hKEkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Offline MARL requires policies that can represent multi-modal cooperative behavior from a fixed dataset while remaining efficient at execution. Several existing generative approaches incur additional computational cost through iterative sampling or a separate teacher–student distillation stage. This paper proposes MeanMAC, which directly couples a one-step MeanFlow policy with team-level Q-learning. A shared actor maps each agent’s local observation, identity, and noise to an action in a single forward pass; a VDN-style team critic evaluates the resulting joint action, and Bellman targets are formed by selecting the best of K joint candidates. The MeanFlow loss, the team-value policy loss, and the Bellman loss are optimized jointly end-to-end without distillation. Across 33 task–dataset configurations on MA-MuJoCo, MPE, and SMACv1, MeanMAC attains the highest average performance over diffusion- and flow-based baselines, and the authors further report O(N) one-step inference, scalability in the number of agents, dataset-support alignment, and offline-to-online adaptation.

Critical strengths

S1. An important research topic in offline MARL

MeanMAC directly trains an executable one-step generative policy with offline behavior modeling and team-value guidance, without a separate teacher–student distillation stage. The shared actor produces each agent’s action from its local observation, identity, and noise in one forward pass. This provides a concrete integration of generative action modeling and value-based policy improvement for decentralized execution.  

S2. Strong benchmark performance with complementary analyses

Across 33 task–dataset configurations, MeanMAC demonstrates strong overall performance on MA-MuJoCo, MPE, and SMACv1. Component ablations and offline-to-online learning curves provide quantitative evidence for the framework’s effectiveness. Rollout and t-SNE visualizations offer complementary qualitative illustrations of the generated behaviors and their relationship to the offline data.

Critical weaknesses

W1. Positioning the contribution relative to prior work

Several components build on ideas explored in prior work: residual MeanFlow-based Q-learning and Best-of-K selection [1], additive value decomposition [2], and Q-guided one-step policies for offline MARL in OM²P [3]. MeanMAC brings these ideas together through a shared residual MeanFlow actor, an IGM-compatible team critic, and selection among complete joint-action candidates. Further clarifying the advances enabled by this integration, supported by a comparison with OM²P under matched model and training budgets, would help establish the contribution relative to closely related approaches.  

W2. Further evaluation of the coordination mechanism

The ablations in Figures 5 and 10 support the usefulness of team-level value coupling, and Appendix A.3 explains why the method selects complete joint-action candidates. It remains difficult to isolate the contribution of this selection rule from the other differences between variants. Clarifying how parameter sharing, critic capacity, and candidate selection are configured in the Independent and w/o IGM variants would make the comparisons easier to interpret. A controlled comparison of joint and per-agent candidate selection under matched sampling and training budgets would further clarify the specific benefit of selecting complete joint actions.

[1] Zeyuan Wang et al. One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow. AAAI, 2026.

[2] Peter Sunehag et al. Value-Decomposition Networks For Cooperative Multi-Agent Learning. arXiv:1706.05296, 2017.

[3] Zhuoran Li et al. OM²P: Offline Multi-Agent Mean-Flow Policy. AAMAS, 2026.

Questions for authors

Q1. Action selection during evaluation

Do Tables 1–3 use actor-only sampling or critic-ranked Best-of-K selection? Could you clarify how the actor-only deployment description relates to action_mode = best in Appendix C?  

Q2. Discrete-action implementation on SMACv1

How are continuous MeanFlow outputs converted into discrete actions on SMACv1? How are gradients and unavailable-action masks handled during training and evaluation?

Reviewer_8QiEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MeanMAC combines a shared one-step MeanFlow actor (residual, and conditioned on agent identity) with additive local critic ensembles, trained with a team Bellman error. Each agent's policy maps its local observation, its identity and its own noise to an action in one forward pass. Training combines MeanFlow regression with Q-guided actor improvement. For Bellman targets, the method generates several complete joint actions and uses the best one. Compared with MAC-Flow, it removes the joint flow teacher and the distillation stage. The evaluation covers 33 task–dataset settings on MA-MuJoCo, MPE and SMACv1, plus ablations, qualitative coordination analyses and offline-to-online experiments.

Critical strengths

S1. A plausible simplification of generative offline MARL: Training the executable one-step generator directly avoids keeping a multi-step joint teacher and a distillation loss. The paper clearly states what local information each agent uses, gives a JVP-based target, and writes out the team Bellman equations. The IGM property holds for the additive critic, and the paper correctly notes that a monotonic learned score is not the same as a monotonic environment return.

S2. Broad and promising results: There are large gains on several MPE tasks and a higher average return on MA-MuJoCo, and SMAC is competitive overall. The paper includes ablations, seed variation and even negative sensitivity results. It also admits that its t-SNE plots do not prove support containment. These results make the method worth studying, even without a new theoretical guarantee.

Critical weaknesses

W1. The controls do not show what the coordination part adds over the closest prior methods:

  • With frozen critics, Eqs. (12), (15) and (46) reduce the actor loss to the average of each agent's local Q-value at its own generated action. So each action gradient is local. Shared parameters and the shared team TD error do couple learning, so I am not saying that MeanMAC is the same as independent training or that it cannot coordinate.
  • But the "Independent" ablation is only described as using local value objectives, which does not say how it differs from this actor loss. Neither "Independent" nor "w/o IGM" has a full description of its critic loss, architecture or target selection (§5.7, Appendix D.2). So the gaps cannot tell us whether the gain comes from team TD fitting, parameter sharing, the generator, or candidate selection.

This matters because of close prior work:

  • MeanFlowQL (Zeyuan Wang et al.) already gives this residual target, and its Eqs. (19)-(20) give direct Q-learning and candidate targets.
  • OM2P (Zhuoran Li et al.) already trains one-step MeanFlow MARL policies with Q supervision, although with local critics.
  • MAC-Flow (Dongsu Lee et al.) already combines a one-step actor with an averaged team critic.

None of them is the full method, but the remaining combination needs a matched comparison or a precisely defined control.

Also, the product policy in Eq. (8) cannot keep arbitrary correlated modes at a fixed observation. For example, if the target is an equal mix of (0,0)(0,0) and (1,1)(1,1), independent agents that match these marginals produce mismatched actions half of the time. The agents can still coordinate by deterministically picking one mode. So the rollout examples should separate useful coordination from keeping multimodal joint behavior.

W2. The evaluated algorithm and the uncertainty reporting need a clear, consistent description:

  • SMAC actions. For the eleven SMAC settings, the paper does not say how MeanMAC represents actions as continuous values, decodes them into discrete actions, masks illegal actions, or passes the actor gradient. The categorical descriptions in Appendix C.5 are about the baselines.
  • Update rule. Algorithm 1 says to update both θ\theta and ϕ\phi on the combined loss, without saying whether the critic is frozen for the actor loss. Minimizing them jointly would change the Bellman fit. Separate updates are probably intended, but this should be stated.
  • Figure 1. The MeanFlow loss drops the adaptive weight wiw_i of Eq. (11). The critic loss squares an ensemble-averaged team Q, instead of averaging per-member squared errors as in Eq. (18). The selector writes an action as equal to an argmax over kk, and the derivative is printed with an undefined symbol. The target-critic labels are also inconsistent. These need fixing before the figure can guide an implementation; they do not show which loss the real code used.
  • Baseline numbers. All 77 baseline mean±std pairs in Table 1, and 66 pairs in six baseline columns of Table 3, exactly match MAC-Flow. That paper uses six seeds and two standard deviations, while MeanMAC labels these tables as five seeds and one standard deviation. For example, Hopper Medium / MAC-Flow is 1023.5±253.01023.5\pm253.0 in both. Reusing published baselines is fine, but the paper should say which numbers are copied and which are rerun, and what ± means, before comparing variability. This does not mean that the means are wrong, or that the authors did not run their own method.
  • Deployment. Appendix B promises one-sample local deployment, but the hyperparameter tables (Tables 5–8) list a "best" action mode with K=5K=5, without separating training from evaluation. Algorithm 1 uses ranking only in Bellman training, so I do not assume centralized evaluation. But the paper should say which policy produced Tables 1–3 and the KK sweep. If ranking is used at execution, it needs team information and extra inference cost.

Minor: Figure 1 in Microsoft Edge. In Edge, most math symbols in Figure 1 show up as empty boxes, so the formulas cannot be read. The figure is reported to look fine in macOS Preview, and it is readable in MuPDF and Poppler. Please check that the PDF export works across readers. This is a small presentation issue and does not affect my rating.

Questions for authors

Q1. What do the ablations change? Please give the exact objectives and the changed parts for "Independent" and "w/o IGM". Which matched result separates team Bellman credit assignment from the inherited generator and parameter sharing? A focused comparison with OM2P, or another matched one-step control, would help. This is a suggestion; I am not asking you to rerun every benchmark.

Q2. How does the method run on SMAC and at test time? How are SMAC actions encoded, decoded and masked, and how does the gradient of Eq. (15) reach the actor? Are θ\theta and ϕ\phi updated with separate losses? Please give the candidate counts for training and for evaluation, and confirm which policy produced the main returns.

Q3. Where do the baseline numbers come from? Which baseline entries were copied and which were rerun, and what do the ± values mean? Please explain the five-seed / one-SD captions, given that the matching source numbers use six seeds and two SDs. A small runnable config covering these policy and evaluation choices would also help readers test the method.

Reviewer_j4DbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MeanMAC, a one-step generative framework for offline multi-agent reinforcement learning. It combines a shared MeanFlow-based actor with team-value Q-learning to enable efficient decentralized action generation and coordinated policy optimization without iterative sampling or teacher–student distillation. Experiments on MA-MuJoCo, MPE, and SMACv1 demonstrate competitive performance, supported by coordination analyses, ablation studies, and offline-to-online adaptation.

Critical strengths
  1. The paper addresses a meaningful challenge in offline MARL by enabling efficient one-step generative action generation without iterative sampling or distillation.

  2. The experiments cover diverse benchmarks, including MA-MuJoCo, MPE, and SMACv1, with additional ablation studies, coordination analyses, and offline-to-online adaptation, providing relatively comprehensive empirical validation.

Critical weaknesses
  1. MeanMAC mainly combines existing techniques from MeanFlow, OM²P, and MAC-Flow, raising concerns about its novelty.

  2. The ablation studies fail to clearly isolate the contributions of MeanFlow modeling and team-value Q-learning.

  3. The effectiveness of Best-of-K joint-action selection lacks sufficient validation, particularly against centralized joint critics and independent action selection.

Questions for authors
  1. Under the additive team-value decomposition (Eqs. 12–14), the advantage of Best-of-K joint-action selection over independent action selection remains unclear.

  2. The team-value actor objective (Eq. 15) appears equivalent to independent Q optimization, raising questions about the performance gap with the Independent variant.

  3. OM²P is a closely related MeanFlow-based multi-agent method, but a direct experimental comparison under consistent settings is missing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.