acceptodds
Under review as a conference paper at ICLR 2027

CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Making

Abstract

How to achieve both fast joint trajectory generation and effective coordination in generative offline multi-agent reinforcement learning? Multi-agent diffusion models generate joint trajectories through iterative sampling but incur high inference latency. Distillation reduces sampling steps, but it may cause the student model to lose information about the cross-agent dependencies learned by the teacher, leading to a lower coordination success rate. A directly trained joint generator can model coordination among agents during one-step or few-step generation. Nonetheless, insufficient modeling of cross-agent dependencies may still impair coordination. Furthermore, its consistency training process introduces heavy computational and memory costs. In this paper, we propose Coordinated few-step Flow (CoFlow), which directly learns a joint averaged-velocity field for distillation-free one-step and few-step multi-agent trajectory generation. CoFlow uses Coordinated Velocity Attention (CVA) to incorporate teammate trajectory information for more accurate prediction of each agent's trajectory. To reduce the cost of consistency training for this joint model, CoFlow approximates the correction term using finite differences. Across 48 configurations on MPE and SMAC, CoFlow supports centralized and decentralized execution. Under centralized execution, our CoFlow outperforms our reproduced baseline by 11.2%, averaging relative gains equally across both benchmark suites, with a 12.93× speedup in model sampling. Its CVA improves normalized scores by 165.1% on average over the same model with cross-agent attention disabled. Compared with the implementation using exact derivatives, finite-difference training achieves a 1.78× speedup and reduces peak GPU memory by 41.1%. Code is provided in the supplementary material.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
offline multi-agent reinforcement learning, generative policy, flow matching, few-step generation, multi-agent coordination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_TXrQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoFlow, which applies MeanFlow to joint trajectory generation for offline multi-agent decision making. It uses cross-agent attention to model coordination and finite differences to approximate directional derivatives in consistency training, reducing training costs. The model generates future observation trajectories in one or a few steps, then uses an inverse dynamics model to obtain actions. Experiments are conducted on MPE and SMAC.

Critical strengths
  1. The method is clearly structured.
  2. The experimental settings and implementation details are well described, and code is provided.
Critical weaknesses

The novelty of the paper is relatively limited.

Questions for authors
  1. MADiff [1] already uses cross-agent attention to model coordination. Could the authors clarify the specific architectural changes introduced by CVA compared with MADiff and the gains attributable to these changes?
  2. MeanFlow [2] already learns an average velocity field for one-step generation without distillation. Beyond the existing ablations, could the authors further clarify the contributions of the average-velocity objective, CVA, and finite differences? Could the authors also compare the total training costs required to reach the same task performance under matched datasets and backbones?
  3. During decentralized execution, each agent predicts trajectories for the entire team from its local information and executes only its own action. Different agents may predict inconsistent teammate plans. How does the method address this issue? Is there any targeted analysis or experiment?

If the authors address my concerns, I will increase my score.

References [1] MADiff: Offline Multi-agent Learning with Diffusion Models. [2] Mean Flows for One-step Generative Modeling.

1 comment
AuthorsOpenReview ↗

Response to Reviewer TXrQ: CVA vs. MADiff Attention, Component Contributions, and Decentralized Execution

Thank you for your positive view of our method structure and implementation details, and for your openness to raising the score.

Q1. How does CVA differ from MADiff attention, and what does it gain? CVA differs from MADiff attention in three ways. MADiff applies multi-head attention to the U-Net skip features and replaces those features with the attention output. CVA makes three changes. (1) It adds the message back through a residual update h + γc, so each agent keeps its own features. (2) Its gate γ is learnable and initialized at zero. Training therefore starts from an independent per-agent backbone and brings in teammate information gradually. (3) It applies the same gated coupling at the bottleneck, so teammate information reaches every resolution of the U-Net. The retrained ablation in Figure 4 measures the gain. Full CVA beats independent per-agent generation on four MPE configurations: 107.2 vs. 61.9 on World Medium (+73%), 131.7 vs. 119.4 on Tag Expert, 132.4 vs. 120.8 on World Expert, and 110.1 vs. 99.4 on Spread Expert (OMAR-normalized scores).

Q2. What do the averaged-velocity objective, CVA, and finite differences each contribute, and what is the total training cost? Each component solves one problem: few-step generation, coordination, or training cost. Section 3 now opens with this mapping.

  • Averaged-velocity objective. It makes one-step generation possible. CoFlow-base shares the architecture but uses ordinary flow matching, and its one-step update collapses to the conditional mean (Appendix B.1).
  • CVA. It provides coordination, as the retrained ablation in Q1 shows.
  • Finite differences. They lower training cost without hurting the final policy. With the same network and batch size, each update is 1.78x faster than the exact JVP and uses 41.1% less peak GPU memory (Figure 6). Every CoFlow result in Table 1 comes from finite-difference training and beats independent runs of MADiff. Proposition B.1 bounds the deviation from the exact-derivative target by O((t-r)|ε|), with ε = 1e-3.
  • Total training cost. CoFlow trains in one stage. Distillation-based methods train a multi-step teacher and then a student, and some also train a Q-function.

Q3. How does CoFlow handle inconsistent teammate plans under decentralized execution? CoFlow-D and CoFlow-C share one set of weights, so one model runs in both modes without retraining. Under decentralized execution, each agent generates a joint plan from its own observations and executes only its own block. Each plan predicts the teammates' trajectories explicitly, and CVA aligns the agent's own behavior with them. CoFlow-D matches or beats CoFlow-C on 6 configurations in Table 1, for example 23.5 vs. 1.3 on MPE World Random, 75.0 vs. 66.8 on Tag Md-Replay, and 10.4 vs. 8.1 on SMAC 3m Poor. Appendix B.3 now gives the execution details. Each agent samples its initial noise independently, and the N local predictions run as one batch. Making the local plans more consistent is a direction for future work.

Reviewer_URQeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper trains a joint averaged-velocity model over all agents’ trajectories, with cross-agent attention in a shared U-Net and a finite-difference stand-in for the derivative term of the consistency loss, so that one to three sampling steps suffice. On MPE and SMAC it reports a gain over its own rerun of centralized MADiff and a large sampling speedup over multi-step diffusion.

Critical strengths

S1. The motivation is clear, and the problem of fast yet coordinated generation in offline MARL is of practical interest.

S2. Extensive experiments are conducted on two benchmarks, and the code is provided.

Critical weaknesses

W1. The method section is hard to follow. Section 3 is largely a sequence of definitions and equations, and it is unclear how these components address the problem raised in the introduction, i.e., why learning a joint averaged velocity preserves the cross-agent dependencies that distillation loses, and what in the gated attention fixes the stated insufficiency of existing joint generators. Key parts of the method, including guidance, time-pair sampling, action decoding and decentralized execution, appear only in the appendix.

W2. The main comparison in Table 1 is problematic. First, although the paper states that MADiff-C was reproduced with the official code, the reported numbers are identical to those in the MADiff paper [2] and the DoF paper [1], whose evaluation setup and datasets are not necessarily identical to those used here, so the comparison is unfair. Second, the method should be compared with MADiff-C, but on Spread the reported numbers are actually those of MADiff-D, the weaker decentralized variant, which makes the reported improvement misleading. When compared with the published MADiff-C results, CoFlow-C performs worse on MPE and comparably on SMAC.

W3. The novelty is limited. The method is essentially a combination of the MeanFlow objective [3] and MADiff’s attention-based joint U-Net [2], and the finite-difference approximation of the derivative term has already been used in OM2P [4], which is not cited.

Minor comments:

  • Sec. 4.3: the reported gain from attention is obtained by disabling attention at inference in a model trained with it, which is not a proper ablation.
  • Table 1: CoFlow results are mean ± one standard deviation over three seeds, while the DoF and MAC-Flow results taken from [5] are ± two standard deviations over six seeds.
  • Table 1: some baseline numbers differ from the cited sources, e.g., MA-CQL on Tag Expert (119.3 vs. 93.9 in [2]) and World Expert (119.8 vs. 71.9), and OMAR on Tag Expert (123.9 vs. 116.2).
  • Sec. 4.1: it is not stated how the number of sampling steps (1–3) was chosen for each configuration.

References:

[1] DoF: A Diffusion Factorization Framework for Offline Multi-Agent Reinforcement Learning. ICLR, 2025.

[2] MADiff: Offline Multi-agent Learning with Diffusion Models. NeurIPS, 2024.

[3] Mean Flows for One-step Generative Modeling. NeurIPS, 2025.

[4] OM2P: Offline Multi-Agent Mean-Flow Policy. AAMAS, 2026.

[5] Multi-agent Coordination via Flow Matching. ICLR, 2026.

Questions for authors

Please refer to the Weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer URQe: Independently Reproduced MADiff Results and Differences from OM2P

Thank you for checking Table 1 so carefully.

W2. MADiff numbers in Table 1. The revised Table 1 uses MADiff results from independent runs of the official code in other published work. The original MADiff numbers are hard to reproduce. The official repository (github.com/zbzhu99/madiff) releases no trained checkpoints, its README states that MADiff-C "is not meant to be a fair comparison with baseline methods," and a user reports irreproducible SMAC results there (issue #22, "Irreproducible result"). We take the MPE results from Table 11 of DoF (ICLR 2025) and the SMAC results from MAC-Flow (ICLR 2026, decentralized execution). The column is renamed MADiff, and the table caption and the Baselines paragraph state the sources. Both independent studies report lower values than the original paper. On World Medium, for example, DoF obtains 80.1 against 158.2 in the original paper. We also fixed the MA-CQL and OMAR transcription errors you found. Against the independent results, CoFlow-C raises the mean score by 18.6% on MPE and by 9.4% on SMAC. CoFlow-D performs on par with MADiff (65.5 vs. 65.8 on MPE, 14.26 vs. 14.34 on SMAC) with only 1-3 sampling steps per decision. The caption also states that the SMAC numbers from MAC-Flow use 6 seeds with ±2 standard deviations.

W3. Novelty and OM2P. We developed the finite-difference correction independently, and both its setting and its design differ from OM2P, which we now cite. (1) OM2P applies finite differences to separate per-agent action policies with no interaction between agents. CoFlow applies them to the joint trajectory velocity field of N agents, so the difference quotient contains the cross-agent Jacobian blocks that CVA introduces. (2) OM2P is a Q-guided decentralized actor-critic that generates actions. CoFlow is a joint trajectory planner without a Q-function, and one model runs both centralized and decentralized execution on MPE and SMAC. (3) We use a signed step with boundary handling, taking a backward difference near t = 1 and applying no correction at r = t. Proposition B.1 gives a local error bound. MeanFlow is a single-agent generator with no cross-agent modeling, and MADiff needs multi-step denoising. CoFlow generates the joint trajectories of the whole team in one to a few steps without distillation. The averaged-velocity objective enables few-step generation, the gated cross-agent coupling provides coordination, and the finite-difference consistency training lowers training cost.

W1. Readability of the method section. Section 3 now opens with a paragraph that links each problem to a component. A one-step Euler update of an instantaneous-velocity model collapses to the conditional mean, so CoFlow learns the averaged velocity over an interval. A backbone that sees only each agent's own features ignores teammates, so CVA adds gated cross-agent messages at every generation update. The averaged-velocity objective needs the network derivative along the sampling path, so CoFlow replaces the exact Jacobian-vector product with a finite difference that costs one extra forward pass and no backpropagation. Appendix B.3 now describes decentralized execution.

Minor comments.

  • α = 0 analysis. The revision states that the α sweep (Figure 3) is an inference-time intervention. The retrained ablation is in Figure 4, where full CVA beats independent generation on all four configurations.
  • Uncertainty conventions. The caption of Table 1 states the source and convention of each baseline.
  • Baseline numbers. We checked the entries you pointed out against the original papers and corrected them.
  • Sampling steps. Each configuration reports the best result within 1-3 steps, as the caption states. Appendix A.6 gives fixed one-step results.

Thank you for the response. However, some of my concerns have not been fully addressed, e.g., it remains unclear how the gated attention overcomes the limitation of existing joint generators. The many basic errors in the results of the original submission raise concerns about the reliability of the experiments, so I believe the paper requires a major revision before it can be accepted, and I will maintain my original score.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response: Corrected Baselines, Retrained Ablation, and Efficiency Details

We thank the reviewers for their feedback. The reviewers agree that fast and coordinated joint trajectory generation matters for offline MARL, and they note the breadth of our experiments and the released code. The revised PDF marks all review-related changes in blue. The main updates follow.

  1. Baselines in Table 1. The official MADiff repository releases no trained checkpoints, and its README states that MADiff-C "is not meant to be a fair comparison with baseline methods." We therefore use MADiff results from independent runs of the official code, taken from DoF (ICLR 2025, Table 11) for MPE and from MAC-Flow (ICLR 2026) for SMAC. Against these results, CoFlow-C raises the mean score by 18.6% on MPE and by 9.4% on SMAC, or 14.0% on average, and it samples 12.93x faster. We also fixed the baseline transcription errors that Reviewer URQe found.

  2. Relation to prior work. A new paragraph at the start of Section 3 states which problem each component solves. Section 3.2 lists how CVA differs from MADiff attention. We now cite OM2P and explain how our method differs from it.

  3. Ablation. Each variant in Figure 4 is trained from scratch with the same configuration. Full CVA beats independent per-agent generation on all four MPE configurations, and the gain reaches 73% on World Medium.

  4. Efficiency. CoFlow uses 1-3 sampling steps per decision. On one A100, a team decision takes about 26 ms of model forward time, 12.93x faster than MADiff with 15 DDIM steps (mean over 24 task-dataset pairs). Decentralized execution batches the local predictions of all agents and takes 26.6 ms with 3 agents and 38.2 ms with 8 agents. Training has a single stage with no teacher, distillation, or Q-function. Finite differences make each update 1.78x faster than the exact JVP and cut peak GPU memory by 41.1%.

  5. Reproducibility. Appendix A.6-A.8 add fixed one-step results, a latency and model-size comparison, and a full hyperparameter table. Appendix B.3 describes decentralized execution.

AuthorsOpenReview ↗

Response to Reviewer iFG4: Retrained Ablation, Model Size and Latency, and Reproduction Details

Thank you for reading the appendix closely and for recognizing our formulation, the boundary handling of finite differences, and the landmark-coverage analysis.

W1. Isolating the incremental contribution. Figure 4 isolates CVA with variants trained from scratch under the same configuration. It does not intervene on a trained network. Only the α sweep in Figure 3 uses inference-time intervention, and the revision now keeps the two analyses apart. Full CVA beats independent per-agent generation on all four MPE configurations, with a 73% gain on World Medium. Our response to Reviewer TXrQ (Q1) lists the three structural differences between CVA and MADiff attention. We agree with your view of CoFlow-base. It shows that the averaged-velocity objective is necessary, and the text now uses it only in this role.

W2. Quality and latency compared with MAC-Flow.

  • Quality. On SMAC, CoFlow-C and MAC-Flow reach similar mean returns (15.69 vs. 15.72). CoFlow reaches this level without a teacher model or distillation. Figure 1 now describes both families neutrally.
  • Sampling steps. Each entry in Table 1 reports the best result within a budget of 1-3 steps, as the caption states. Appendix A.6 (Table 4) reports fixed one-step results for every configuration. With return guidance, K steps take 2K network evaluations.
  • Latency. The two models differ in size by about two orders of magnitude, so raw latencies are not comparable. CoFlow has about 64.5M parameters, and the MAC-Flow student has 0.27M-0.80M. Their outputs also differ. The MAC-Flow student outputs the current actions. CoFlow generates the 24-step future trajectory of the whole team, which supports planning and lets one inspect coordination. Both run in milliseconds and meet the real-time needs of these tasks. Batching the local predictions keeps a decentralized team decision close to a centralized one in cost. The 12.93x speedup is measured against a multi-step diffusion planner that also generates full trajectories. Appendix A.7 (Table 5) reports the measurements on one A100.
TaskParams (CoFlow / MAC-Flow)CoFlow-CCoFlow-D (batched)MAC-Flow
Spread Expert64.5M / 0.80M26.4 ms26.6 ms0.16 ms
Tag Expert64.5M / 0.80M26.6 ms26.5 ms0.16 ms
3m Good64.6M / 0.27M26.6 ms26.5 ms0.35 ms
8m Good64.7M / 0.28M26.3 ms38.2 ms0.29 ms
Output per decision-24-step joint trajectory24-step joint trajectorycurrent actions

W3. Effect of finite differences on final performance. Finite differences do not degrade the final policy. Every CoFlow result in Table 1 comes from finite-difference training, and CoFlow-C beats independent runs of MADiff by 14.0% on average. Proposition B.1 bounds the deviation from the exact-derivative target by O((t-r)|ε|), which is small for ε = 1e-3. Each update is also 1.78x faster and uses 41.1% less peak GPU memory, which lowers the cost of training the joint multi-agent model.

W4. Decentralized execution and reproduction details.

  • Execution. Our response to Reviewer TXrQ (Q3) covers this. Each agent samples its initial noise independently, and the local predictions run in parallel (Appendix B.3).
  • Hyperparameters. The fraction of time pairs with r = t is ρ = 0.5, and the finite-difference step is ε0 = 1e-3. Table 6 (Appendix A.8) lists training steps, batch size, history length C, and planning horizon H for each task. The target return is set on the normalized scale, 0.9 for MPE and 1.0 for SMAC. W2 describes the sampling steps.

Thanks for the author's response.

The authors have clearly distinguished between the inference-time interventions in Figure 3 and the from-scratch ablation training in Figure 4. They have also provided additional results—specifically, fixed one-step outcomes, a latency comparison with MAC-Flow, and details on training parameters—while acknowledging that the average SMAC returns for CoFlow-C and MAC-Flow are comparable, and updating the wording in Figure 1 accordingly.

However, I have a few remaining questions. First, I would like to see a direct comparison between CVA and the original MADiff attention mechanism, using the same training objectives and budget, to better evaluate the gains provided by the new design. Second, I am curious whether parameter sharing has any impact when dealing with heterogeneous agents.

AuthorsOpenReview ↗

Response to Reviewer iFG4: CVA vs. Original MADiff Attention, and Parameter Sharing with Heterogeneous Agents

Thank you for your reply and for confirming the distinction between Figures 3 and 4, as well as the added fixed one-step results, latency comparison, and training details. We answer your two questions below.

Q1. Direct comparison between CVA and the original MADiff attention under the same training objective and budget.

We ran this comparison. The two variants share the same code, the same training objective as the Figure 4 ablation, and the same hyperparameters and budget (250k steps, batch size 32, learning rate 2e-4). Only the cross-agent attention module differs:

  • Original MADiff attention: the multi-head attention output replaces the U-Net skip features, with no residual path, no gate, and no attention at the bottleneck.
  • CVA: a residual update h + γc, where γ is a zero-initialized learnable scalar gate; attention is also applied at the bottleneck.

Scores are OMAR-normalized, evaluated over 10 episodes at the final checkpoint:

TaskOriginal MADiff attentionCVAGain
MPE World Medium98.2107.2+9.2%
MPE Tag Expert124.7131.7+5.6%
MPE World Expert115.3132.4+14.8%
MPE Spread Expert113.5110.1-3.0%
Mean of four tasks112.9120.4+6.6%

CVA improves the mean score by 6.6%. Relative to independent per-agent generation in Figure 4, CVA improves the mean score by 19.9%, while the original MADiff attention, which also adds cross-agent attention, improves it by 12.5%. The gain therefore comes mainly from the gated residual design of CVA, not only from adding cross-agent attention.

Q2. Does parameter sharing hurt heterogeneous agents?

All agents in CoFlow share one U-Net, and the inputs carry no explicit role labels. CVA assigns attention by content, so the shared network separates agent types on its own. In Figure 5, attention on SMAC-2s3z forms a block-diagonal Stalker/Zealot grouping without supervision, and attention on MPE-Tag concentrates among the predators.

Results on a heterogeneous task support this. SMAC 2s3z has two unit types (2 Stalkers and 3 Zealots), and CoFlow-C outperforms MADiff on all three datasets:

2s3z datasetMADiffDoFMAC-FlowCoFlow-CCoFlow-D
Good15.918.519.520.018.3
Medium15.618.117.618.718.8
Poor8.510.08.59.810.1
Mean13.3315.5315.2016.1715.73

CoFlow-C scores 21.3% higher than MADiff and 6.4% higher than MAC-Flow on average, so parameter sharing does not limit CoFlow on this heterogeneous task. Sharing also keeps the model size independent of the number of agents, and the same weights run decentralized execution (CoFlow-D) without retraining.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.