acceptodds
Withdrawn from ICLR 2027

MA-FPPO: Multi-Agent Flow-Pretrained Policy Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-agent flow policies learn cooperative behavior from fixed offline datasets, but often struggle to complete tasks in situations not covered by the offline data. In these situations, agents must both adapt to changes in the environment and coordinate with one another, yet action patterns learned offline are often insufficient for effective adaptation and coordination. To address this problem, we propose Multi-Agent Flow-Pretrained Policy Optimization (MA-FPPO), which uses online fine-tuning to improve the cooperative behavior of models pretrained with flow matching through new interactions with the environment. Building on the behavior learned during pretraining, we construct policies with explicit action likelihoods for discrete and continuous action spaces. We then update the pretrained model using shared team advantages to further improve coordination based on team performance. Our method achieves, on average, relative gains of 52.8% over the strongest listed offline baselines across 30 settings and 29.8% over purely online learning across 38 comparisons with matched online budgets and evaluation protocols. Code and additional materials are provided in the supplementary material.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multi-Agent Reinforcement Learning, Flow Matching, Offline Pretraining, Online Fine-Tuning, Policy Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Liv7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MA-FPPO, an offline-to-online fine-tuning framework for multi-agent flow policies. It first trains a one-step student through flow matching, distillation, and value guidance. The pretrained student is then converted into a categorical or Gaussian policy with explicit action likelihoods by fixing its latent input, enabling online PPO updates with a centralized critic.

Experiments across SMAC, SMACv2, MPE, and MA-MuJoCo evaluate different offline data qualities, pretraining methods, online policy constructions, and optimization components, demonstrating promising overall performance.

Critical strengths
  1. The paper provides a straightforward approach to converting pretrained multi-agent flow students into explicit probabilistic policies compatible with PPO, supporting both discrete and continuous control. The training procedure and major implementation details are reasonably well described.

  2. The experiments cover diverse cooperative tasks and offline data qualities, including comparisons with random initialization, behavior cloning initialization, and direct generative fine-tuning. The results support the practical effectiveness of the overall framework, complemented by ablations of reference KL regularization and centralized value estimation.

Critical weaknesses
  1. MAC-Flow already includes online fine-tuning experiments that continue training from offline checkpoints using newly collected interactions. However, this paper primarily uses MAC-Flow as an offline reference, while its Direct Flow Policy Fine-tuning baseline employs a newly constructed generative MAPPO implementation rather than the original MAC-Flow online update mechanism. Without a direct comparison against the original online MAC-Flow method under matched offline initialization and online interaction budgets, it remains unclear how much additional benefit is provided by the proposed fixed-latent policy construction and PPO optimization. This limits the evidence supporting the method's incremental novelty.

  2. The paper compares explicit probabilistic policies with fixed latents against direct generative fine-tuning, using the performance differences to support the proposed policy construction. However, the appendix does not fully specify the stochastic transition kernel, noise parameterization, sampling procedure, or PPO likelihood ratios of the generative baseline, making independent reproduction difficult. Furthermore, the two approaches differ not only in latent sampling but also in their exploration distributions and deployment policies. Consequently, the observed performance gap cannot be attributed specifically to fixing the latent or the proposed optimization mechanism without more controlled comparisons.

Questions for authors
  1. MAC-Flow already performs online fine-tuning from offline checkpoints using newly collected trajectories. Could the authors compare MA-FPPO directly against the original MAC-Flow online training procedure under identical offline initialization and matched online environment-interaction budgets? What specific advantages does the proposed PPO-based optimization provide over the online update mechanism in MAC-Flow?

  2. Could the authors fully specify the generative MAPPO baseline, including its sampling process, stochastic transition kernel, noise parameterization, number of generation steps, and PPO likelihood-ratio computation? In particular, how are action or trajectory likelihoods evaluated when random latent sampling and learned transition noise are involved during training? These details are essential for reproducing the key comparison.

  3. The current comparison between Gaussian MAPPO and generative MAPPO simultaneously changes latent sampling, exploration distributions, and deployment policies. Could the authors provide more controlled ablations that isolate the effect of fixing the latent while keeping factors such as exploration variance and deployment behavior comparable? This would help clarify which components are responsible for the improvements in training stability and final performance.

  4. The abstract and conclusion report an average improvement of 29.8% over purely online learning across 38 comparisons. However, the appendix does not explicitly enumerate these comparisons or provide the corresponding aggregation formula. Could the authors report the paired endpoint results, online interaction budgets, checkpoint selection criteria, aggregation procedure, and uncertainty estimates? Additionally, since offline data collection and pretraining costs are excluded, could the authors clarify the scope of the reported improvement?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Liv7

W1. MAC-Flow already reports online fine-tuning, but no comparison is made under the same conditions. The online stage of MAC-Flow keeps its offline objectives, namely offline Q-learning, a behavior regularizer, and distillation. It only adds newly collected data to the training set and does not optimize the policy with respect to online returns. MA-FPPO performs online policy optimization: PPO updates each agent's policy with a shared team advantage. The revised related work section states this difference. On the three tasks for which MAC-Flow reports online results in its Figure 5, MA-FPPO reaches higher final performance:

TaskMAC-Flow offline (its Tables 1–2)MAC-Flow after online training (end of its Figure 5)MA-FPPO (our main table)
HalfCheetah Medium (MA-MuJoCo)4358.5≈590010223.1
Spread Medium (MPE, normalized)80.1≈7587.5
5m_vs_6m Medium (SMAC)15.6≈1818.5

The MAC-Flow online values are read from the end points of its curves. MAC-Flow measures online training in gradient steps and MA-FPPO in environment steps, so the table illustrates the trend rather than a matched budget.

W2. Direct flow policy fine-tuning is described too briefly to be reproduced. Appendix A.2.2 now gives the complete description, marked in blue. The key difference from MAPPO lies in the likelihood ratio. MAPPO computes the ratio of new and old action probabilities under an explicit Gaussian or categorical policy. Direct flow policy fine-tuning treats the stochastic flow generation path as the policy and computes, for each agent, the ratio of new and old path densities over the whole stored path (Eqs. (14)–(15)), in which the fixed latent density cancels. Each agent's path ratio is clipped separately and weighted by one shared team advantage from the centralized critic, so all agents are updated toward the same cooperative outcome. Our comparison uses the one-step case of this construction (Eqs. (16)–(17)), and Algorithm 1 gives training and deployment, including the range and initial value of the exploration noise and the online budget, consistent with the supplementary code. The construction follows concurrent work on multi-agent flow policy optimization that has not yet been published, and the implementation evaluated here is our own. To our knowledge, no published work applies this construction in the multi-agent setting.

W3. The two online constructions differ in several aspects (random or fixed z, the source of exploration noise, and action selection at deployment), so it is unclear which aspect matters. The comparison is designed at the level of complete online policy constructions. Both start from the same pretrained student and use the same online budget, and they differ only in how the optimizable online policy is constructed, which is the choice a practitioner faces. At this level, the fixed latent construction is better in all 12 settings. The revision states the conclusion at this level, and the appendix additionally reports component ablations of reference KL regularization and the local critic.

W4. It is not stated which 38 comparisons give the 29.8% gain or how it is computed. Appendix Table 4 lists all 38 settings, and Eq. (13) gives the computation. For each setting, the relative gain is MA-FPPO return divided by the return of MAPPO trained from scratch, minus one. The 38 gains are averaged with equal weight. The mean is 29.8%, the median is 9.3%, and MA-FPPO is better in 23 of 38 settings. The 38 settings are those among the 48 in which both methods use 50M online environment steps. The budgets differ for 3m and 2s3z, where MA-FPPO uses 15M steps, and for the Good and Medium datasets of 8m, where MA-FPPO uses 31.122M steps. These 8 settings are excluded. SMACv2 uses only the Replay datasets, which excludes the 2 Good settings and leaves 38 of the 48. The comparison is made under the same additional online interaction, as discussed in our response to Reviewer skgs (W4).

Thank you for the detailed response. The authors have addressed my questions, and the clarifications are helpful.

Reviewer_skgsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies how offline-pretrained multi-agent flow policies can further improve performance on cooperative tasks through online interaction. The method first follows the pretraining procedure using a flow teacher, one-step student distillation, and offline value guidance. It then fixes the student's latent input to zero and uses its output as the logits of a discrete categorical policy or the mean of a continuous Gaussian policy. During the online stage, a centralized critic estimates shared team advantages, and the policy is updated through per-agent PPO objectives, optionally with KL regularization relative to the initial policy. The paper reports results on MPE, SMAC, SMACv2, and two multi-agent MuJoCo dataset collections, with comparisons against random initialization, BC pretraining, Direct Flow Policy Fine-tuning, and several component ablations.

Critical strengths
  1. The online policy construction is simple and supports both discrete and continuous actions. The method retains the pretrained student's parameters and initial deterministic action mapping, using explicit categorical or Gaussian likelihoods to enable PPO optimization with team advantages. For continuous actions, likelihoods are computed using samples before clipping; for discrete actions, the same available-action masks are used during sampling and updates. These details make the procedure straightforward to implement and provide a direct approach to online adaptation of existing offline generative policies.

  2. The experiments cover a broad range of settings and provide useful design comparisons. The paper evaluates multiple multi-agent benchmarks and offline data qualities, shows the offline-to-online learning process, and compares BC initialization, Direct Flow Policy Fine-tuning, reference KL, and critic information. The authors also explain the additional online interaction, different data sources, and scope of the simplified theory. These materials help assess the practical utility of the procedure across tasks.

Critical weaknesses
  1. The core method mainly integrates existing techniques, with insufficient independent technical contribution to offline-to-online multi-agent adaptation.

    The offline training stage of MA-FPPO follows the existing MAC-Flow procedure, including the flow teacher, student distillation, and offline value guidance. The online stage uses the centralized critic, team advantage estimation, and PPO clipping of standard MAPPO. The main new design is to fix the student's latent input and convert its output into a categorical or Gaussian policy, enabling conventional online policy optimization. However, the paper does not propose a new adaptation structure or optimization mechanism specifically addressing multi-agent coordination failures caused by insufficient offline data coverage. Nor does it sufficiently explain why combining flow pretraining with MAPPO would provide adaptation benefits distinct from those of general policy initialization. In particular, the online stage no longer uses the full generative distribution of the flow policy, and the paper does not explain how the multimodal cooperative behavior learned during pretraining helps agents handle uncovered states after this conversion.

    Appendix A.2.1 explains fixing the latent input as a way to separate pretrained behavior from online exploration, while Appendix B.6 analyzes the optimization error introduced by latent sampling under a simplified single-step affine policy model. These results do not address actual multi-agent coordination, the problem setting's "situations not covered by the offline data," or the complete MAPPO training process. Nor do they prove that this combination provides a specific adaptation advantage. The paper's Limitations and appendices note that the existing experiments have not separately identified the effects of factors such as latent randomness and multimodal action modeling. Although the authors compare flow and BC pretraining, as well as Gaussian and generative online policies, these comparisons still do not individually distinguish the contributions of fixing the latent input, constructing an explicit probabilistic policy, and related design choices to final performance. Thus, the current method is closer to an implementation approach for converting a MAC-Flow pretrained policy into a standard MAPPO initialization, and its independent technical contribution and relevance to the stated research problem have not been sufficiently substantiated.

  2. The main experiments do not directly validate adaptation under insufficient offline data coverage, and there are also limitations in the training statistics and key comparison conditions.

    The paper presents multi-agent coordination failures in states not covered by the offline data as its main motivation, using coordination failure after an environmental change as an example in Figure 1. However, the main experiments pretrain on offline data of different qualities in the corresponding benchmark environments and then perform online MAPPO fine-tuning. They do not explicitly construct or quantify the distributional difference between the offline data and the states visited online, nor do they evaluate coordination degradation and recovery under controlled conditions of insufficient offline data coverage. Low-quality offline data may lead to a poor initial policy, but this does not directly establish that the observed performance improvements arise from adaptation to uncovered states. Thus, the current results mainly support the finding that pretrained policies can improve task returns through further online reinforcement learning. They are not yet sufficient to validate the recovery of coordination under the insufficient offline data coverage emphasized by the paper.

    In addition, §4.1 of the main text states that error bands are computed using three independent training seeds, whereas the supplementary code README describes the existing error bands as results from three evaluation seeds for the same trained model and states that these bands do not reflect variation across independent training seeds. These two statistical protocols cannot be directly equated, making the training stability of the main performance results difficult to verify. The reported average performance gain of 29.8% over purely online MAPPO is also based only on matched budgets for additional online interaction and excludes the costs of offline data collection and MAC-Flow pretraining. Therefore, the experiments do not fully validate the claimed adaptation benefits in situations not covered by the offline data. The reported 29.8% gain supports a comparison under matched additional online-interaction budgets, but does not establish an advantage in total training cost.

Questions for authors
  1. The mechanism of combining MAC-Flow pretraining with MAPPO online optimization:

    The paper connects an existing MAC-Flow pretrained model to standard MAPPO by fixing the student's latent input and constructing an explicit categorical or Gaussian policy. The online stage no longer uses the full generative distribution of the flow policy. Please explain what pretrained behavioral information useful for online policy adaptation is retained by this conversion and how it helps agents adjust their cooperative behavior in states not covered by the offline data. What specific advantages does this design offer over general BC-pretrained initialization? Can the existing method analysis or ablation results distinguish the contributions of flow pretraining itself, fixing the latent input, and constructing an explicit action distribution?

  2. Experimental validation of adaptation and coordination in situations not covered by the offline data:

    The paper presents coordination failures in states not covered by the offline data as its main motivation and uses a change in environmental geometry as an example in Figure 1. However, the main experiments use offline data of different qualities and perform online fine-tuning in the corresponding benchmark environments. Please clarify the specific definition of "insufficient offline data coverage" in the experiments and identify which experimental settings confirm that the online policy visits states insufficiently covered by the offline data. Could you provide an analysis of the distributional difference between the offline data and online states, or compare performance degradation and online recovery for MAC-Flow initialization, BC initialization, and MAPPO trained from scratch under controlled distribution shifts, to validate the actual benefits of the method for cooperative adaptation under insufficient offline data coverage?

  3. Independent training runs and the reliability of performance statistics:

    Regarding the discrepancy in statistical protocols discussed in the weaknesses, please clarify which tables and curves correspond to each of the two protocols described in the main text and supplementary material, and provide the actual numbers of independent training runs and evaluations used in the main experiments, together with the rules for computing the error bands. Could you provide performance results for individual training seeds to establish the stability of the reported main performance advantages and the 29.8% average relative improvement across independent training runs?

2 comments
AuthorsOpenReview ↗

Response to Reviewer skgs

W1. The contribution is limited. It is unclear what pretraining retains, why it is better than BC initialization, and how much each design contributes. Pretraining retains an initial policy that already selects high-return actions. Behavioral cloning learns the average action of the data, and when several good actions coexist, the averaged action can be poor. After the same online training, flow pretraining outperforms BC initialization in all 4 settings. The contribution of each design is isolated by two single-factor comparisons: changing only the pretraining method (flow vs. BC, 4/4) and changing only the online construction (fixed latent vs. direct fine-tuning, 12/12). The appendix additionally reports component ablations of reference KL regularization and the local critic.

W2. The paper claims to address out-of-distribution situations, but the experiments do not construct or quantify them. Thank you for this point. The revision states the motivation more precisely: the quality of offline data limits the performance of offline policies, and online training is used to go beyond this limit. The experiments are organized by dataset quality and quantify this directly. In 46 of 48 settings, online fine-tuning surpasses the pretrained initialization, with a mean relative gain of 102.6%, and gains are typically larger on lower-quality data (Appendix A.2 and the relative gain heatmap). Figure 1 and the introduction have been updated accordingly.

W3. The main text mentions three training seeds, while the code description mentions three evaluation seeds. The reported results use three training seeds (0, 1, and 2). The code description in the supplementary material has been corrected to match, and the launcher runs these three seeds by default (see General response, item 3).

W4. The 29.8% gain does not account for offline data collection and pretraining costs. The question studied in this paper is whether online training can further improve a pretrained policy beyond the level supported by the offline data, including expert data. We therefore compare results under the same amount of additional online interaction. The revision states this protocol in the abstract and the main text. At deployment, MA-FPPO uses the same one-step network as the pretrained model, so the inference cost is identical.

The authors' response provides a clearer explanation of how pretrained information is retained and how the online policy is constructed, and the existing comparisons offer positive evidence for the overall method. However, since the approach primarily combines existing MAC-Flow pretraining with MAPPO online optimization, assessing its contribution still requires competitive offline-to-online baselines to evaluate the incremental gains of this combination. The two comparisons cited in the response remain limited in scope: comparing Flow pretraining with BC pretraining involves differences across the entire pretraining procedure, while comparing MA-FPPO against the authors' implementation of Direct Flow Policy Fine-tuning (generative MAPPO) only supports an advantage over that generative baseline. Table 5 extends the controlled comparison with Direct Flow Policy Fine-tuning to 45 settings, further supporting MA-FPPO's advantage over that implementation, but it still does not answer how much incremental gain MA-FPPO brings over the original MAC-Flow online adaptation procedure under a matched online interaction budget. These results are informative, but offer limited support for the combination's advantage over existing offline-to-online methods.

As other reviewers noted, the original MAC-Flow online adaptation procedure is a relevant baseline. The authors added this comparison in their response, but the MAC-Flow online results are taken from curve endpoints in the original paper rather than from independent runs under comparable experimental settings, so the available description does not confirm that the two experimental configurations are comparable. Moreover, MAC-Flow and MA-FPPO report their online budgets in gradient update steps and environment interaction steps, respectively, and the authors explicitly acknowledge that the budgets are not matched. Consequently, the current comparison leaves it unclear how much of the reported performance difference is attributable to the proposed adaptation scheme itself. Overall, my concern regarding the method's incremental contribution remains unresolved, so I maintain my original rating.

Reviewer_u9UhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Multi-Agent Flow-Pretrained Policy Optimization (MA-FPPO), an offline-to-online framework for cooperative multi-agent reinforcement learning that fine-tunes offline-pretrained multi-agent flow policies through online interaction.

Critical strengths

The paper provides a simple and practical recipe for fine-tuning one-step flow student policies in an online setting and evaluates it across broad environments with varying dataset qualities. The comparisons of pretraining choices (flow vs. BC initialization) and online policy choices under a shared online algorithm offer useful empirical insight into offline2online multi-agent learning with flow policies.

Critical weaknesses

1. Limited novelty: As far as I understand, the offline pretraining comes from MAC-Flow and the online fine-tuning comes from MAPPO; that is, the proposed MA-FPPO is a simple combination of two existing methods. One possible opportunity for specializing this algorithm, I believe, is to modify how flow policies are used during online fine-tuning so as to benefit from the expressive capability of the offline-pretrained generative policies. However, the paper deliberately reduces the pretrained flow policy to a deterministic policy via a fixed latent (z=0z = 0), which raises questions about the novelty of the proposed method.

2. Experimental evaluation: This is perhaps the part of the current version of the paper that concerns me most. The claimed empirical improvement is not a fair comparison, since the headline comparison is largely reported against offline-only baselines, which receive no online interaction, whereas the MA-FPPO results are offline-to-online results, as shown in the main tables. While the paper acknowledges this as a limitation, presenting these numbers as the primary result in both the abstract and the conclusion may confuse readers, which worries me. This made me wonder whether online fine-tuning of offline-pretrained policies in the multi-agent setting has been so difficult that this comparison would be fair, i.e., whether it is meaningful simply to show that online interaction helps. However, when I checked the existing literature cited in this paper, such as MAC-Flow, I found that offline and offline-to-online results are reported separately. Therefore, I strongly recommend comparisons against offline-to-online baselines to demonstrate the effectiveness of MA-FPPO.

3. Clarity and presentation: I believe the clarity of the method description and the writing can be improved significantly, especially in Section 3, where the method is presented. For example, the offline objective in Eq. 3 defines only LFM\mathcal{L}_{\mathrm{FM}} in the main text. Ldistill\mathcal{L}_{\mathrm{distill}} and Lguide\mathcal{L}_{\mathrm{guide}}, which constitute most of the student policy optimization, appear only in the Appendix, and LQ\mathcal{L}_{\mathrm{Q}} is never written as an equation, including the multi-agent value aggregation.

I am willing to increase my score if these concerns, especially those regarding the experiments, are adequately addressed. Beyond novelty, I value practical instantiations of algorithms and works that propose readily usable recipes that the community may find helpful. However, for such work, I believe the empirical rigor should be properly justified, and the current version of the paper requires significant improvement in this regard.

Questions for authors

See Weaknesses please

4 comments
AuthorsOpenReview ↗

Response to Reviewer u9Uh

W1. MA-FPPO combines MAC-Flow and MAPPO, and fixing z = 0 discards the multimodality of the flow policy. The fixed latent is the construction selected by matched experiments. From the same pretrained student and with the same online budget, keeping z random is worse than fixing z in all 12 settings, as shown in Appendix A.2.2 and Figure 8. The multimodal information is used during offline pretraining. The flow teacher models the multiple action modes in the data. Distillation and value guidance then train the one-step student to select the higher-return mode. During online training, exploration comes from the added Gaussian noise, PPO receives explicit action likelihoods, and the shared team advantage updates every agent toward the same cooperative objective. The revised main text states this rationale.

W2. Comparing offline-plus-online results with offline-only methods is unfair. Methods of the same type should be compared. To our knowledge, no published work performs online policy optimization of multi-agent flow policies after offline pretraining, so the available public baselines are offline methods and online MAPPO trained from scratch. To provide comparisons of the same type, we implemented two offline-to-online alternatives: behavioral cloning pretraining followed by MAPPO, and flow pretraining followed by direct flow policy fine-tuning. Under the same online budget, MA-FPPO is better in all 4 settings of the first comparison and in all 12 settings of the second. MA-FPPO is also better than MAPPO trained from scratch in 23 of 38 settings. The revision places these comparisons first in the abstract, introduction, results, and conclusion.

W3. Section 3 is unclear, and several training objectives appear only in the appendix, some without equations. We revised Section 3 to give all four offline objectives with equations: the flow matching objective of the teacher, the distillation objective of the one-step student, the value guidance objective that steers the student toward higher-return actions, and the Q-value objective together with the aggregation of agent Q-values.

Thank you for the authors' efforts in addressing my concerns. However, being the first to combine particular existing algorithms does not, by itself, establish substantial methodological novelty. Given that MA-FPPO largely combines MAC-Flow and MAPPO with a fixed-latent student policy, comparisons against existing offline RL methods followed by online fine-tuning are essential but remain missing. The provided BC-to-online baseline lacks the value-based optimization used during MA-FPPO's offline pretraining, while direct flow policy fine-tuning is better characterized as an ablation of the proposed policy construction than as an independent, competitive offline-to-online baseline. Moreover, continuing MAC-Flow training with online data using its original offline objectives would constitute a meaningful baseline, regardless of the difference in optimization objectives, and many other instantiations are possible. As my concerns regarding methodological novelty and empirical rigor remain unresolved, I maintain my original score.

AuthorsOpenReview ↗

Response to Reviewer u9Uh: Continued MAC-Flow Training and Online Baselines

We thank the reviewer for the follow-up and would like to clarify two points.

1. Continued MAC-Flow training with its original objectives is already reported in the MAC-Flow paper. RQ3 and Figure 5 of MAC-Flow continue from the offline checkpoint and optimize the same offline objectives on newly collected rollouts. This procedure applies an offline training method to new data and does not optimize the policy with respect to online returns, so it remains an offline method rather than online policy optimization. The same Figure 5 also reports existing offline RL methods (OMIGA, TD3BC, and CQL) fine-tuned online under the same protocol. Both sets of results are listed in the table below, which is also added to the revised Appendix A.2.4 as Table 6 (blue text).

2. Online policy optimization built on the MAC-Flow pretrained student is reported in the revised Appendix A.2.3, Table 5 (blue text). This baseline keeps the random latent and performs online optimization with shared team advantages, which is direct flow policy fine-tuning, and it is evaluated at the same online steps as MA-FPPO. Across 45 settings, MA-FPPO attains the higher return in 38, direct flow policy fine-tuning in 4, and the two differ by less than 0.5% in 3.

TaskOMIGA (online fine-tuned)TD3BC (online fine-tuned)CQL (online fine-tuned)MAC-Flow offline → continued with offline objectivesDirect flow policy fine-tuning (matched online steps, Appendix Table 5)MA-FPPO
HalfCheetah Medium≈2750≈3500≈29004358.5 → ≈59008153.6610223.11
Spread Medium (normalized)≈30≈30≈−980.1 → ≈7541.4287.45
5m_vs_6m Medium——≈1415.6 → ≈1818.3118.50

The values from OMIGA to MAC-Flow are read from the end points of the curves in MAC-Flow's Figure 5. "—" indicates that the method is not shown in the corresponding panel of that figure.

Thank you for adding Table 6. However, the results for OMIGA, TD3BC, CQL, and MAC-Flow are extracted from the previously published MAC-Flow paper rather than reproduced under a common experimental protocol. I believe several experimental settings may differ between the two papers, and while I attempted to compare them, I found it difficult to establish their comparability, such as the online training budgets, which are reported in different units (gradient updates versus environment transitions). Although the revised version of the paper acknowledges that the budgets differ, it does not provide sufficient details to establish comparability. I would therefore appreciate a comprehensive clarification of the experimental settings across these methods, rather than relying on cross-paper results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers: summary of revisions and main results

We thank all reviewers for their careful reading and constructive feedback. All changes in the revised PDF are marked in blue.

Main results. Across 48 task–quality settings, online fine-tuning with MA-FPPO improves over its own pretrained initialization in 46 settings, with a mean relative return gain of 102.6%. Across the 30 settings with published offline results, MA-FPPO exceeds the strongest listed offline reference by 52.8% on average. Deployment uses the same one-step network as the pretrained model, so the inference cost is unchanged.

1. Novelty. To our knowledge, MA-FPPO is the first method that applies online policy optimization driven by team returns, here PPO with shared team advantages, to multi-agent flow policies after offline pretraining. The online stage of MAC-Flow keeps its offline training objectives and only adds newly collected data, so it does not optimize the policy with respect to online returns. Our contributions are a practical online construction, in which a fixed latent yields explicit action likelihoods for PPO with shared team advantages, and a systematic set of matched comparisons.

2. Comparisons with methods of the same type. The revised paper moves three matched comparisons to the front of the results. Each uses the same pretrained starting point and the same online budget.

  • Flow pretraining vs. behavioral cloning pretraining: MA-FPPO is better in all 4 settings.
  • Fixed latent construction vs. direct flow policy fine-tuning: MA-FPPO is better in all 12 settings.
  • MA-FPPO vs. MAPPO trained from scratch: MA-FPPO is better in 23 of 38 settings, with a mean gain of 29.8% and a median gain of 9.3%. Appendix Table 4 and Eq. (13) give the details.

The 52.8% gain over offline methods is reported as contextual evidence, and the revision states that MA-FPPO uses additional online interaction.

3. Training seeds. The reported results use three training seeds (0, 1, and 2). The code description in the supplementary material has been corrected accordingly, and the launcher runs these three seeds by default.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.