- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MA-FPPO, an offline-to-online fine-tuning framework for multi-agent flow policies. It first trains a one-step student through flow matching, distillation, and value guidance. The pretrained student is then converted into a categorical or Gaussian policy with explicit action likelihoods by fixing its latent input, enabling online PPO updates with a centralized critic.
Experiments across SMAC, SMACv2, MPE, and MA-MuJoCo evaluate different offline data qualities, pretraining methods, online policy constructions, and optimization components, demonstrating promising overall performance.
Critical strengths
-
The paper provides a straightforward approach to converting pretrained multi-agent flow students into explicit probabilistic policies compatible with PPO, supporting both discrete and continuous control. The training procedure and major implementation details are reasonably well described.
-
The experiments cover diverse cooperative tasks and offline data qualities, including comparisons with random initialization, behavior cloning initialization, and direct generative fine-tuning. The results support the practical effectiveness of the overall framework, complemented by ablations of reference KL regularization and centralized value estimation.
Critical weaknesses
-
MAC-Flow already includes online fine-tuning experiments that continue training from offline checkpoints using newly collected interactions. However, this paper primarily uses MAC-Flow as an offline reference, while its Direct Flow Policy Fine-tuning baseline employs a newly constructed generative MAPPO implementation rather than the original MAC-Flow online update mechanism. Without a direct comparison against the original online MAC-Flow method under matched offline initialization and online interaction budgets, it remains unclear how much additional benefit is provided by the proposed fixed-latent policy construction and PPO optimization. This limits the evidence supporting the method's incremental novelty.
-
The paper compares explicit probabilistic policies with fixed latents against direct generative fine-tuning, using the performance differences to support the proposed policy construction. However, the appendix does not fully specify the stochastic transition kernel, noise parameterization, sampling procedure, or PPO likelihood ratios of the generative baseline, making independent reproduction difficult. Furthermore, the two approaches differ not only in latent sampling but also in their exploration distributions and deployment policies. Consequently, the observed performance gap cannot be attributed specifically to fixing the latent or the proposed optimization mechanism without more controlled comparisons.
Questions for authors
-
MAC-Flow already performs online fine-tuning from offline checkpoints using newly collected trajectories. Could the authors compare MA-FPPO directly against the original MAC-Flow online training procedure under identical offline initialization and matched online environment-interaction budgets? What specific advantages does the proposed PPO-based optimization provide over the online update mechanism in MAC-Flow?
-
Could the authors fully specify the generative MAPPO baseline, including its sampling process, stochastic transition kernel, noise parameterization, number of generation steps, and PPO likelihood-ratio computation? In particular, how are action or trajectory likelihoods evaluated when random latent sampling and learned transition noise are involved during training? These details are essential for reproducing the key comparison.
-
The current comparison between Gaussian MAPPO and generative MAPPO simultaneously changes latent sampling, exploration distributions, and deployment policies. Could the authors provide more controlled ablations that isolate the effect of fixing the latent while keeping factors such as exploration variance and deployment behavior comparable? This would help clarify which components are responsible for the improvements in training stability and final performance.
-
The abstract and conclusion report an average improvement of 29.8% over purely online learning across 38 comparisons. However, the appendix does not explicitly enumerate these comparisons or provide the corresponding aggregation formula. Could the authors report the paired endpoint results, online interaction budgets, checkpoint selection criteria, aggregation procedure, and uncertainty estimates? Additionally, since offline data collection and pretraining costs are excluded, could the authors clarify the scope of the reported improvement?
2 comments
Thank you for the detailed response. The authors have addressed my questions, and the clarifications are helpful.
Response to Reviewer Liv7
W1. MAC-Flow already reports online fine-tuning, but no comparison is made under the same conditions. The online stage of MAC-Flow keeps its offline objectives, namely offline Q-learning, a behavior regularizer, and distillation. It only adds newly collected data to the training set and does not optimize the policy with respect to online returns. MA-FPPO performs online policy optimization: PPO updates each agent's policy with a shared team advantage. The revised related work section states this difference. On the three tasks for which MAC-Flow reports online results in its Figure 5, MA-FPPO reaches higher final performance:
The MAC-Flow online values are read from the end points of its curves. MAC-Flow measures online training in gradient steps and MA-FPPO in environment steps, so the table illustrates the trend rather than a matched budget.
W2. Direct flow policy fine-tuning is described too briefly to be reproduced. Appendix A.2.2 now gives the complete description, marked in blue. The key difference from MAPPO lies in the likelihood ratio. MAPPO computes the ratio of new and old action probabilities under an explicit Gaussian or categorical policy. Direct flow policy fine-tuning treats the stochastic flow generation path as the policy and computes, for each agent, the ratio of new and old path densities over the whole stored path (Eqs. (14)–(15)), in which the fixed latent density cancels. Each agent's path ratio is clipped separately and weighted by one shared team advantage from the centralized critic, so all agents are updated toward the same cooperative outcome. Our comparison uses the one-step case of this construction (Eqs. (16)–(17)), and Algorithm 1 gives training and deployment, including the range and initial value of the exploration noise and the online budget, consistent with the supplementary code. The construction follows concurrent work on multi-agent flow policy optimization that has not yet been published, and the implementation evaluated here is our own. To our knowledge, no published work applies this construction in the multi-agent setting.
W3. The two online constructions differ in several aspects (random or fixed z, the source of exploration noise, and action selection at deployment), so it is unclear which aspect matters. The comparison is designed at the level of complete online policy constructions. Both start from the same pretrained student and use the same online budget, and they differ only in how the optimizable online policy is constructed, which is the choice a practitioner faces. At this level, the fixed latent construction is better in all 12 settings. The revision states the conclusion at this level, and the appendix additionally reports component ablations of reference KL regularization and the local critic.
W4. It is not stated which 38 comparisons give the 29.8% gain or how it is computed. Appendix Table 4 lists all 38 settings, and Eq. (13) gives the computation. For each setting, the relative gain is MA-FPPO return divided by the return of MAPPO trained from scratch, minus one. The 38 gains are averaged with equal weight. The mean is 29.8%, the median is 9.3%, and MA-FPPO is better in 23 of 38 settings. The 38 settings are those among the 48 in which both methods use 50M online environment steps. The budgets differ for 3m and 2s3z, where MA-FPPO uses 15M steps, and for the Good and Medium datasets of 8m, where MA-FPPO uses 31.122M steps. These 8 settings are excluded. SMACv2 uses only the Replay datasets, which excludes the 2 Good settings and leaves 38 of the 48. The comparison is made under the same additional online interaction, as discussed in our response to Reviewer skgs (W4).