- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The submission discusses techniques to improve results of cooperative multi-agent vision-language action models. For this a 3-stage fine-tuning pipeline is proposed. It consists of: 1) task-specific data collection if initialization fails; 2) data/reward filtering (policy is only updated with positive advantage); 3) adopting DSRL (prior work).
Critical strengths
- Multi-agent vision language action models are timely.
- The paper is generally easy to follow and the proposed techniques are mostly reasonable.
Critical weaknesses
- I find the proposed techniques to contain little novelty: First, initialization-aware data collection (Sec. 4.1) has been studied in RL before, e.g., in areas such as "Learning from Observations" (LfO) where task-agnostic and task-specific datasets are considered. One can view the proposed human data collection as curation of a task-specific dataset while the remaining data are task-agnostic. Second, creating a high-quality subset of the data (by focusing on "transitions with positive advantage") which is then treated differently has also been considered. These techniques may differ from what was proposed in Sec. 4.2 but these related works weren't covered in the related work section. The online component (Sec. 4.3) essentially "adopts DSRL" (l.296).
- The related work section discusses very little of this prior work. Much of the prior work may have been performed in single-agent RL. However, I would have expected a more thorough discussion which outlines how the multi-agent settings differ.
- "we use CHORUS ..., and implement our ... on its checkpoint" (l.320ff) reads to me like the proposed techniques haven't been assessed on its own. Do they work on top of other baselines? Without such a study, it is hard to assess the generality of the proposed techniques.
- what are the caveats of focusing "only on transitions with positive advantage" (l.238)? Your favorite AI will give you an answer that should probably be discussed in this submission.
Minor points:
- the paper's writing is occasionally off, please fix.
Questions for authors
see above
7 comments
Official Response of Submission 13918 to Reviewer Y4pu, Part2
W4: Caveats of focusing "only on transitions with positive advantage"
We focus on positive-advantage transitions primarily for training efficiency and stability. For flow-matching VLAs, explicitly penalizing negative-advantage actions is challenging due to the intractable action log-likelihood. Moreover, including negative samples under a fixed training budget may dilute the supervision from beneficial actions. Regarding that beneficial actions can be directly reinforced using the standard flow-matching loss, we focus "only on transitions with positive advantage".
To investigate whether negative-advantage samples provide additional benefits, we implement an advantage-conditioned training baseline following the classifier-free guidance (CFG) formulation in (RECAP), which incorporates both positive- and negative-advantage data. As shown in the table below, we can find the comparable performance to our positive-only filtering, but the advantage-conditioned baseline incurs approximately 2x training time cost due to additional conditional training and CFG-related computation.
| Task | Credit-filtered tuning | Advantage-Conditioned CFG |
|---|---|---|
| Lift Pot | 42 | 44 |
| Handover Mic | 52 | 51 |
| Three Robots Stack Cubes | 61 | 59 |
| Take Photo | 69 | 66 |
| Average | 56 | 55 |
| Average Training Cost (H) | 20 | 45 |
These findings suggest that, in our setting, incorporating negative-advantage samples through a more complex training objective does not provide sufficient performance gains to justify the additional computational cost. We therefore favor positive-advantage filtering as a simpler and more efficient approach.
Invitation to reviewer-author discussion
Dear Reviewer,
We are the authors of Submission 13918: "Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning". We’ve uploaded a point-to-point response to your comments, which we believe addresses all of your concerns.
As the reviewer-author discussion period is going to end, we would greatly appreciate it if you could take a moment to share your thoughts on our rebuttal. Your feedback is valuable to us, and we’d be happy to further clarify any remaining points if needed.
Best regards, Authors of Submission 13918
Official Response of Submission 13918 to Reviewer Y4pu, Part1
We thank Reviewer Y4pu for their positive feedback on the contribution of the paper. Below is our detailed response:
Response
We acknowledge that individual components of our framework build upon established RL techniques. However, we identify the specific challenges of adapting RL techniques to large-scale flow-matching VLAs and developing a complementary three-stage training pipeline of reinforced fine-tuning for cooperative multi-agent VLAs.
Specifically, Initialization-Aware Data Collection addresses the difficulty of effective exploration in challenging cooperative tasks with sparse rewards by selectively incorporating human demonstrations to bootstrap policy learning. Offline Credit-Filter tuning incorporates multi-agent credit assignment into advantage-filtered supervised fine-tuning, reinforcing beneficial individual behaviors while avoiding unstable direct RL updates to flow-matching VLAs. Online latent-space RL further improves coordination through stable latent-space exploration with frozen VLA weights. Although freezing the VLA limits direct improvement of its underlying capabilities, the preceding offline stage compensates by strengthening the base policy.
Thus, our contribution is not the invention of three independent RL algorithms, but an effective and empirically validated framework that addresses the complementary challenges of data coverage, individual credit assignment, and stable online optimization in cooperative multi-agent VLAs.
Beyond introducing entirely new algorithms, we believe that systematically identifying practical challenges and establishing effective training recipes can also provide valuable research contributions. For example, MAPPO (Yu et al., NeurIPS 2022) demonstrated the effectiveness of adapting existing PPO methods to cooperative multi-agent settings through extensive empirical analysis, without relying on domain-specific algorithmic innovations. Similarly, our work investigates how established RL techniques can be effectively adapted to cooperative multi-agent VLAs. We believe these empirical findings and the resulting training pipeline represent meaningful contributions beyond the novelty of individual components.
Thanks for the suggestion. We have expanded the Related Work to better acknowledge the foundations of our approach and clarify the differences between single-agent RL, MARL, and cooperative multi-agent VLAs.
We have included the additional discussions in Section 2.
We would like to clarify the role of CHORUS in our work. Our goal is not to propose another multi-agent VLA architecture, but to investigate how reinforcement fine-tuning can further improve cooperative VLAs beyond supervised fine-tuning (SFT). CHORUS serves as a representative SFT-based multi-agent VLA framework with a few useful tricks, providing a common initialization for evaluating our post-training pipeline. Using the same SFT checkpoint also allows us to isolate the improvements introduced by reinforced fine-tuning rather than architectural differences.