acceptodds
Under review as a conference paper at ICLR 2027

Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning

11 views

Abstract

We study reinforcement learning (RL) methods for cooperative multi-agent Vision-Language-Action (VLA) models. This problem is challenging because VLAs are pretrained on large-scale single-agent data and therefore lack the fine-grained coordination skills required for inter-robot collaboration. Supervised fine-tuning (SFT) on multi-robot demonstrations partially bridges this gap, but its performance is bounded by the demonstration data and cannot improve from its own experience. We present a three-stage reinforced fine-tuning (RFT) pipeline for multi-agent VLAs. First, initialization-aware data collection sweeps over initial configurations and invokes human demonstrations only when the pretrained VLA repeatedly fails, yielding robustness to initialization shift with reduced human cost. Second, offline credit-filtered tuning assigns credit to individual agents and fine-tunes on per-agent trajectories with positive advantage rather than on entire joint rollouts. Third, we find existing online RL for VLAs are less effective for hard multi-agent tasks, which we attribute to noisy co-exploration and unstable updates. We instead use online latent-space fine tuning, which freeze the VLA and perform RL in its latent noise space. We evaluate our multi-agent VLA with both and backbones across 11 tasks in RoboTwin, RoboFactory and real-world manipulation with two Franka robots. Our multi-agent VLA improves the average success rate by , , and on RoboTwin, RoboFactory, and real-world tasks, respectively. Code available at https://anonymous.4open.science/r/mavla_rft-2BC0/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language-Action models, multi-agent reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Y4puOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The submission discusses techniques to improve results of cooperative multi-agent vision-language action models. For this a 3-stage fine-tuning pipeline is proposed. It consists of: 1) task-specific data collection if initialization fails; 2) data/reward filtering (policy is only updated with positive advantage); 3) adopting DSRL (prior work).

Critical strengths
  • Multi-agent vision language action models are timely.
  • The paper is generally easy to follow and the proposed techniques are mostly reasonable.
Critical weaknesses
  • I find the proposed techniques to contain little novelty: First, initialization-aware data collection (Sec. 4.1) has been studied in RL before, e.g., in areas such as "Learning from Observations" (LfO) where task-agnostic and task-specific datasets are considered. One can view the proposed human data collection as curation of a task-specific dataset while the remaining data are task-agnostic. Second, creating a high-quality subset of the data (by focusing on "transitions with positive advantage") which is then treated differently has also been considered. These techniques may differ from what was proposed in Sec. 4.2 but these related works weren't covered in the related work section. The online component (Sec. 4.3) essentially "adopts DSRL" (l.296).
  • The related work section discusses very little of this prior work. Much of the prior work may have been performed in single-agent RL. However, I would have expected a more thorough discussion which outlines how the multi-agent settings differ.
  • "we use CHORUS ..., and implement our ... on its checkpoint" (l.320ff) reads to me like the proposed techniques haven't been assessed on its own. Do they work on top of other baselines? Without such a study, it is hard to assess the generality of the proposed techniques.
  • what are the caveats of focusing "only on transitions with positive advantage" (l.238)? Your favorite AI will give you an answer that should probably be discussed in this submission.

Minor points:

  • the paper's writing is occasionally off, please fix.
Questions for authors

see above

7 comments
AuthorsOpenReview ↗

Official Response of Submission 13918 to Reviewer Y4pu, Part1

We thank Reviewer Y4pu for their positive feedback on the contribution of the paper. Below is our detailed response:

Response

W1: The proposed techniques to contain little novelty.

We acknowledge that individual components of our framework build upon established RL techniques. However, we identify the specific challenges of adapting RL techniques to large-scale flow-matching VLAs and developing a complementary three-stage training pipeline of reinforced fine-tuning for cooperative multi-agent VLAs.

Specifically, Initialization-Aware Data Collection addresses the difficulty of effective exploration in challenging cooperative tasks with sparse rewards by selectively incorporating human demonstrations to bootstrap policy learning. Offline Credit-Filter tuning incorporates multi-agent credit assignment into advantage-filtered supervised fine-tuning, reinforcing beneficial individual behaviors while avoiding unstable direct RL updates to flow-matching VLAs. Online latent-space RL further improves coordination through stable latent-space exploration with frozen VLA weights. Although freezing the VLA limits direct improvement of its underlying capabilities, the preceding offline stage compensates by strengthening the base policy.

Thus, our contribution is not the invention of three independent RL algorithms, but an effective and empirically validated framework that addresses the complementary challenges of data coverage, individual credit assignment, and stable online optimization in cooperative multi-agent VLAs.

Beyond introducing entirely new algorithms, we believe that systematically identifying practical challenges and establishing effective training recipes can also provide valuable research contributions. For example, MAPPO (Yu et al., NeurIPS 2022) demonstrated the effectiveness of adapting existing PPO methods to cooperative multi-agent settings through extensive empirical analysis, without relying on domain-specific algorithmic innovations. Similarly, our work investigates how established RL techniques can be effectively adapted to cooperative multi-agent VLAs. We believe these empirical findings and the resulting training pipeline represent meaningful contributions beyond the novelty of individual components.

W2: More detailed discussions on the related work related to the prior works.

Thanks for the suggestion. We have expanded the Related Work to better acknowledge the foundations of our approach and clarify the differences between single-agent RL, MARL, and cooperative multi-agent VLAs.

We have included the additional discussions in Section 2.

W3: Other baselines besides CHORUS.

We would like to clarify the role of CHORUS in our work. Our goal is not to propose another multi-agent VLA architecture, but to investigate how reinforcement fine-tuning can further improve cooperative VLAs beyond supervised fine-tuning (SFT). CHORUS serves as a representative SFT-based multi-agent VLA framework with a few useful tricks, providing a common initialization for evaluating our post-training pipeline. Using the same SFT checkpoint also allows us to isolate the improvements introduced by reinforced fine-tuning rather than architectural differences.

AuthorsOpenReview ↗

Official Response of Submission 13918 to Reviewer Y4pu, Part2

W4: Caveats of focusing "only on transitions with positive advantage"

We focus on positive-advantage transitions primarily for training efficiency and stability. For flow-matching VLAs, explicitly penalizing negative-advantage actions is challenging due to the intractable action log-likelihood. Moreover, including negative samples under a fixed training budget may dilute the supervision from beneficial actions. Regarding that beneficial actions can be directly reinforced using the standard flow-matching loss, we focus "only on transitions with positive advantage".

To investigate whether negative-advantage samples provide additional benefits, we implement an advantage-conditioned training baseline following the classifier-free guidance (CFG) formulation in π0.6∗\pi^*_{0.6} (RECAP), which incorporates both positive- and negative-advantage data. As shown in the table below, we can find the comparable performance to our positive-only filtering, but the advantage-conditioned baseline incurs approximately 2x training time cost due to additional conditional training and CFG-related computation.

TaskCredit-filtered tuningAdvantage-Conditioned CFG
Lift Pot4244
Handover Mic5251
Three Robots Stack Cubes6159
Take Photo6966
Average5655
Average Training Cost (H)2045

These findings suggest that, in our setting, incorporating negative-advantage samples through a more complex training objective does not provide sufficient performance gains to justify the additional computational cost. We therefore favor positive-advantage filtering as a simpler and more efficient approach.

AuthorsOpenReview ↗

Invitation to reviewer-author discussion

Dear Reviewer,

We are the authors of Submission 13918: "Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning". We’ve uploaded a point-to-point response to your comments, which we believe addresses all of your concerns.

As the reviewer-author discussion period is going to end, we would greatly appreciate it if you could take a moment to share your thoughts on our rebuttal. Your feedback is valuable to us, and we’d be happy to further clarify any remaining points if needed.

Best regards, Authors of Submission 13918

Reviewer_rKV6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a three-stage reinforced fine-tuning pipeline for cooperative multi-agent VLAs. Specifically, initialization-aware data collection adds expert demonstrations only for initial configurations where the pretrained policy has zero observed successes, offline credit-filtered tuning uses an ordered agent-combined critic to retain per-agent trajectory segments with positive estimated advantage, and online adaptation applies DSRL in latent-noise space while freezing the VLA backbone. The method is evaluated with π0 and π0.5 on eight simulated RoboTwin/RoboFactory tasks and three real-world tasks involving two Franka robots.

Critical strengths
  1. This paper addresses an important gap between single-agent VLA pretraining and low-level coordination between embodied agents.

  2. The evaluation spans different VLA backbones, settings, simulation environments, and real-world manipulation.

Critical weaknesses

(1) The source of the gains is not isolated sufficiently. The main end-to-end baseline is CHORUS, while the proposed pipeline combines targeted demonstrations, positive-advantage filtering, and an existing latent-space RL approach. Stronger controlled alternatives are needed, such as joint-rollout filtering, return/advantage-weighted imitation, random or top-k data filtering, and alternative credit-assignment methods under the same data and compute budget. It remains unclear how much improvement is due to agent-wise credit assignment rather than additional successful data or generic filtering.

(2) The offline credit assignment depends on a fixed sequential order of agents (e.g., left then right, or the dataset order). In cooperative tasks, the telescoping contribution assigned to an agent can change when this order changes, but no order-permutation or order-averaging ablation is reported. In addition, the monotonic-improvement proposition relies on accurate advantages, support coverage, and retained actions remaining non-negative-advantage under the updated preceding-agent policy. These are strong assumptions, empirical critic calibration and sensitivity to the ordering would be important to substantiate the claim.

(3) The main performance plots report point estimates without clear run-to-run uncertainty. The stated 100 simulation evaluations and 50 real-world trials quantify evaluation episodes for a checkpoint, but do not replace independent training seeds or confidence intervals.

(4) The initialization-aware rule estimates zero success using K=5 rollouts in simulation but only K=1 in the real world. A single stochastic failure can therefore trigger human data collection in the physical setting. The paper also does not quantify how many demonstrations or how much expert time the rule actually saves, nor compare against a baseline using the same human-demonstration budget allocated without the success-rate filter.

Questions for authors

See weaknesses

2 comments
AuthorsOpenReview ↗

Official Response of Submission 13918 to Reviewer rKV6, Part 1

Official Response of Submission 13918 to Reviewer rKV6

We thank Reviewer rKV6 for their positive feedback on the contribution and the experiment of the paper. Below is our detailed response:

Response

W1: Isolating the Contributions of Data Collection and Credit Assignment & Ablations on alternative strategy.

To better isolate the contribution of agent-wise credit assignment, we combine the existing ablation in Table 1 with an additional joint-return-based filtering baseline on four challenging tasks using the π0.5\pi_{0.5} backbone.

(1) Offline tuning without additional demonstrations. As shown in Table 1 in the paper, applying credit-filtered tuning to autonomously collected rollouts without additional human demonstrations improves the average success rate from 42.8% to 50.3% (+7.5%), suggesting that offline tuning can benefit from autonomous experience rather than relying solely on additional expert demonstrations.

(2) Agent-wise vs. joint-return filtering. We further replace agent-wise advantage filtering with joint-return-based filtering using the same offline dataset and training procedure. This comparison directly evaluates whether individual credit assignment provides additional benefits over generic trajectory-level filtering.

TaskCHORUSJoint-Return-Based FilteringOurs
Lift Pot243842
Handover Mic435052
Three Robots Stack Cubes495861
Take Photo556669
Average42.853.056.0

The results show that agent-wise filtering outperforms joint-return filtering by 3% on average. Unlike joint-return filtering, which applies a shared selection criterion to all agents in a trajectory, our method identifies beneficial individual actions even when the overall task fails, as well as the negative individual actions when the task successes. These results provide evidence that agent-wise credit assignment contributes to the improvement beyond simply introducing additional successful data or performing generic return-based filtering.

We have added the experiments in Table 2.

W2: The assumptions on post-update credit consistency

To empirically examine the credit-consistency assumption in Proposition 4.1, we conduct an additional experiment on the Lift Pot task with the fixed agent order of (Left, Right). We first select Right-agent actions assigned positive advantages by credit-filtered tuning. We then replace the preceding Left-agent actions with those sampled from its offline-updated policy, while keeping the selected Right-agent actions unchanged. For each resulting action pair, we re-evaluate the Right-agent advantage conditioned on the updated Left-agent action and measure the proportion that remains positive.

We find that 93% of the originally positive-advantage Right-agent actions retain positive advantages under the updated Left-agent behavior. This high consistency rate suggests that the beneficial behaviors identified by credit-filtered tuning are largely preserved despite changes in the preceding agent's policy. Importantly, a consistency rate below 100% does not necessarily imply policy degradation, since performance improvement depends on the expected joint advantage under the updated policy rather than every individual action maintaining a positive advantage.

We have added the experiments in Appendix B.

AuthorsOpenReview ↗

Official Response of Submission 13918 to Reviewer rKV6, Part 2

W2: Sensitivity to the ordering of the agents for credit assignment.

We further investigate the sensitivity of credit-filtered tuning to agent ordering on the four environments used in our ablation study. Specifically, we reverse the default ordering from (Left, Right) to (Right, Left) for two-arm tasks, from (1, 2, 3) to (3, 2, 1) for three-arm tasks, and from (1, 2, 3, 4) to (4, 3, 2, 1) for four-arm tasks. Notably, credit-filtered tuning independently retains the top 5% of samples for each agent according to its estimated advantages. Therefore, changing the ordering does not affect the number of retained samples. We evaluate the downstream policy performance under each ordering.

TaskCHORUSReverse Agent orderingOurs
Lift Pot244142
Handover Mic435452
Three Robots Stack Cubes496161
Take Photo556769
Average42.855.856.0

As shown in Table above, while credit-filtered tuning is not strictly invariant to agent ordering, its performance improvements persist under reversed decision orders, indicating that its effectiveness does not rely on a single predefined ordering.

We have added the experiments in Section 5.2.

W3: Run-to-Run Uncertainty.

To address this, we report the standard deviation across different trainings as the error bar in the main performance plot Figure 2.

W4: Initialization-aware Data Collection rule

(1) Justification for K=1 in the real-world experiments: Our method does not aim to identify initializations with strictly zero true success probability. Instead, the criterion serves as a practical heuristic for identifying configurations where autonomous exploration fails within a limited interaction budget. Increasing K reduces potentially unnecessary human interventions but incurs additional autonomous exploration costs.

(2) Human demonstration cost: We additionally report the number of expert demonstrations triggered by Initialization-aware Data Collection for each task in Appendix C, same as the table below. Compared with collecting one demonstration for every initialization, Initialization-aware Data Collection reduces the demonstration count by 88% in simulation and 43% in real-world experiments.

Taskπ0\pi_0π0.5\pi_{0.5}
RoboTwin
Lift Pot3035
Grab Roller00
Handover Mic1316
Put Bottles Dustbin3633
RoboFactory
Camera Alignment00
Three Robots Stack Cubes47
Take Photo13
Long Pipeline Delivery00
Real World
Handover—29
Lift—29
Handover & Drop—28

(3) Comparison under matched demonstration budgets. We retain the same autonomous rollouts and match the human demonstration budget. Specifically, if Initialization-aware Data Collection requests demonstrations for M empirically zero-success initializations, the random baseline instead collects demonstrations from M randomly sampled initializations. Both methods subsequently undergo the same offline credit-filtered tuning.

The additional results are summarized below. Ours consistently outperforms random demonstration collection, suggesting that allocating expert effort to difficult initializations is more effective than random allocation under the same budget. Similarly, agent-wise advantage filtering outperforms joint-return filtering, highlighting the importance of distinguishing individual agent contributions rather than treating all agents in a joint rollout equally. We will include these ablations in the revised manuscript.

TaskCHORUSRandom DemoOurs
Lift Pot243542
Handover Mic434852
Three Robots Stack Cubes495861
Take Photo556869
Average42.852.356.0

We have added the experiments in Section 5.2.

Reviewer_ptCdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a three-stage reinforced fine-tuning pipeline for cooperative multi-agent VLAs. Starting from pretrained single-agent VLAs, the authors first perform CHORUS-style SFT on multi-agent demonstrations to obtain an initial cooperative policy. The proposed pipeline then consists of three stages: (1) initialization-aware data collection, which identifies challenging configurations and supplements policy rollouts with human demonstrations; (2) offline credit-filtered tuning, which first trains a multi-agent critic to estimate agent-wise advantages, then filters the collected data to retain high-advantage per-agent samples and fine-tunes the VLA using a standard flow-matching loss; and (3) online latent-space RL, which freezes the VLA and optimizes a latent noise policy using DSRL. Experiments on RoboTwin, RoboFactory, and real-world robotic tasks demonstrate improved multi-agent cooperation.

Critical strengths
  1. The paper addresses an important and practical problem of improving multi-agent cooperation in pretrained VLAs. The proposed three-stage training pipeline provides a systematic approach to improving data coverage, credit assignment, and online exploration.

  2. The experiments cover two VLA backbones, multiple simulation benchmarks, and real-world robotic tasks, demonstrating consistent performance improvements. The ablation studies also provide useful insights into the effectiveness of different training stages.

Critical weaknesses
  1. The comparison with CHORUS may be affected by differences in observation inputs. The proposed method introduces an additional shared third-person RGB observation, but it is unclear how this input is integrated into the pretrained VLA and whether all baselines use the same observation setting.

  2. The effectiveness of the proposed data collection and credit-filtering strategies is not sufficiently demonstrated. The data used in the "w/o IA" baseline and the additional human demonstration budget are not clearly reported. Comparisons with simpler data collection and filtering strategies under matched data budgets are also missing.

  3. The experiments do not fully establish the necessity of the three-stage training pipeline. In particular, the authors do not evaluate applying DSRL directly to the CHORUS checkpoint without offline tuning. Furthermore, the real-world experiments use an offline variant of DSRL, but its implementation and differences from the online version are not sufficiently explained.

  4. The presentation could be improved, as several important details of the training pipeline and experimental setup are not clearly explained in the main text. Some key information is only provided in the appendix, making it difficult to fully understand the methodology and experimental protocol.

Questions for authors
  1. Could the authors provide more details about the initial CHORUS-style SFT? In particular, how is the additional shared third-person RGB observation integrated into the pretrained VLAs? Is this view used from the initial SFT stage, and do all baselines use the same observation inputs? The original CHORUS paper also highlights distribution shifts caused by changes in camera input semantics. Have the authors investigated whether introducing the additional view causes similar issues?

  2. How reliable is the zero-success initialization criterion with only K=5 rollouts in simulation and K=1 in real-world experiments? How many additional human demonstrations are collected? Also, could the authors clarify how the data in the "w/o IA" baseline is collected and processed?

  3. Could the authors provide stronger comparisons to validate the proposed data collection and filtering strategies? How does initialization-aware collection compare with random data collection under the same human demonstration budget? Additionally, Appendix C.2 states that only approximately 5% of the original data is retained after advantage filtering, while all human demonstration frames are included. Given this aggressive filtering, how does agent-wise advantage filtering compare with simpler success-based or joint-return-based filtering under matched data budgets?

  4. How sensitive is the proposed credit-filtered tuning to agent ordering? Could the authors compare different orderings in terms of the retained training samples and downstream policy performance?

  5. Stage 2 uses advantage estimates for data filtering, followed by standard supervised flow-matching fine-tuning. We suggest renaming this stage to "credit-filtered SFT" rather than "offline RL" to better reflect the actual policy optimization procedure.

  6. Could the authors include a CHORUS + Online RL (DSRL) baseline that skips the offline tuning stage, under the same online interaction budget? If DSRL fails to improve the original CHORUS checkpoint but succeeds after offline tuning, this would provide stronger evidence that offline tuning is important for bootstrapping effective multi-agent online RL.

  7. Could the authors clarify the offline DSRL algorithm used in real-world experiments and how it differs from the online version used in simulation?

4 comments
AuthorsOpenReview ↗

Official Response of Submission 13918 to Reviewer ptCd, Part 1

We thank Reviewer ptCd for their positive feedback on the contribution and the experiment of the paper. Below is our detailed response:

Response

W1 & Q1: The details about the CHORUS-Style SFT and Camera Distribution Shift introduced by the shared third-person RGB observation.

We follow CHORUS in performing initial SFT on multi-robot demonstrations. Each VLA receives its own wrist-camera RGB observation, proprioceptive state, a shared third-person RGB observation, and the language instruction, while predicting actions for its corresponding robot. The shared view is introduced from the initial CHORUS-style SFT stage and retained throughout offline and online RL. All compared methods use the same observation inputs.

Regarding distribution shift, π\pi-series VLAs are also pretrained on bimanual manipulation data with third-person and wrist-camera observations. We map the shared third-person view to the same camera slot used for third-person observations in the bimanual pretraining data, preserving the pretrained camera-slot semantics rather than repurposing slots for different camera roles, nor concatenating observations and predicting joint actions. This design avoids the specific camera-role mismatch identified in CHORUS.

We have added the details in Appendix C.1.

W2 & Q2: Reliability of Initialization-aware Data Collection and human demonstration budget

Reliability of the zero-success criterion. Our criterion does not aim to identify initializations with strictly zero true success probability. Instead, it serves as a practical heuristic to identify configurations where successful trajectories are difficult to obtain within a limited exploration budget. Increasing K may reduce unnecessary human interventions, but also increases the cost of autonomous rollouts.

Additional human demonstrations. The number of additional demonstrations depends on the pretrained policy's performance across initializations: more challenging tasks generally require more human intervention. We report the number of additional demonstrations in Appendix C.2, same as below:

Taskπ0\pi_0π0.5\pi_{0.5}
RoboTwin
Lift Pot3035
Grab Roller00
Handover Mic1316
Put Bottles Dustbin3633
RoboFactory
Camera Alignment00
Three Robots Stack Cubes47
Take Photo13
Long Pipeline Delivery00
Real World
Handover—29
Lift—29
Handover & Drop—28

Compared with collecting one demonstration for every initialization, Initialization-aware Data Collection reduces the demonstration count by 88% in simulation and 43% in real-world experiments.

Clarification of w/o IA. The w/o IA baseline follows the same initialization sampling and autonomous rollout procedure as Initialization-aware Data Collection, but does not introduce human demonstrations for empirically zero-success initializations. We admit that 'w/o IA' is an imprecise statement and have renamed it to 'w/o HD' in the revised manuscript.

AuthorsOpenReview ↗

Official Response of Submission 13918 to Reviewer ptCd, Part 2

W2 & Q3: The ablation experiments about the random additional human demonstration under the same human budget and joint-return-based filtering strategy.

We conducted two additional ablations on the four challenging tasks used in Tables 1 and 2, with the π0.5\pi_{0.5} backbone and the same initial CHORUS checkpoint.

(1) Random additional demonstration under the same human budget. We retain the same autonomous rollouts and match the human demonstration budget. Specifically, if IA requests demonstrations for M empirically zero-success initializations, the random baseline instead collects demonstrations from M randomly sampled initializations. Both methods subsequently undergo the same offline credit-filtered tuning.

(2) Joint-return-based filtering. We replace agent-wise advantage filtering with a simpler joint-return-based filtering strategy, while keeping the remaining training procedure unchanged.

The additional results are summarized below. Ours consistently outperforms random demonstration collection, suggesting that allocating expert effort to difficult initializations is more effective than random allocation under the same budget. Similarly, agent-wise advantage filtering outperforms joint-return filtering, highlighting the importance of distinguishing individual agent contributions rather than treating all agents in a joint rollout equally. We will include these ablations in the revised manuscript.

TaskCHORUSRandom DemoJoint-Return-Based FilteringOurs
Lift Pot24353842
Handover Mic43485052
Three Robots Stack Cubes49585861
Take Photo55686669
Average42.852.35356.0

We have added the experiments in Table 2.

Q4: How sensitive is the proposed credit-filtered tuning to agent ordering?

We further investigate the sensitivity of credit-filtered tuning to agent ordering on the four environments used in our ablation study. Specifically, we reverse the default ordering from (Left, Right) to (Right, Left) for two-arm tasks, from (1, 2, 3) to (3, 2, 1) for three-arm tasks, and from (1, 2, 3, 4) to (4, 3, 2, 1) for four-arm tasks. Notably, credit-filtered tuning independently retains the top 5% of samples for each agent according to its estimated advantages. Therefore, changing the ordering does not affect the number of retained samples. We evaluate the downstream policy performance under each ordering.

TaskCHORUSReverse Agent orderingOurs
Lift Pot244142
Handover Mic435452
Three Robots Stack Cubes496161
Take Photo556769
Average42.855.856.0

As shown in Table above, estimating agent advantage with reversed agent orders bring similar result, showing our method is relatively insensitive to agent ordering.

We have added the experiments in Table 2.

Q5: Rename 'offline RL' to 'credit-filtered SFT'

Thanks for the suggestion. We agree that our offline policy update employs a supervised flow-matching objective. However, we retain the term offline RL to describe the overall learning paradigm, rather than the form of the final policy loss. Specifically, we learns a reward-based critic from offline interaction trajectories, estimates agent-wise advantages through multi-agent credit assignment, and uses these advantages to guide policy improvement. This is closely related to regression-based offline RL methods.

AuthorsOpenReview ↗

Official Response of Submission 13918 to Reviewer ptCd, Part 3

W3 & Q6: CHORUS + Online RL (DSRL) baseline.

We additionally evaluate CHORUS + DSRL under the same online interaction budget across the four ablation tasks. As shown in the table below, directly applying DSRL yields modest improvements but remains less effective than our full pipeline.

TaskCHORUS+Online RL+Offline RL+Offline RL+Online RL
Lift Pot24284250
Handover Mic43445273
Three Robots Stack Cubes49516165
Take Photo55626988
Average42.846.256.069.0

This reflects the complementary roles of offline and online tuning. Specifically, offline credit-filtered tuning directly updates the VLA parameters, leveraging agent-wise credit assignment to strengthen collaborative behaviors and establish a capable policy initialization. In contrast, online DSRL freezes the VLA backbone and optimizes only the latent noise policy to maintain training stability. While this design enables stable online exploration, it also limits the extent to which DSRL can improve an underperforming backbone, particularly on challenging initializations where successful behaviors are rarely observed. Consequently, DSRL alone provides limited gains from the original CHORUS checkpoint, whereas offline tuning first establishes a stronger behavioral foundation, allowing subsequent online RL to refine coordination through trial-and-error interactions. These results highlight the complementary roles of the two stages: offline tuning improves the underlying policy capabilities, while online RL performs stable, interaction-driven refinement.

We have added the experiments in Table 1.

W3 & Q7: The clarification on the offline DSRL in real-world experiments.

Both the real-world and simulation experiments employ the same DSRL formulation, using SAC to optimize the latent noise policy while keeping the VLA backbone frozen. The optimization objective remains identical in both settings.

In simulation, DSRL iteratively alternates between collecting new trajectories using the updated policy and performing SAC updates, allowing the training data distribution to evolve with policy improvements. In real-world experiments, we first collect a fixed dataset of 50 trajectories per task, followed by off-policy SAC training for one iteration only. We have corrected the name to DSRL with one iteration to avoid misunderstanding.

We have clarified these implementation details in Appendix C.2.

AuthorsOpenReview ↗

Invitation to reviewer-author discussion

Dear Reviewer,

We are the authors of Submission 13918: "Cooperative Multi-Agent Vision-Language-Action Models via Reinforced Fine Tuning". We’ve uploaded a point-to-point response to your comments, which we believe addresses all of your concerns.

As the reviewer-author discussion period is going to end, we would greatly appreciate it if you could take a moment to share your thoughts on our rebuttal. Your feedback is valuable to us, and we’d be happy to further clarify any remaining points if needed.

Best regards, Authors of Submission 13918

Comments on the paper

2 comments

Dear reviewers,

Thanks for serving as a reviewer. As the discussion period comes to a close and the authors have submitted their rebuttals, I kindly ask you to take a moment to review them and provide any final comments.

If you have already updated your comments, please disregard this message.

Thank you once again for your dedication to the OpenReview process.

Best,

Area Chair

AuthorsOpenReview ↗

Official Response of Submission 13918 to Area Chair

Dear Area Chair

We sincerely thank the reviewers for their constructive feedback. Our work investigates reinforcement fine-tuning for cooperative multi-agent VLAs, proposing a three-stage pipeline encompassing initialization-aware data collection, offline credit-filtered tuning, and online RL. We demonstrate consistent improvements across 11 simulation and real-world tasks, using both π0\pi_0 and π0.5\pi_{0.5} backbones.

We thank all reviewers for their constructive feedback and discussions. We have carefully addressed all concerns raised by the reviewers. Below is a summary of our responses:

Reviewer ptCd:

We clarified that the shared third-person RGB observation is mapped to the corresponding third-person camera slot in the pretrained bimanual data, preserving camera-slot semantics. We also explained that the zero-success criterion is a practical heuristic for identifying challenging initializations under a limited exploration budget, rather than a guarantee of zero success probability.

To address the experimental concerns, we added matched-budget random demonstrations, joint-return-based filtering, reversed agent ordering, and CHORUS + online RL comparisons on four tasks (Section 5.2, Table 2), alongside human demonstration counts (Appendix C.2). We also clarified that the term "the offline DSRL algorithm" in the original paper is ambiguous, and it is actually an one-iteration DSRL for a proof-of-concept result with same optimazation objectives to the simulation experiments.

Reviewer rKV6:

We isolated the contribution of credit-filtered tuning without additional demonstrations (Section 5.2, Table 1) and added controlled ablations for matched-budget random demonstrations, joint-return-based filtering, reversed agent ordering (Section 5.2, Table 2).

We additionally examined post-update credit-sign consistency, providing empirical evidence supporting the credit-preservation assumption, and explained that the zero-success criterion is a practical heuristic for identifying challenging initializations under a limited exploration budget, rather than a guarantee of zero success probability.

Reviewer Y4pu:

We clarified that our contribution lies in identifying and addressing the challenges of RL post-training for cooperative flow-matching VLAs, rather than claiming novelty for every individual component. We substantially expanded the Related Work to acknowledge prior research in data collection, advantage-based policy optimization, and multi-agent credit assignment.

We also clarified that CHORUS can be seen as a general pipeline towards multi-agent VLA through SFT, rather than a method-centric work. We also examines the generality of our pipelines on different backbones: π0\pi_0 and π0.5\pi_{0.5}. Finally, we added an advantage-conditioned CFG baseline, demonstrating comparable performance to positive-only filtering but at approximately twice the training cost.

Sincerely, Authors of Paper 13918

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.