acceptodds
Under review as a conference paper at ICLR 2027

MMA: A Simple and Flexible Multi-Model Framework for Action

Abstract

Vision-language models (VLMs) and vision-language-action models (VLAs) provide diverse pretrained representations for robot action learning, shaped by different data, architectures, and objectives. Combining their strengths is challenging because their interfaces differ, while running multiple models increases inference cost. We introduce MMA, a framework that jointly learns a fused policy and single-source branches that can be deployed independently. Learnable action queries and branch-local adaptation connect each source to a shared action interface and action head. An Action-Centric Mixture-of-Transformers (MoT) fuses branch representations, and a stop-gradient fused-query teacher transfers information to the branches through query alignment and relational matching. The resulting policies support either full-fusion or single-branch inference. On LIBERO, RoboTwin 2.0, and real-world manipulation tasks, the fused policy achieves average success rates of 99.2%, 93.1%, and 93.8%, respectively, with absolute success-rate gains of 1.7%, 5.5%, and 15.0% over the strongest matched single-source baselines. All three MMA branches also improve over their matched baselines across these evaluations. Each branch runs only its own source and the shared action head, providing a lower-cost deployment choice alongside fusion. MMA reuses pretrained components without access to their original pretraining data or additional large-scale pretraining. MMA offers a reference point for future work on combining pretrained VLMs and VLAs while balancing policy performance and deployment cost.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Multi-Model Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WqJ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MMA, a framework for combining heterogeneous pretrained VLM and VLA sources for robot action learning. It uses learnable action queries and branch-specific adapters to create a unified interface, an Action-Centric Mixture-of-Transformers to fuse representations, and query distillation to transfer fused knowledge to individual branches. A shared flow-matching action head supports both full-fusion and independent single-branch deployment. Experiments on LIBERO, RoboTwin 2.0, and real-world manipulation tasks show improvements over matched independently trained baselines for both the fused policy and individual branches.

Critical strengths

The paper studies a practical problem. It combines pretrained models during training and improves individual branches while allowing them to run independently at inference.

Critical weaknesses

1.The work is primarily an engineering contribution. The paper combines existing techniques for multi-model fusion, action queries, and knowledge distillation into a unified framework. Its contribution mainly lies in system combination and implementation, with relatively limited methodological novelty.

2.Experimental details are insufficiently reported. The paper does not specify some loss weights, the distillation temperature, or the detailed action-head configuration. The real-world experiments and inference latency measurements also lack detailed settings. Providing this information would improve reproducibility and help readers interpret the results.

3.Lack of evaluation with different numbers of models. The paper uses only three models without sufficiently explaining this choice. I suggest adding experiments with two and four models to test whether the framework works across different model counts and to measure the additional benefit of each added model.

Questions for authors

Could the authors respond to the concerns raised in the weaknesses section?

4 comments
AuthorsOpenReview ↗

Response to Reviewer WqJ1 (Part 1/3)

We thank all three reviewers for your thoughtful feedback and constructive suggestions. We appreciate the reviewers’ recognition of MMA’s practicality and deployment flexibility (EeFY, 57mt, WqJ1), flexible framework and comprehensive evaluation (EeFY), and consistent performance gains (57mt). At the same time, we acknowledge the shared concerns regarding different model combinations (EeFY, 57mt, WqJ1), individual contributions and statistical validation (EeFY, 57mt), and reproducibility (57mt, WqJ1). We provide point-by-point responses to your concerns below.

Q1: Please clarify the methodological contribution beyond combining existing action queries, multi-model fusion, and knowledge distillation.

MMA combines a unified action interface, shared action head, action-centric fusion, and fusion-to-branch distillation to jointly learn from heterogeneous pretrained VLM/VLA sources while preserving efficient single-source inference. Average branch gains over independent baselines are (LIBERO: +1.8%, RoboTwin: +6.4%), and fusion achieves (LIBERO: 99.2%, RoboTwin: 93.1%), averaging branch gains across three sources and RoboTwin scores across Clean and Random. This simple design can inform practical robotic systems. Below, we extend the original Table 4 ablations to examine each component’s contribution, and we will clarify the design’s relationship to prior work in the revised manuscript.

To clarify the contribution of each component, we organize MMA into three components and evaluate their effects through controlled ablations.

MMA = Multi-Source Joint Training + Action-Centric MoT Fusion + Fusion-to-Branch Query Distillation.

  • Multi-Source Joint Training: Jointly optimizes all individual branches and a mean-query fusion policy using a shared action head and action supervision, without query distillation.
  • Action-Centric MoT Fusion: Replaces mean-query fusion with Action-Centric MoT and Pool for learned cross-source fusion.
  • Fusion-to-Branch Query Distillation: Uses detached fused queries as teacher targets to guide individual branches through cosine alignment and relational matching.

Table. Component ablations of MMA (success rate, %). Parentheses show decreases from the corresponding Full MMA branch or fusion policy (%). Avg. drop is the arithmetic mean of the decreases across LIBERO, Clean, and Random.

SettingPolicy branchLIBERO Avg.CleanRandomAvg. drop
Full MMAQwen98.590.889.5-
Cosmos98.292.591.8-
ABot98.992.292.0-
Fusion99.293.392.9-
Joint trainingQwen96.3 (-2.2)84.2 (-6.6)82.3 (-7.2)-5.3
Cosmos96.3 (-1.9)84.9 (-7.6)84.2 (-7.6)-5.7
ABot97.6 (-1.3)87.8 (-4.4)87.3 (-4.7)-3.5
Fusion97.7 (-1.5)88.2 (-5.1)87.4 (-5.5)-4.0
Joint training + MoT fusionQwen97.4 (-1.1)86.8 (-4.0)85.2 (-4.3)-3.1
Cosmos97.3 (-0.9)87.6 (-4.9)86.9 (-4.9)-3.6
ABot98.3 (-0.6)88.7 (-3.5)87.9 (-4.1)-2.7
Fusion98.4 (-0.8)90.1 (-3.2)89.3 (-3.6)-2.5
Joint training + Query distillationQwen97.2 (-1.3)86.1 (-4.7)84.3 (-5.2)-3.7
Cosmos97.2 (-1.0)86.5 (-6.0)86.1 (-5.7)-4.2
ABot98.2 (-0.7)88.6 (-3.6)87.8 (-4.2)-2.8
Fusion98.3 (-0.9)88.7 (-4.6)87.9 (-5.0)-3.5
Independent baselinesQwen96.1 (-2.4)84.4 (-6.4)82.0 (-7.5)-5.4
Cosmos96.5 (-1.7)84.6 (-7.9)84.4 (-7.4)-5.7
ABot97.5 (-1.4)88.0 (-4.2)87.2 (-4.8)-3.5

(1) Role of Joint Training. Joint training alone achieves average branch success rates of (LIBERO: 96.7%, RoboTwin: 85.1%), close to independent training (LIBERO: 96.7%, RoboTwin: 85.1%). Mean-query fusion achieves (LIBERO: 97.7%, RoboTwin: 87.8%), also close to the strongest independent baseline (LIBERO: 97.5%, RoboTwin: 87.6%). Joint training provides the foundation for MoT fusion and query distillation while preserving efficient single-branch inference.

(2) Contributions of Fusion and Distillation. Both components are essential to achieving full MMA performance. Replacing MoT with mean-query fusion reduces average branch performance by (LIBERO: -1.0%, RoboTwin: -4.9%) and fusion performance by (LIBERO: -0.9%, RoboTwin: -4.8%). Removing query distillation reduces average branch performance by (LIBERO: -0.9%, RoboTwin: -4.3%) and fusion performance by (LIBERO: -0.8%, RoboTwin: -3.4%). These results support combining learned cross-source fusion with fusion-to-branch knowledge transfer.

AuthorsOpenReview ↗

Response to Reviewer WqJ1 (Part 2/3)

Q2: Please provide the loss weights, distillation temperature, action-head configuration, real-world experimental settings, and inference-latency measurement protocol.

Training Settings. Baselines, MMA, and ablations use matched settings within each benchmark, except for the ablated components:

  • LIBERO: 50,000 steps, 5,000 warmup steps, cosine decay to 1e-6, 7-dimensional actions, chunk length 8, and no proprioceptive inputs.
  • RoboTwin 2.0: 500,000 steps, 14-dimensional actions, chunk length 50, and 224 × 224 images.

Both use AdamW with backbone/module learning rates 1e-5/1e-4, β = (0.9, 0.95), ε and weight decay 1e-8, gradient clipping 1.0, BF16, and DeepSpeed ZeRO-2. The effective batch size is 64 (8 GPUs × 8 samples; accumulation factor 1).

Loss and Sampling. We set λF=1.0\lambda_F=1.0 (fusion loss), λD=1.0\lambda_D=1.0 (distillation), αc=1.0\alpha_c=1.0 (cosine alignment), αr=1.0\alpha_r=1.0 (relational matching), and τd=0.1\tau_d=0.1 (temperature). Flow matching uses R=8R=8 noise/time samples per demonstration, Beta parameters (1.5, 1.0), and 4 Euler steps at inference; RR leaves the data batch size unchanged. Distillation and relational ablations set λD=0\lambda_D=0 and αr=0\alpha_r=0, respectively.

Model Configuration. Each source has Q=64Q=64 action queries, shared dimension D=2048D=2048, and a one-layer, eight-head adapter. MoT uses two layers and eight heads. The 16-layer DiT-B action head uses 12 heads of dimension 64, conditioning dimension 2048, output and decoder hidden dimensions 1024, and dropout 0.2. As specified in Appendix A.1.1, each baseline matches its MMA branch in backbone, query interface, adapter, and action-head architecture, so branch gains do not require greater deployment capacity.

Real-World Training and Evaluation. RDT-1B, X-VLA, VLA-Adapter, OpenVLA-OFT, π0\pi_0, and π0.5\pi_{0.5} are initialized from official weights and fine-tuned with official default settings for 10,000 steps on 200 trajectories across four tasks. MMA and same-source baselines use the same data and steps with our LIBERO/RoboTwin training settings. Qwen3.5 and Cosmos-Reason2 use official VLM weights; ABot-M0 uses its pretrained VLM component. MMA shares an action head through the unified interface, while independent baselines train separate heads.

On a Franka robot arm, we evaluate Placement (three blocks into a box), Replacement (red block replaced by blue), Sorting (red, green, blue from left to right), and Stacking (three-block tower), with 20 trials per task. Task descriptions and demonstration images appear in Appendix A.1.4 and Figure A.1.

Inference Costs. Table 5 reports parameters, peak GPU memory, and latency; Figure 6 shows the timing breakdown, with the protocol in Appendix A.1.3. We use an NVIDIA GeForce RTX 5090 and 50 warmup iterations, keeping precision, inputs, batch size, action horizon, and inference sampling fixed across policies; model and input settings follow training. Median latency spans in-memory PIL images to normalized CPU actions, excluding video decoding, input resizing, RPC, and environment execution. Single-branch inference runs one source and the shared head; fusion runs all sources, fusion, and the shared head.

Supplementary Updates. We will consolidate configurations, schedules, preprocessing, benchmark protocols, real-world dataset details and images, and hardware/timing settings in the revised supplement. For Tables 1, 2, and 4, we will report seed values/counts, independent runs, test trials, checkpoint selection, aggregation, available repeated-run means/standard deviations, and confidence intervals with their calculation methods. We will distinguish training variability from fixed-checkpoint evaluation uncertainty, provide per-task real-world success counts, and separately assess branch-versus-baseline and fusion-versus-strongest-branch comparisons.

AuthorsOpenReview ↗

Response to Reviewer WqJ1 (Part 3/3)

Q3: Why were three models selected? Please evaluate two- and four-model variants and quantify the benefit of adding each source.

We select three source models of different sizes for our experiments, including: (1) Qwen3.5 (0.8B, a pretrained VLM), (2) Cosmos-Reason2 (2B, a VLM trained for physical reasoning), and (3) ABot-M0 (4B, a pretrained VLA). These models span distinct pretraining regimes and are compatible with the unified action interface, enabling a systematic study of multi-source collaboration for robot action learning.

We agree with the reviewers that broader evaluation across model combinations and source counts is needed. To enable a more comprehensive evaluation and comparison, we introduce a fourth source, GR00T-N1.7 (2B, a generalist VLA). As with ABot-M0, we retain only its vision-language component and initialize its action head from MMA’s shared action head.

Source Combinations.

To reduce training time and computational cost, we conduct the additional RoboTwin 2.0 experiments on a subset of 10 tasks rather than the full set of 50 tasks. Both training and evaluation use the following tasks: adjust_bottle, beat_block_hammer, blocks_ranking_rgb, click_alarmclock, dump_bin_bigbin, move_can_pot, open_microwave, stack_blocks_two, scan_object, and turn_switch.

Table. Independent baselines and MMA source combinations (success rate, %).

Source combinationPolicy branchLIBERO Avg.Clean (10 tasks)Random (10 tasks)
Independent baselinesQwen96.184.982.4
Cosmos96.585.684.5
ABot97.588.287.6
GR00T97.990.189.8
MMA (Qwen + Cosmos)Qwen97.4 (+1.3)88.6 (+3.7)86.8 (+4.4)
Cosmos97.6 (+1.1)89.5 (+3.9)88.2 (+3.7)
Fusion98.190.889.4
MMA (Qwen + ABot)Qwen97.8 (+1.7)89.7 (+4.8)88.1 (+5.7)
ABot98.2 (+0.7)90.6 (+2.4)90.1 (+2.5)
Fusion98.692.091.2
MMA (Cosmos + ABot)Cosmos97.7 (+1.2)90.1 (+4.5)89.0 (+4.5)
ABot98.3 (+0.8)91.0 (+2.8)90.6 (+3.0)
Fusion98.792.591.8
MMA (Qwen + Cosmos + ABot)Qwen98.5 (+2.4)91.4 (+6.5)90.0 (+7.6)
Cosmos98.2 (+1.7)92.7 (+7.1)91.9 (+7.4)
ABot98.9 (+1.4)92.3 (+4.1)92.2 (+4.6)
Fusion99.294.193.4
MMA (Qwen + Cosmos + ABot + GR00T)Qwen98.9 (+2.8)92.2 (+7.3)90.3 (+7.9)
Cosmos98.8 (+2.3)93.0 (+7.4)92.3 (+7.8)
ABot98.9 (+1.4)93.2 (+5.0)92.6 (+5.0)
GR00T99.1 (+1.2)93.8 (+3.7)93.5 (+3.7)
Fusion99.494.994.1

(1) Benefits of Two-Model Collaboration. Average branch gains over independent baselines are Qwen (LIBERO: +1.5%, RoboTwin: +4.7%), Cosmos (LIBERO: +1.2%, RoboTwin: +4.2%), and ABot (LIBERO: +0.8%, RoboTwin: +2.7%). All three sources benefit, with larger gains for smaller, lower-performing models in these experiments.

(2) Comparison of Model Counts.

MetricNumber of modelsLIBERORoboTwin
Average branch gain2+1.1+3.8
3+1.8+6.2
4+2.2+6.7
Fusion gain1 -> 2+1.8+5.8
2 -> 3+0.7+2.5
3 -> 4+0.2+0.8

From two to four models, the three original branches gain an additional (LIBERO: +1.0%, RoboTwin: +2.9%) on average. Four-model fusion improves over the mean independent baseline by (LIBERO: +2.7%, RoboTwin: +9.0%), while successive model additions yield diminishing gains. Branch comparisons and the baseline reference use Qwen, Cosmos, and ABot; two-model results average all pairwise combinations, and RoboTwin averages Clean and Randomized.

(3) Contribution of Each Source. Removing Qwen, Cosmos, or ABot changes fusion performance by (LIBERO: -0.5%, RoboTwin: -1.6%), (LIBERO: -0.6%, RoboTwin: -2.2%), and (LIBERO: -1.1%, RoboTwin: -3.7%), respectively. Each source contributes, with the largest drop after removing ABot, supporting the value of its vision-language component.

Thank you for the detailed responses and the time spent conducting these additional experiments. The expanded component ablations, evaluations with different model combinations and source counts, and clarifications of the experimental settings have substantially addressed my main concerns and strengthened the empirical support for the framework. I appreciate these efforts and will take the additional results and clarifications into consideration when determining my final score.

Reviewer_57mtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how to combine heterogeneous pretrained VLM/VLA backbones in one robot policy without running all of them at deployment. MMA gives each source (Qwen3.5-0.8B, Cosmos-Reason2-2B, ABot-M0) learnable action queries and a small adapter feeding a shared flow-matching action head, fuses the queries with a two-layer Action-Centric MoT, and distills the detached fused queries into each branch. The authors position it against encoder aggregation and weight merging as jointly training a fused policy and separately deployable branches. Validation covers LIBERO, RoboTwin 2.0 and four real-robot tasks against same-source baselines trained alone.

Critical strengths

Every MMA branch beats its same-source baseline in all three settings (e.g., Qwen 83.2 to 90.2 on RoboTwin, Table 2), and fusion is highest in each table. This matters because the central claim is that a cheap branch (38 to 85 ms versus 136.5 ms for fusion, Table 5) gains from joint training. Table 4 supports the mechanism: removing distillation lowers every RoboTwin branch by 3.8 to 4.9 points, and mean fusion lowers fusion from 93.1 to 88.3. This is the main reason to lean toward acceptance.

Critical weaknesses

Appendix A.1.1 does not state explicitly whether the independent baselines used MMA's schedule (50k/500k updates, batch 64), and MMA branches also receive fusion-loss gradients (Table A.3). Even without distillation, branches exceed baselines by 0.7 to 2.8 RoboTwin points, and only one source combination is tested, so gains from added supervision are not separated from gains from other sources. This limits how much of the branch gain can be credited to multi-model knowledge.

Loss weights (lambda_F, lambda_D, alpha_c, alpha_r, tau_d, R), seeds, variance and simulation episode counts are not reported. LIBERO margins are 0.3 to 2.4 points near ceiling, and on the real robot fusion's 75/80 versus the best branch's 68/80 is not significant alone (Fisher exact p about 0.12, reviewer calculation). This weakens fusion-over-branch claims more than branch-over-baseline claims.

Questions for authors

Request to verify an existing result: did the same-source baselines use the same updates, batch size and learning rates as MMA, and how many episodes and seeds underlie Tables 1, 2 and 4?

Request for a new experiment: can you report a two-source variant, or a single-source run keeping the fusion loss and distillation, to show the gains depend on combining different sources?

Clarification: please give lambda_F, lambda_D, alpha_c, alpha_r, tau_d and R, and how the external real-world baselines in Table 3 were trained.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 57mt (Part 1/4)

We thank all three reviewers for your thoughtful feedback and constructive suggestions. We appreciate the reviewers’ recognition of MMA’s practicality and deployment flexibility (EeFY, 57mt, WqJ1), flexible framework and comprehensive evaluation (EeFY), and consistent performance gains (57mt). At the same time, we acknowledge the shared concerns regarding different model combinations (EeFY, 57mt, WqJ1), individual contributions and statistical validation (EeFY, 57mt), and reproducibility (57mt, WqJ1). We provide point-by-point responses to your concerns below.

Q1: Did the independent baselines use the same steps, batch size, and learning rates as MMA? Please distinguish gains from additional fusion supervision from those due to multi-source learning.

To clarify the contribution of each component, we organize MMA into three components and evaluate their effects through controlled ablations.

MMA = Multi-Source Joint Training + Action-Centric MoT Fusion + Fusion-to-Branch Query Distillation.

  • Multi-Source Joint Training: Jointly optimizes all individual branches and a mean-query fusion policy using a shared action head and action supervision, without query distillation.
  • Action-Centric MoT Fusion: Replaces mean-query fusion with Action-Centric MoT and Pool for learned cross-source fusion.
  • Fusion-to-Branch Query Distillation: Uses detached fused queries as teacher targets to guide individual branches through cosine alignment and relational matching.

Table. Component ablations of MMA (success rate, %). Parentheses show decreases from the corresponding Full MMA branch or fusion policy (%). Avg. drop is the arithmetic mean of the decreases across LIBERO, Clean, and Random.

SettingPolicy branchLIBERO Avg.CleanRandomAvg. drop
Full MMAQwen98.590.889.5-
Cosmos98.292.591.8-
ABot98.992.292.0-
Fusion99.293.392.9-
Joint trainingQwen96.3 (-2.2)84.2 (-6.6)82.3 (-7.2)-5.3
Cosmos96.3 (-1.9)84.9 (-7.6)84.2 (-7.6)-5.7
ABot97.6 (-1.3)87.8 (-4.4)87.3 (-4.7)-3.5
Fusion97.7 (-1.5)88.2 (-5.1)87.4 (-5.5)-4.0
Joint training + MoT fusionQwen97.4 (-1.1)86.8 (-4.0)85.2 (-4.3)-3.1
Cosmos97.3 (-0.9)87.6 (-4.9)86.9 (-4.9)-3.6
ABot98.3 (-0.6)88.7 (-3.5)87.9 (-4.1)-2.7
Fusion98.4 (-0.8)90.1 (-3.2)89.3 (-3.6)-2.5
Joint training + Query distillationQwen97.2 (-1.3)86.1 (-4.7)84.3 (-5.2)-3.7
Cosmos97.2 (-1.0)86.5 (-6.0)86.1 (-5.7)-4.2
ABot98.2 (-0.7)88.6 (-3.6)87.8 (-4.2)-2.8
Fusion98.3 (-0.9)88.7 (-4.6)87.9 (-5.0)-3.5
Independent baselinesQwen96.1 (-2.4)84.4 (-6.4)82.0 (-7.5)-5.4
Cosmos96.5 (-1.7)84.6 (-7.9)84.4 (-7.4)-5.7
ABot97.5 (-1.4)88.0 (-4.2)87.2 (-4.8)-3.5

(1) Role of Joint Training. Joint training alone achieves average branch success rates of (LIBERO: 96.7%, RoboTwin: 85.1%), close to independent training (LIBERO: 96.7%, RoboTwin: 85.1%). Mean-query fusion achieves (LIBERO: 97.7%, RoboTwin: 87.8%), also close to the strongest independent baseline (LIBERO: 97.5%, RoboTwin: 87.6%). Joint training provides the foundation for MoT fusion and query distillation while preserving efficient single-branch inference.

(2) Contributions of Fusion and Distillation. Both components are essential to achieving full MMA performance. Replacing MoT with mean-query fusion reduces average branch performance by (LIBERO: -1.0%, RoboTwin: -4.9%) and fusion performance by (LIBERO: -0.9%, RoboTwin: -4.8%). Removing query distillation reduces average branch performance by (LIBERO: -0.9%, RoboTwin: -4.3%) and fusion performance by (LIBERO: -0.8%, RoboTwin: -3.4%). These results support combining learned cross-source fusion with fusion-to-branch knowledge transfer.

Additional Clarification about Fair Comparison and Model Capacity

Independent baselines, MMA, and ablation variants use the same training settings within each benchmark: 50,000 steps for LIBERO and 500,000 for RoboTwin, AdamW with backbone/module learning rates of 1e-5/1e-4, and an effective batch size of 64 (8 GPUs × 8 samples). LIBERO uses 5,000 warmup steps followed by cosine decay to 1e-6.

Parameter Counts and Inference Costs. Table 5 and Figure 6 report deployment parameters, memory, and latency, with the measurement protocol in Appendix A.1.3. MMA improves individual branches without increasing their deployment model capacity.

AuthorsOpenReview ↗

Response to Reviewer 57mt (Part 2/4)

Q2: Please provide a two-source experiment, or a single-source experiment retaining fusion loss and distillation, to test whether the gains depend on combining different sources.

We select three source models of different sizes for our experiments, including: (1) Qwen3.5 (0.8B, a pretrained VLM), (2) Cosmos-Reason2 (2B, a VLM trained for physical reasoning), and (3) ABot-M0 (4B, a pretrained VLA). These models span distinct pretraining regimes and are compatible with the unified action interface, enabling a systematic study of multi-source collaboration for robot action learning.

We agree with the reviewers that broader evaluation across model combinations and source counts is needed. To enable a more comprehensive evaluation and comparison, we introduce a fourth source, GR00T-N1.7 (2B, a generalist VLA). As with ABot-M0, we retain only its vision-language component and initialize its action head from MMA’s shared action head.

To reduce training time and computational cost, we conduct the additional RoboTwin 2.0 experiments on a subset of 10 tasks rather than the full set of 50 tasks. Both training and evaluation use the following tasks: adjust_bottle, beat_block_hammer, blocks_ranking_rgb, click_alarmclock, dump_bin_bigbin, move_can_pot, open_microwave, stack_blocks_two, scan_object, and turn_switch.

Table. Independent baselines and MMA source combinations (success rate, %).

Source combinationPolicy branchLIBERO Avg.Clean (10 tasks)Random (10 tasks)
Independent baselinesQwen96.184.982.4
Cosmos96.585.684.5
ABot97.588.287.6
GR00T97.990.189.8
MMA (Qwen + Cosmos)Qwen97.4 (+1.3)88.6 (+3.7)86.8 (+4.4)
Cosmos97.6 (+1.1)89.5 (+3.9)88.2 (+3.7)
Fusion98.190.889.4
MMA (Qwen + ABot)Qwen97.8 (+1.7)89.7 (+4.8)88.1 (+5.7)
ABot98.2 (+0.7)90.6 (+2.4)90.1 (+2.5)
Fusion98.692.091.2
MMA (Cosmos + ABot)Cosmos97.7 (+1.2)90.1 (+4.5)89.0 (+4.5)
ABot98.3 (+0.8)91.0 (+2.8)90.6 (+3.0)
Fusion98.792.591.8
MMA (Qwen + Cosmos + ABot)Qwen98.5 (+2.4)91.4 (+6.5)90.0 (+7.6)
Cosmos98.2 (+1.7)92.7 (+7.1)91.9 (+7.4)
ABot98.9 (+1.4)92.3 (+4.1)92.2 (+4.6)
Fusion99.294.193.4
MMA (Qwen + Cosmos + ABot + GR00T)Qwen98.9 (+2.8)92.2 (+7.3)90.3 (+7.9)
Cosmos98.8 (+2.3)93.0 (+7.4)92.3 (+7.8)
ABot98.9 (+1.4)93.2 (+5.0)92.6 (+5.0)
GR00T99.1 (+1.2)93.8 (+3.7)93.5 (+3.7)
Fusion99.494.994.1

(1) Benefits of Two-Model Collaboration. Average branch gains over independent baselines are Qwen (LIBERO: +1.5%, RoboTwin: +4.7%), Cosmos (LIBERO: +1.2%, RoboTwin: +4.2%), and ABot (LIBERO: +0.8%, RoboTwin: +2.7%). All three sources benefit, with larger gains for smaller, lower-performing models in these experiments.

(2) Comparison of Model Counts.

MetricNumber of modelsLIBERORoboTwin
Average branch gain2+1.1+3.8
3+1.8+6.2
4+2.2+6.7
Fusion gain1 -> 2+1.8+5.8
2 -> 3+0.7+2.5
3 -> 4+0.2+0.8

From two to four models, the three original branches gain an additional (LIBERO: +1.0%, RoboTwin: +2.9%) on average. Four-model fusion improves over the mean independent baseline by (LIBERO: +2.7%, RoboTwin: +9.0%), while successive model additions yield diminishing gains. Branch comparisons and the baseline reference use Qwen, Cosmos, and ABot; two-model results average all pairwise combinations, and RoboTwin averages Clean and Randomized.

(3) Contribution of Each Source. Removing Qwen, Cosmos, or ABot changes fusion performance by (LIBERO: -0.5%, RoboTwin: -1.6%), (LIBERO: -0.6%, RoboTwin: -2.2%), and (LIBERO: -1.1%, RoboTwin: -3.7%), respectively. Each source contributes, with the largest drop after removing ABot, supporting the value of its vision-language component.

AuthorsOpenReview ↗

Response to Reviewer 57mt (Part 3/4)

Q3: How many training seeds and test trials underlie Tables 1, 2, and 4? Please address the small LIBERO margins and the uncertainty in the real-world comparison of 75/80 versus 68/80 successes.

We conduct five independent training runs using the same set of five random seeds for LIBERO and RoboTwin. Each run evaluates all MMA branches and the fusion policy, with the number of test trials per task unchanged from the preceding experiments. For RoboTwin 2.0, each policy is evaluated over 100 test trials per task under each condition (Clean and Randomized) in each training run. We report the mean and standard deviation across the five runs.

Table. MMA (Qwen + Cosmos + ABot): success rates across five independent training runs (mean ± standard deviation, %). RoboTwin 2.0 uses the same 10-task subset as the source-combination experiments.

Policy branchLIBERO Avg.Clean (10 tasks)Random (10 tasks)
Qwen98.4 ± 0.291.5 ± 0.589.9 ± 0.5
Cosmos98.3 ± 0.392.6 ± 0.591.9 ± 0.5
ABot98.8 ± 0.292.5 ± 0.492.1 ± 0.5
Fusion99.2 ± 0.294.2 ± 0.593.5 ± 0.6

LIBERO Performance. LIBERO success rates are already near saturation. Across five runs, fusion achieves 99.2 ± 0.2%, versus 98.8 ± 0.2% for the strongest branch, retaining an average advantage of 0.4% with low variability.

Real-World Evaluation. Table 3 includes 1,040 real-robot tests across 13 policies and four tasks, with 20 trials per policy per task, representing substantial experimental effort. We plan to expand both the dataset and evaluation scale for a more comprehensive comparison, but completing additional real-world tests within the rebuttal period is difficult given the time required for physical execution and task resets. Nevertheless, all three MMA branches outperform their corresponding baselines, with an average gain of 9.6%. Fusion achieves 93.8% success, exceeding the strongest same-source baseline by 15.0%. These results provide empirical support for MMA’s effectiveness in the evaluated real-world tasks under a common testing protocol.

Reviewer_EeFYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MMA, a multi-model framework that integrates heterogeneous pretrained VLM and VLA representations for robotic manipulation. The framework introduces a unified action interface, an Action-Centric Mixture-of-Transformers (MoT) for multi-source feature fusion, and a fusion-to-branch knowledge distillation mechanism. MMA supports both fused inference and independent single-branch deployment. Experiments on LIBERO, RoboTwin 2.0, and real-world manipulation tasks demonstrate improvements over the corresponding single-source baselines.

Critical strengths
  1. Flexible framework. MMA provides a unified interface for integrating heterogeneous pretrained models while supporting both multi-model fusion and independent single-branch deployment.

  2. Comprehensive evaluation. The authors conduct experiments on LIBERO, RoboTwin 2.0, and real-world manipulation tasks, demonstrating consistent improvements over matched single-source baselines.

  3. Clear technical presentation. The overall framework and training objectives are well described, and the appendix provides additional analyses of feature representations and knowledge transfer.

Critical weaknesses
  1. Insufficient validation of knowledge complementarity between pretrained VLMs and VLAs. The paper emphasizes the complementary knowledge of pretrained VLMs and VLAs, but only evaluates a fixed combination of Qwen3.5, Cosmos-Reason2, and ABot-M0. Moreover, the framework primarily uses their vision-language components rather than complete VLA action policies. I recommend adding experiments with different model combinations (e.g., Qwen + Cosmos and Qwen + ABot) and leave-one-source-out ablations to verify whether pretrained VLAs provide additional complementary information.

  2. Unclear contributions of joint training, feature fusion, and knowledge distillation. Compared with independently trained baselines, MMA introduces several changes simultaneously, including a shared action head, joint training, feature fusion, and knowledge distillation. The existing ablations do not fully isolate their individual contributions. I suggest adding a jointly trained baseline with a shared action head but without fusion or distillation, as well as comparisons under matched parameter counts or computational budgets.

  3. Insufficient statistical validation of experimental results. Some improvements are relatively small. For example, the fused policy outperforms the strongest individual branch by only 0.3 percentage points on LIBERO. Additionally, each real-world task is evaluated over only 20 trials, where a single successful trial corresponds to 5 percentage points. I recommend reporting results across multiple random seeds or independent training runs, together with confidence intervals, to establish the reliability of these improvements.

  4. Redundant architectural illustrations. Figures 1–5 repeatedly present similar components, including pretrained backbones, action queries, adapters, and the shared action head, making the presentation somewhat repetitive. I suggest merging identical components or simplifying repeated structures so that each figure more clearly highlights its main technical contribution.

Questions for authors
  1. Insufficient validation of knowledge complementarity between pretrained VLMs and VLAs. The paper emphasizes the complementary knowledge of pretrained VLMs and VLAs, but only evaluates a fixed combination of Qwen3.5, Cosmos-Reason2, and ABot-M0. Moreover, the framework primarily uses their vision-language components rather than complete VLA action policies. I recommend adding experiments with different model combinations (e.g., Qwen + Cosmos and Qwen + ABot) and leave-one-source-out ablations to verify whether pretrained VLAs provide additional complementary information.

  2. Unclear contributions of joint training, feature fusion, and knowledge distillation. Compared with independently trained baselines, MMA introduces several changes simultaneously, including a shared action head, joint training, feature fusion, and knowledge distillation. The existing ablations do not fully isolate their individual contributions. I suggest adding a jointly trained baseline with a shared action head but without fusion or distillation, as well as comparisons under matched parameter counts or computational budgets.

  3. Insufficient statistical validation of experimental results. Some improvements are relatively small. For example, the fused policy outperforms the strongest individual branch by only 0.3 percentage points on LIBERO. Additionally, each real-world task is evaluated over only 20 trials, where a single successful trial corresponds to 5 percentage points. I recommend reporting results across multiple random seeds or independent training runs, together with confidence intervals, to establish the reliability of these improvements.

  4. Redundant architectural illustrations. Figures 1–5 repeatedly present similar components, including pretrained backbones, action queries, adapters, and the shared action head, making the presentation somewhat repetitive. I suggest merging identical components or simplifying repeated structures so that each figure more clearly highlights its main technical contribution.

4 comments
AuthorsOpenReview ↗

Response to Reviewer EeFY (Part 1/3)

We thank all three reviewers for your thoughtful feedback and constructive suggestions. We appreciate the reviewers’ recognition of MMA’s practicality and deployment flexibility (EeFY, 57mt, WqJ1), flexible framework and comprehensive evaluation (EeFY), and consistent performance gains (57mt). At the same time, we acknowledge the shared concerns regarding different model combinations (EeFY, 57mt, WqJ1), individual contributions and statistical validation (EeFY, 57mt), and reproducibility (57mt, WqJ1). We provide point-by-point responses to your concerns below.

Q1: Please evaluate different source combinations and leave-one-source-out variants, and clarify whether the VLA source contributes complementary information through its vision-language component.

We select three source models of different sizes for our experiments, including: (1) Qwen3.5 (0.8B, a pretrained VLM), (2) Cosmos-Reason2 (2B, a VLM trained for physical reasoning), and (3) ABot-M0 (4B, a pretrained VLA). These models span distinct pretraining regimes and are compatible with the unified action interface, enabling a systematic study of multi-source collaboration for robot action learning.

We agree with the reviewers that broader evaluation across model combinations and source counts is needed. To enable a more comprehensive evaluation and comparison, we introduce a fourth source, GR00T-N1.7 (2B, a generalist VLA). As with ABot-M0, we retain only its vision-language component and initialize its action head from MMA’s shared action head.

To reduce training time and computational cost, we conduct the additional RoboTwin 2.0 experiments on a subset of 10 tasks rather than the full set of 50 tasks. Both training and evaluation use the following tasks: adjust_bottle, beat_block_hammer, blocks_ranking_rgb, click_alarmclock, dump_bin_bigbin, move_can_pot, open_microwave, stack_blocks_two, scan_object, and turn_switch.

Table. Independent baselines and MMA source combinations (success rate, %).

Source combinationPolicy branchLIBERO Avg.Clean (10 tasks)Random (10 tasks)
Independent baselinesQwen96.184.982.4
Cosmos96.585.684.5
ABot97.588.287.6
GR00T97.990.189.8
MMA (Qwen + Cosmos)Qwen97.4 (+1.3)88.6 (+3.7)86.8 (+4.4)
Cosmos97.6 (+1.1)89.5 (+3.9)88.2 (+3.7)
Fusion98.190.889.4
MMA (Qwen + ABot)Qwen97.8 (+1.7)89.7 (+4.8)88.1 (+5.7)
ABot98.2 (+0.7)90.6 (+2.4)90.1 (+2.5)
Fusion98.692.091.2
MMA (Cosmos + ABot)Cosmos97.7 (+1.2)90.1 (+4.5)89.0 (+4.5)
ABot98.3 (+0.8)91.0 (+2.8)90.6 (+3.0)
Fusion98.792.591.8
MMA (Qwen + Cosmos + ABot)Qwen98.5 (+2.4)91.4 (+6.5)90.0 (+7.6)
Cosmos98.2 (+1.7)92.7 (+7.1)91.9 (+7.4)
ABot98.9 (+1.4)92.3 (+4.1)92.2 (+4.6)
Fusion99.294.193.4
MMA (Qwen + Cosmos + ABot + GR00T)Qwen98.9 (+2.8)92.2 (+7.3)90.3 (+7.9)
Cosmos98.8 (+2.3)93.0 (+7.4)92.3 (+7.8)
ABot98.9 (+1.4)93.2 (+5.0)92.6 (+5.0)
GR00T99.1 (+1.2)93.8 (+3.7)93.5 (+3.7)
Fusion99.494.994.1

(1) Benefits of Two-Model Collaboration. Average branch gains over independent baselines are Qwen (LIBERO: +1.5%, RoboTwin: +4.7%), Cosmos (LIBERO: +1.2%, RoboTwin: +4.2%), and ABot (LIBERO: +0.8%, RoboTwin: +2.7%). All three sources benefit, with larger gains for smaller, lower-performing models in these experiments.

(2) Comparison of Model Counts.

MetricNumber of modelsLIBERORoboTwin
Average branch gain2+1.1+3.8
3+1.8+6.2
4+2.2+6.7
Fusion gain1 -> 2+1.8+5.8
2 -> 3+0.7+2.5
3 -> 4+0.2+0.8

From two to four models, the three original branches gain an additional (LIBERO: +1.0%, RoboTwin: +2.9%) on average. Four-model fusion improves over the mean independent baseline by (LIBERO: +2.7%, RoboTwin: +9.0%), while successive model additions yield diminishing gains. Branch comparisons and the baseline reference use Qwen, Cosmos, and ABot; two-model results average all pairwise combinations, and RoboTwin averages Clean and Randomized.

(3) Contribution of Each Source. Removing Qwen, Cosmos, or ABot changes fusion performance by (LIBERO: -0.5%, RoboTwin: -1.6%), (LIBERO: -0.6%, RoboTwin: -2.2%), and (LIBERO: -1.1%, RoboTwin: -3.7%), respectively. Each source contributes, with the largest drop after removing ABot, supporting the value of its vision-language component.

AuthorsOpenReview ↗

Response to Reviewer EeFY (Part 2/3)

Q2: Please separate the effects of shared-head joint training, fusion, distillation, and additional supervision, including a shared-head baseline without fusion or distillation and parameter- or compute-matched comparisons.

To clarify the contribution of each component, we organize MMA into three components and evaluate their effects through controlled ablations.

MMA = Multi-Source Joint Training + Action-Centric MoT Fusion + Fusion-to-Branch Query Distillation.

  • Multi-Source Joint Training: Jointly optimizes all individual branches and a mean-query fusion policy using a shared action head and action supervision, without query distillation.
  • Action-Centric MoT Fusion: Replaces mean-query fusion with Action-Centric MoT and Pool for learned cross-source fusion.
  • Fusion-to-Branch Query Distillation: Uses detached fused queries as teacher targets to guide individual branches through cosine alignment and relational matching.

Table. Component ablations of MMA (success rate, %). Parentheses show decreases from the corresponding Full MMA branch or fusion policy (%). Avg. drop is the arithmetic mean of the decreases across LIBERO, Clean, and Random.

SettingPolicy branchLIBERO Avg.CleanRandomAvg. drop
Full MMAQwen98.590.889.5-
Cosmos98.292.591.8-
ABot98.992.292.0-
Fusion99.293.392.9-
Joint trainingQwen96.3 (-2.2)84.2 (-6.6)82.3 (-7.2)-5.3
Cosmos96.3 (-1.9)84.9 (-7.6)84.2 (-7.6)-5.7
ABot97.6 (-1.3)87.8 (-4.4)87.3 (-4.7)-3.5
Fusion97.7 (-1.5)88.2 (-5.1)87.4 (-5.5)-4.0
Joint training + MoT fusionQwen97.4 (-1.1)86.8 (-4.0)85.2 (-4.3)-3.1
Cosmos97.3 (-0.9)87.6 (-4.9)86.9 (-4.9)-3.6
ABot98.3 (-0.6)88.7 (-3.5)87.9 (-4.1)-2.7
Fusion98.4 (-0.8)90.1 (-3.2)89.3 (-3.6)-2.5
Joint training + Query distillationQwen97.2 (-1.3)86.1 (-4.7)84.3 (-5.2)-3.7
Cosmos97.2 (-1.0)86.5 (-6.0)86.1 (-5.7)-4.2
ABot98.2 (-0.7)88.6 (-3.6)87.8 (-4.2)-2.8
Fusion98.3 (-0.9)88.7 (-4.6)87.9 (-5.0)-3.5
Independent baselinesQwen96.1 (-2.4)84.4 (-6.4)82.0 (-7.5)-5.4
Cosmos96.5 (-1.7)84.6 (-7.9)84.4 (-7.4)-5.7
ABot97.5 (-1.4)88.0 (-4.2)87.2 (-4.8)-3.5

(1) Role of Joint Training. Joint training alone achieves average branch success rates of (LIBERO: 96.7%, RoboTwin: 85.1%), close to independent training (LIBERO: 96.7%, RoboTwin: 85.1%). Mean-query fusion achieves (LIBERO: 97.7%, RoboTwin: 87.8%), also close to the strongest independent baseline (LIBERO: 97.5%, RoboTwin: 87.6%). Joint training provides the foundation for MoT fusion and query distillation while preserving efficient single-branch inference.

(2) Contributions of Fusion and Distillation. Both components are essential to achieving full MMA performance. Replacing MoT with mean-query fusion reduces average branch performance by (LIBERO: -1.0%, RoboTwin: -4.9%) and fusion performance by (LIBERO: -0.9%, RoboTwin: -4.8%). Removing query distillation reduces average branch performance by (LIBERO: -0.9%, RoboTwin: -4.3%) and fusion performance by (LIBERO: -0.8%, RoboTwin: -3.4%). These results support combining learned cross-source fusion with fusion-to-branch knowledge transfer.

AuthorsOpenReview ↗

Response to Reviewer EeFY (Part 3/3)

Q3: Please report repeated runs and uncertainty estimates, particularly for the small LIBERO fusion advantage and the real-world results based on 20 trials per task.

We conduct five independent training runs using the same set of five random seeds for LIBERO and RoboTwin. Each run evaluates all MMA branches and the fusion policy, with the number of test trials per task unchanged from the preceding experiments. For RoboTwin 2.0, each policy is evaluated over 100 test trials per task under each condition (Clean and Randomized) in each training run. We report the mean and standard deviation across the five runs.

Table. MMA (Qwen + Cosmos + ABot): success rates across five independent training runs (mean ± standard deviation, %). RoboTwin 2.0 uses the same 10-task subset as the source-combination experiments.

Policy branchLIBERO Avg.Clean (10 tasks)Random (10 tasks)
Qwen98.4 ± 0.291.5 ± 0.589.9 ± 0.5
Cosmos98.3 ± 0.392.6 ± 0.591.9 ± 0.5
ABot98.8 ± 0.292.5 ± 0.492.1 ± 0.5
Fusion99.2 ± 0.294.2 ± 0.593.5 ± 0.6

LIBERO Performance. LIBERO success rates are already near saturation. Across five runs, fusion achieves 99.2 ± 0.2%, versus 98.8 ± 0.2% for the strongest branch, retaining an average advantage of 0.4% with low variability.

Real-World Evaluation. Table 3 includes 1,040 real-robot tests across 13 policies and four tasks, with 20 trials per policy per task, representing substantial experimental effort. We plan to expand both the dataset and evaluation scale for a more comprehensive comparison, but completing additional real-world tests within the rebuttal period is difficult given the time required for physical execution and task resets. Nevertheless, all three MMA branches outperform their corresponding baselines, with an average gain of 9.6%. Fusion achieves 93.8% success, exceeding the strongest same-source baseline by 15.0%. These results provide empirical support for MMA’s effectiveness in the evaluated real-world tasks under a common testing protocol.

Q4: Please simplify or merge the repeated components in Figures 1–5 so that each figure highlights a distinct technical contribution.

Thank you for highlighting the repetition in Figures 1–5. We will remove or merge redundant illustrations, retain a clear framework overview, and simplify component figures to emphasize their distinct roles. Additional details will be moved to the supplementary material to keep the main text concise and readable.

Additional Clarification about Fair Comparison and Model Capacity

For fair comparison, our independent baselines, MMA models, and ablation variants follow the same training settings within each benchmark, except for the components being ablated. The original submission specifies the following training schedules:

  • LIBERO: 50,000 training steps with learning-rate warmup followed by cosine decay to 1e-6. We use 5,000 warmup steps, 7-dimensional actions, and an action chunk length of 8, without proprioceptive inputs.
  • RoboTwin 2.0: 500,000 training steps, with 14-dimensional actions, an action chunk length of 50, and 224 × 224 input images.

Shared Training Settings. Both benchmarks use AdamW with learning rates of 1e-5 for the backbone and 1e-4 for the remaining trainable modules, β = (0.9, 0.95), ε = 1e-8, and weight decay = 1e-8. Training uses 8 GPUs with 8 samples per GPU and a gradient accumulation factor of 1, giving an effective batch size of 64. We also use gradient clipping at 1.0, BF16 precision, and DeepSpeed ZeRO-2.

Matched Model Configurations. As described in Appendix A.1.1 (Comparison groups), each baseline matches its MMA branch in backbone, query interface, adapter, and action-head architecture, with Q=64Q=64 (action queries per source) and D=2048D=2048 (shared feature dimension). We use a one-layer, eight-head adapter and a 16-layer DiT-B action head (12 heads, dropout 0.2, decoder width 1024), with R=8R=8 (noise/time samples per demonstration) for training and 4 Euler steps for inference.

Parameter Counts and Inference Costs. Table 5 reports parameter counts, inference latency, and peak GPU memory for the individual branches and fused policy; Figure 6 provides a breakdown of inference time, and Appendix A.1.3 describes the measurement protocol. Branch improvements therefore do not require increased deployment model capacity.

Dear Reviewer,

Thank you again for your thoughtful review and constructive feedback. In our response, we addressed your questions and provided additional evidence and clarifications.

We would appreciate hearing whether our response addresses your concerns. If any points remain unclear, we would be happy to clarify them.

Thank you for your time and consideration.

Comments on the paper

2 comments
Area_Chair_akYBOpenReview ↗

Please Review the Author Rebuttal and Reassess Your Rating

Dear Reviewers,

Thank you for your time and effort in reviewing the submissions.

The authors have now submitted their rebuttals. I kindly ask you to carefully review the authors' responses and assess whether they have adequately addressed your questions and concerns.

If the rebuttal resolves your concerns, please consider updating your review and adjusting your rating accordingly. If any concerns remain, please feel free to raise them in the discussion.

Your timely feedback and active participation are greatly appreciated, as they are essential to reaching a fair, well-informed decision.

Thank you again for your valuable contributions!

Best, Your AC

Dear AC,

Thank you for already following up with the reviewers. As today is the last day of the discussion period, we would be grateful if you could send a brief final reminder encouraging them to let us know whether our rebuttal addresses their concerns and, if possible, finalize their ratings before the discussion closes.

Thank you again for your help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.