- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MMA, a framework for combining heterogeneous pretrained VLM and VLA sources for robot action learning. It uses learnable action queries and branch-specific adapters to create a unified interface, an Action-Centric Mixture-of-Transformers to fuse representations, and query distillation to transfer fused knowledge to individual branches. A shared flow-matching action head supports both full-fusion and independent single-branch deployment. Experiments on LIBERO, RoboTwin 2.0, and real-world manipulation tasks show improvements over matched independently trained baselines for both the fused policy and individual branches.
Critical strengths
The paper studies a practical problem. It combines pretrained models during training and improves individual branches while allowing them to run independently at inference.
Critical weaknesses
1.The work is primarily an engineering contribution. The paper combines existing techniques for multi-model fusion, action queries, and knowledge distillation into a unified framework. Its contribution mainly lies in system combination and implementation, with relatively limited methodological novelty.
2.Experimental details are insufficiently reported. The paper does not specify some loss weights, the distillation temperature, or the detailed action-head configuration. The real-world experiments and inference latency measurements also lack detailed settings. Providing this information would improve reproducibility and help readers interpret the results.
3.Lack of evaluation with different numbers of models. The paper uses only three models without sufficiently explaining this choice. I suggest adding experiments with two and four models to test whether the framework works across different model counts and to measure the additional benefit of each added model.
Questions for authors
Could the authors respond to the concerns raised in the weaknesses section?
4 comments
Response to Reviewer WqJ1 (Part 2/3)
Q2: Please provide the loss weights, distillation temperature, action-head configuration, real-world experimental settings, and inference-latency measurement protocol.
Training Settings. Baselines, MMA, and ablations use matched settings within each benchmark, except for the ablated components:
- LIBERO: 50,000 steps, 5,000 warmup steps, cosine decay to 1e-6, 7-dimensional actions, chunk length 8, and no proprioceptive inputs.
- RoboTwin 2.0: 500,000 steps, 14-dimensional actions, chunk length 50, and 224 × 224 images.
Both use AdamW with backbone/module learning rates 1e-5/1e-4, β = (0.9, 0.95), ε and weight decay 1e-8, gradient clipping 1.0, BF16, and DeepSpeed ZeRO-2. The effective batch size is 64 (8 GPUs × 8 samples; accumulation factor 1).
Loss and Sampling. We set (fusion loss), (distillation), (cosine alignment), (relational matching), and (temperature). Flow matching uses noise/time samples per demonstration, Beta parameters (1.5, 1.0), and 4 Euler steps at inference; leaves the data batch size unchanged. Distillation and relational ablations set and , respectively.
Model Configuration. Each source has action queries, shared dimension , and a one-layer, eight-head adapter. MoT uses two layers and eight heads. The 16-layer DiT-B action head uses 12 heads of dimension 64, conditioning dimension 2048, output and decoder hidden dimensions 1024, and dropout 0.2. As specified in Appendix A.1.1, each baseline matches its MMA branch in backbone, query interface, adapter, and action-head architecture, so branch gains do not require greater deployment capacity.
Real-World Training and Evaluation. RDT-1B, X-VLA, VLA-Adapter, OpenVLA-OFT, , and are initialized from official weights and fine-tuned with official default settings for 10,000 steps on 200 trajectories across four tasks. MMA and same-source baselines use the same data and steps with our LIBERO/RoboTwin training settings. Qwen3.5 and Cosmos-Reason2 use official VLM weights; ABot-M0 uses its pretrained VLM component. MMA shares an action head through the unified interface, while independent baselines train separate heads.
On a Franka robot arm, we evaluate Placement (three blocks into a box), Replacement (red block replaced by blue), Sorting (red, green, blue from left to right), and Stacking (three-block tower), with 20 trials per task. Task descriptions and demonstration images appear in Appendix A.1.4 and Figure A.1.
Inference Costs. Table 5 reports parameters, peak GPU memory, and latency; Figure 6 shows the timing breakdown, with the protocol in Appendix A.1.3. We use an NVIDIA GeForce RTX 5090 and 50 warmup iterations, keeping precision, inputs, batch size, action horizon, and inference sampling fixed across policies; model and input settings follow training. Median latency spans in-memory PIL images to normalized CPU actions, excluding video decoding, input resizing, RPC, and environment execution. Single-branch inference runs one source and the shared head; fusion runs all sources, fusion, and the shared head.
Supplementary Updates. We will consolidate configurations, schedules, preprocessing, benchmark protocols, real-world dataset details and images, and hardware/timing settings in the revised supplement. For Tables 1, 2, and 4, we will report seed values/counts, independent runs, test trials, checkpoint selection, aggregation, available repeated-run means/standard deviations, and confidence intervals with their calculation methods. We will distinguish training variability from fixed-checkpoint evaluation uncertainty, provide per-task real-world success counts, and separately assess branch-versus-baseline and fusion-versus-strongest-branch comparisons.
Response to Reviewer WqJ1 (Part 3/3)
Q3: Why were three models selected? Please evaluate two- and four-model variants and quantify the benefit of adding each source.
We select three source models of different sizes for our experiments, including: (1) Qwen3.5 (0.8B, a pretrained VLM), (2) Cosmos-Reason2 (2B, a VLM trained for physical reasoning), and (3) ABot-M0 (4B, a pretrained VLA). These models span distinct pretraining regimes and are compatible with the unified action interface, enabling a systematic study of multi-source collaboration for robot action learning.
We agree with the reviewers that broader evaluation across model combinations and source counts is needed. To enable a more comprehensive evaluation and comparison, we introduce a fourth source, GR00T-N1.7 (2B, a generalist VLA). As with ABot-M0, we retain only its vision-language component and initialize its action head from MMA’s shared action head.
Source Combinations.
To reduce training time and computational cost, we conduct the additional RoboTwin 2.0 experiments on a subset of 10 tasks rather than the full set of 50 tasks. Both training and evaluation use the following tasks: adjust_bottle, beat_block_hammer, blocks_ranking_rgb, click_alarmclock, dump_bin_bigbin, move_can_pot, open_microwave, stack_blocks_two, scan_object, and turn_switch.
Table. Independent baselines and MMA source combinations (success rate, %).
| Source combination | Policy branch | LIBERO Avg. | Clean (10 tasks) | Random (10 tasks) |
|---|---|---|---|---|
| Independent baselines | Qwen | 96.1 | 84.9 | 82.4 |
| Cosmos | 96.5 | 85.6 | 84.5 | |
| ABot | 97.5 | 88.2 | 87.6 | |
| GR00T | 97.9 | 90.1 | 89.8 | |
| MMA (Qwen + Cosmos) | Qwen | 97.4 (+1.3) | 88.6 (+3.7) | 86.8 (+4.4) |
| Cosmos | 97.6 (+1.1) | 89.5 (+3.9) | 88.2 (+3.7) | |
| Fusion | 98.1 | 90.8 | 89.4 | |
| MMA (Qwen + ABot) | Qwen | 97.8 (+1.7) | 89.7 (+4.8) | 88.1 (+5.7) |
| ABot | 98.2 (+0.7) | 90.6 (+2.4) | 90.1 (+2.5) | |
| Fusion | 98.6 | 92.0 | 91.2 | |
| MMA (Cosmos + ABot) | Cosmos | 97.7 (+1.2) | 90.1 (+4.5) | 89.0 (+4.5) |
| ABot | 98.3 (+0.8) | 91.0 (+2.8) | 90.6 (+3.0) | |
| Fusion | 98.7 | 92.5 | 91.8 | |
| MMA (Qwen + Cosmos + ABot) | Qwen | 98.5 (+2.4) | 91.4 (+6.5) | 90.0 (+7.6) |
| Cosmos | 98.2 (+1.7) | 92.7 (+7.1) | 91.9 (+7.4) | |
| ABot | 98.9 (+1.4) | 92.3 (+4.1) | 92.2 (+4.6) | |
| Fusion | 99.2 | 94.1 | 93.4 | |
| MMA (Qwen + Cosmos + ABot + GR00T) | Qwen | 98.9 (+2.8) | 92.2 (+7.3) | 90.3 (+7.9) |
| Cosmos | 98.8 (+2.3) | 93.0 (+7.4) | 92.3 (+7.8) | |
| ABot | 98.9 (+1.4) | 93.2 (+5.0) | 92.6 (+5.0) | |
| GR00T | 99.1 (+1.2) | 93.8 (+3.7) | 93.5 (+3.7) | |
| Fusion | 99.4 | 94.9 | 94.1 |
(1) Benefits of Two-Model Collaboration. Average branch gains over independent baselines are Qwen (LIBERO: +1.5%, RoboTwin: +4.7%), Cosmos (LIBERO: +1.2%, RoboTwin: +4.2%), and ABot (LIBERO: +0.8%, RoboTwin: +2.7%). All three sources benefit, with larger gains for smaller, lower-performing models in these experiments.
(2) Comparison of Model Counts.
| Metric | Number of models | LIBERO | RoboTwin |
|---|---|---|---|
| Average branch gain | 2 | +1.1 | +3.8 |
| 3 | +1.8 | +6.2 | |
| 4 | +2.2 | +6.7 | |
| Fusion gain | 1 -> 2 | +1.8 | +5.8 |
| 2 -> 3 | +0.7 | +2.5 | |
| 3 -> 4 | +0.2 | +0.8 |
From two to four models, the three original branches gain an additional (LIBERO: +1.0%, RoboTwin: +2.9%) on average. Four-model fusion improves over the mean independent baseline by (LIBERO: +2.7%, RoboTwin: +9.0%), while successive model additions yield diminishing gains. Branch comparisons and the baseline reference use Qwen, Cosmos, and ABot; two-model results average all pairwise combinations, and RoboTwin averages Clean and Randomized.
(3) Contribution of Each Source. Removing Qwen, Cosmos, or ABot changes fusion performance by (LIBERO: -0.5%, RoboTwin: -1.6%), (LIBERO: -0.6%, RoboTwin: -2.2%), and (LIBERO: -1.1%, RoboTwin: -3.7%), respectively. Each source contributes, with the largest drop after removing ABot, supporting the value of its vision-language component.
Thank you for the detailed responses and the time spent conducting these additional experiments. The expanded component ablations, evaluations with different model combinations and source counts, and clarifications of the experimental settings have substantially addressed my main concerns and strengthened the empirical support for the framework. I appreciate these efforts and will take the additional results and clarifications into consideration when determining my final score.
Response to Reviewer WqJ1 (Part 1/3)
We thank all three reviewers for your thoughtful feedback and constructive suggestions. We appreciate the reviewers’ recognition of MMA’s practicality and deployment flexibility (
EeFY,57mt,WqJ1), flexible framework and comprehensive evaluation (EeFY), and consistent performance gains (57mt). At the same time, we acknowledge the shared concerns regarding different model combinations (EeFY,57mt,WqJ1), individual contributions and statistical validation (EeFY,57mt), and reproducibility (57mt,WqJ1). We provide point-by-point responses to your concerns below.MMA combines a unified action interface, shared action head, action-centric fusion, and fusion-to-branch distillation to jointly learn from heterogeneous pretrained VLM/VLA sources while preserving efficient single-source inference. Average branch gains over independent baselines are (LIBERO: +1.8%, RoboTwin: +6.4%), and fusion achieves (LIBERO: 99.2%, RoboTwin: 93.1%), averaging branch gains across three sources and RoboTwin scores across Clean and Random. This simple design can inform practical robotic systems. Below, we extend the original Table 4 ablations to examine each component’s contribution, and we will clarify the design’s relationship to prior work in the revised manuscript.
To clarify the contribution of each component, we organize MMA into three components and evaluate their effects through controlled ablations.
MMA = Multi-Source Joint Training + Action-Centric MoT Fusion + Fusion-to-Branch Query Distillation.
Table. Component ablations of MMA (success rate, %). Parentheses show decreases from the corresponding Full MMA branch or fusion policy (%). Avg. drop is the arithmetic mean of the decreases across LIBERO, Clean, and Random.
(1) Role of Joint Training. Joint training alone achieves average branch success rates of (LIBERO: 96.7%, RoboTwin: 85.1%), close to independent training (LIBERO: 96.7%, RoboTwin: 85.1%). Mean-query fusion achieves (LIBERO: 97.7%, RoboTwin: 87.8%), also close to the strongest independent baseline (LIBERO: 97.5%, RoboTwin: 87.6%). Joint training provides the foundation for MoT fusion and query distillation while preserving efficient single-branch inference.
(2) Contributions of Fusion and Distillation. Both components are essential to achieving full MMA performance. Replacing MoT with mean-query fusion reduces average branch performance by (LIBERO: -1.0%, RoboTwin: -4.9%) and fusion performance by (LIBERO: -0.9%, RoboTwin: -4.8%). Removing query distillation reduces average branch performance by (LIBERO: -0.9%, RoboTwin: -4.3%) and fusion performance by (LIBERO: -0.8%, RoboTwin: -3.4%). These results support combining learned cross-source fusion with fusion-to-branch knowledge transfer.