- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper frames VLM-to-VLA adaptation as two separate gaps. One is an input-distribution gap: web images versus robot scenes. The other is an objective gap: next-token text versus action prediction. To bridge them, the authors propose Embodied Trajectory-Coupled (ETC) data: VQA-style supervision built from the same scenes and trajectories as the robot action data. It covers scene grounding, task-oriented affordance and planning, and image-plane waypoint forecasting. The data is organized into a three-stage recipe:
- Distribution Bridging: ETC-only fine-tuning of the VLM, including the ViT.
- Objective Bridging: action and ETC co-training.
- Retentive Adaptation: downstream action and ETC co-training.
The paper also reports compositional induction. Adding ETC for out-of-distribution visual-language conditions, with no OOD action training, improves real-robot performance on held-out compositions. Experiments use PaliGemma/π0-FAST on SimplerEnv (Bridge), LIBERO, VLABench, and two WidowX tasks, with CKA, t-SNE, attention, and probing analyses.
Main contributions and key innovations:
- A unifying framing. The “dual gap” decomposition is clean and intuitive. It motivates aligning the input distribution first and the objective second, all under one next-token objective.
- ETC as a data category. The paper argues that auxiliary supervision helps because it is coupled to the action data’s scenes and trajectories, not merely because it is multimodal. The ETC vs. generic MM comparison (Table 1) is the most useful empirical takeaway.
- A factorial ablation. Table 1 crosses Stage 1 (with or without bridging) with the Stage 2 signal (action / +MM / +ETC / +MM+ETC). This lets the authors report an interaction effect: ETC co-training helps far more after Distribution Bridging (+20.5 vs. +5.2 on VLABench T1).
- Cheap OOD supervision. The real-robot results suggest that OOD ETC produced automatically, using Gemini labels or GPT-Image-2 synthesized scenes, can improve held-out compositions without OOD action training.
Critical strengths
- The paper addresses an important, timely question with a clear thesis and readable exposition.
- The ETC vs. MM comparison under matched ratios is informative. Its gains are consistent across SimplerEnv and both VLABench tracks.
- The ETC category ablation (Fig. 3b) gives an interpretable picture. Trajectory-like supervision helps execution most, while affordance and planning supervision helps instruction understanding most.
- The real-robot experiments include a stage-wise ablation of where ETC is applied (Fig. 5a).
- The ETC construction pipeline, question templates, and scoring rules are documented in useful detail (Appendix C–D).
Critical weaknesses
- Novelty relative to existing embodied VQA co-training and mid-training.
In the main simulation experiments, “ETC” is largely existing datasets: ShareRobot for Bridge and VLA-OS for LIBERO. Using embodied VQA built from robot trajectories, both before policy training and as a co-training signal, is close to prior work the paper cites, including Robo2VLM, RoboBrain, embodied mid-training (Du et al., 2026), Hancock et al. (2025), ChatVLA, π0.5-style co-training, and Lin et al. (2026a). The contribution is mostly in the organization and naming of these ingredients. The paper should say more sharply what is new relative to Du et al. (2026) and Lin et al. (2026a/b) specifically.
- Compute and data are not matched.
The ETC conditions get an extra 20k-step Stage 1 and additional ETC gradient signal at every Stage 2 step. Action-only baselines get neither. The MM control matches the ratio but changes the distribution, so it cannot separate “in-domain images” from “trajectory-coupled semantics.” Several controls would make the central claim much more convincing:
- Compute-matched action-only training (more steps).
- In-domain captioning without trajectory-derived targets.
- ETC with shuffled or corrupted answers.
- Stage 1 is benchmark-specific, which narrows the generality claim.
Table 4 shows a separate Distribution-Bridged VLM per benchmark (ShareRobot, VLA-OS, VLABench VQA). This is domain-matched mid-training on the target environment’s own trajectories, not a general embodied bridge. This should be stated plainly. It also makes the Fig. 4b comparison with π0/π0.5, which the authors already caveat, less meaningful.
- Results that cut against the narrative are under-discussed or moved to the appendix.
- LIBERO: ETC co-training lowers SR after Distribution Bridging (90.7 → 86.8). The best LIBERO number in Table 1 is SigLIP–Gemma action-only (91.5). The paper sets LIBERO aside as “less diagnostic,” which reads as post hoc.
- SigLIP+Gemma (Table 5): ETC co-training drops Bridge SR from 38.5 to 17.7.
- Qwen2.5-VL-3B (Table 6): Distribution Bridging hurts LIBERO in every condition (e.g., 74.3 → 66.6 for action-only, 78.2 → 73.2 for +ETC). This contradicts the claim that Stage 1 gives a better initialization, at least beyond PaliGemma.
- Frozen-ViT bridging (Fig. 2b): this variant falls well below no bridging (67.3 vs. 79.2), so Stage 1 can substantially hurt when configured differently.
- Real robot (Fig. 5a): adding Stage 1 ETC gives no benefit. ✗✓✓ and ✓✓✓ both score 95.83, and ✓✗✗ equals the no-ETC baseline (66.67). The real-robot evidence therefore supports Stages 2–3 but not Stage 1.
These results deserve explicit discussion. Taken together, they suggest the recipe is more backbone- and benchmark-sensitive than the abstract implies.
- The “preservation” evidence is partly tautological or format-driven.
Held-out ShareRobot/VLA-OS scores of 91.92/60.00 with ETC co-training, versus 0.00/0.05 with action-only training (Table 13), are measured on held-out splits of the same ETC data used for co-training. Near-zero scores after action-only training most likely reflect output-format collapse (the model emits action tokens or malformed text) rather than lost representations. Probing-based or format-robust evaluations would support the preservation claim better.
Table 13 also contains entries that need explanation:
- Zero MM scores despite MM training: PaliGemma-ETC + MM+ETC on Bridge scores 0.00 on both A-OKVQA and COCO, even though MM data was in the training mix. This suggests an evaluation or formatting issue.
- MM+ETC beats ETC alone: on VLA-OS, MM+ETC (72.46) outperforms ETC alone (60.00), which conflicts with the claim that MM competes with ETC.
- Loss of general ability: PaliGemma-ETC action-only loses nearly all general MM ability (LaTeX-OCR 2.66 vs. 61.91 without Stage 1). This sits awkwardly with Appendix F.1’s claim that Distribution Bridging preserves pretrained VLM structure.
- Compositional induction is weakly supported.
- Simulation: on VLABench T2, adding OOD ETC decreases SR (41.4 → 38.9) and PS (49.8 → 48.2), improving only IS (+2.2). In simulation, then, OOD ETC improves understanding metrics but not task success.
- Real robot: there are two tasks with small evaluation counts, and the ID/OOD trade-off is under-reported. On the flower task, OOD ETC reduces in-domain score from 47.22 to 37.50/38.89.
- “Without action demonstrations” needs qualification. Section 4.3 states that OOD trajectories were used to construct ETC. If the human-derived OOD ETC includes trajectory-waypoint targets, it carries image-space action information from OOD teleoperation. The paper should specify which ETC categories were used for OOD, and whether OOD robot trajectories were collected.
- Data volume is a confound. Gemini OOD ETC has more records than human OOD ETC (1540 vs. 1120), so the “Gemini beats human” conclusion is confounded with quantity.
- Statistical rigor.
No simulation result reports seeds or confidence intervals, and several conclusions rest on 1–4 point differences (e.g., LIBERO, SimplerEnv +2.0). Real-robot results use 18 ID / 12 OOD rollouts with partial-credit scoring. One rollout-count inconsistency should be checked. With 12 rollouts and scores in 25-point increments, every attainable average is a multiple of about 2.08. The coffee-bag OOD scores 69.44 and 80.56 are not; they are consistent with 36 rollouts instead. Please clarify the evaluation counts.
- Missing details.
- What MM data was used for the VLABench MM ablations? A.3 specifies only Bridge and LIBERO.
- Is VLABench T2 ETC data excluded from the Table 1 / Section 4.2 runs? Table 10 lists T2 ETC, which appears to be the 6% OOD mixture in Section 4.3. If it is included anywhere in Section 4.2, it would leak into the cross-category evaluation.
- How many episodes per simulation benchmark, and what are the definitions of VLABench T3/T4?
- How much manual verification did the Gemini labels require? The paper claims “without additional human annotation.”
- Sensitivity to the 1:4 ETC-to-action ratio and to Stage 1 length.
- Fig. 1c presents an optimization landscape that appears purely conceptual. It should be labeled as illustrative so readers do not read it as an empirical result.
Questions for authors
- Does the gain from ETC persist against an action-only baseline trained for the same total number of samples and steps?
- Why does Distribution Bridging hurt Qwen2.5-VL-3B, and why does ETC co-training hurt SigLIP+Gemma on Bridge?
- For real-robot OOD ETC, were OOD trajectories teleoperated? Does the OOD ETC include trajectory waypoints?
- How many OOD rollouts were run for the coffee-bag task?
- Can you explain the 0.00 A-OKVQA/COCO scores for PaliGemma-ETC + MM+ETC in Table 13?
7 comments
Response part2
W2&Q1: Issue of ETC-specific Gains and Experimental Controls.
(a) Longer action-only training. For VLABench, all runs start from the same 20k-step Stage 1 checkpoint. With an action-to-ETC sample ratio of 4:1, 100k Stage 2 updates with Full ETC process the same total number of training samples as 125k action-only updates. Extending action-only training from 100k to 125k updates improves SR by only 1.9 percentage points.
(b) In-domain ETC without trajectory-derived targets. Co-training on Track 1 action data with Scene-grounding and Task-oriented ETC improves all three metrics over action-only training. Full ETC adds a further 6.7 SR points.
(c) Shuffled supervision. Shuffling VLABench ETC question–answer pairings drops all three metrics below both action-only baselines.
(d) Same domain, disjoint trajectories. Co-training Track 1 action data with Track 2 ETC improves the success rate by 18.9 percentage points over the 100k-step action-only baseline.
(e) Cross-domain ETC. Co-training on Track 1 action data and LIBERO ETC improves all three metrics over action-only training, suggesting that embodied supervision from one environment can support action learning in another.
| Setting | SR | PS | IS |
|---|---|---|---|
| Action-only (100k) | 44.3 | 53.1 | 53.7 |
| Action-only (125k) | 46.2 | 51.8 | 54.6 |
| Co-train without trajectory-derived targets | 58.1 | 72.1 | 75.7 |
| Co-train with shuffled or corrupted answers | 42.1 | 47.9 | 49.6 |
| Only OOD ETC | 63.2 | 73.6 | 74.8 |
| LIBERO ETC | 62.4 | 72.9 | 73.1 |
| Co-train with Full ETC | 64.8 | 77.3 | 77.8 |
Conclusion. On VLABench Track 1, Full ETC still outperforms action-only training with the same Stage 1 checkpoint, total samples processed, and Stage 2 micro-steps. Correctly paired embodied questions and answers improve action learning. Compared with Full ETC, success rates are lower when ETC comes from different scenes, when 2D end-effector trajectory targets are removed, or when question–answer pairs are shuffled.
W3: Issue of Benchmark-Specific Bridging and Baseline Comparability
(a) Benchmark-specific Stage 1, shared training recipe. Thanks for pointing that Stage 1 is benchmark-specific. For each benchmark, we train a separate Distribution-Bridged VLM using ETC derived from that benchmark’s training trajectories. Our claim is that the same training recipe can be applied across benchmarks, with a separately trained checkpoint for each.
(b) Pretraining differences in Fig. 4b. We initialize our model from the pretrained PaliGemma VLM, with no prior robot-action pretraining. The π-series baselines undergo extensive robot-action pretraining. π₀.₅ also trains on subtask prediction, object detection, and multimodal web data alongside action data from diverse robots and environments. Fig. 4b therefore serves as a benchmark comparison with robot-pretrained baselines under their respective training setups.
Response part3
W4&Q2: Issue of Bridging Effects
(a) Bridging effects on LIBERO
Effect of Distribution Bridging on Qwen2.5-VL-3B. Stage 1 is designed to help VLA policies use task-relevant cues as visual scenes and language instructions change. On standard LIBERO, Qwen’s success rate decreases with Stage 1 pretraining (Table 6). We hypothesize that Stage 1 helps Qwen learn task representations that generalize across changes in scenes and instructions. To assess robustness, we evaluate Qwen on LIBERO-Plus, which perturbs object layouts, lighting, language instructions, and background textures while preserving the underlying tasks. Adding Stage 1 improves success rates across all four perturbation types under both Stage 2 training settings. The policy trained with both bridging stages achieves the highest success rate in each category. We report these results and the accompanying analysis in Appendix B.
| Stage 1 ETC pretraining | Stage 2 training | Object layout (%) | Lighting conditions (%) | Language instructions (%) | Background textures (%) |
|---|---|---|---|---|---|
| wo/ | Action only | 36.36 | 54.30 | 7.18 | 31.10 |
| wo/ | Action + VLA-OS ETC | 54.81 | 67.01 | 18.21 | 67.91 |
| w/ | Action only | 62.60 | 65.29 | 13.85 | 64.76 |
| w/ | Action + VLA-OS ETC | 68.05 | 76.29 | 19.74 | 75.98 |
Benchmark-dependent effects on LIBERO. On standard LIBERO, adding ETC after Distribution Bridging reduces PaliGemma’s SR from 90.7 to 86.8. The SigLIP–Gemma action-only baseline achieves 91.5. These results show that ETC co-training can reduce in-domain success rates. Together, these results suggest that the benefits of bridging depend on the backbone and evaluation conditions, and may not be reflected in standard in-domain success rates.
(b) Why Objective Bridging may hurt SigLIP–Gemma. SigLIP and Gemma are pretrained separately and enter Stage 2 without joint vision–language pretraining. We hypothesize that ETC co-training is harder to optimize in this setting because the model must learn visual–language alignment and action prediction simultaneously. Distribution Bridging provides a dedicated phase for aligning visual features with embodied language before action learning begins.
(c) Effect of freezing the ViT during Distribution Bridging. Freezing the ViT prevents ETC supervision from directly adapting visual features to embodied scenes and action-relevant geometry. ETC supervision covers spatial relations, object grounding, and action-relevant geometry. We compare Stage 1 training with the ViT unfrozen and frozen to assess the contribution of visual adaptation. The unfrozen setting achieves 81.2 SR, compared with 67.3 for the frozen setting and 79.2 for the raw PaliGemma baseline. The 13.9-point gap between the two Stage 1 settings shows that updating the visual encoder improves downstream performance in this experiment.
(d) Limited additional gains from Stage 1 on the real robot. On the Coffee task, both variants with Stages 2 and 3 enabled score 95.83, with or without Stage 1. The task’s narrow deployment setting may expose fewer of the visual distribution shifts that Stage 1 is designed to address.
Response part1
We sincerely thank the reviewer for the detailed and constructive feedback, which helped us further examine our experiments and clarify the contributions of our work. We have conducted additional analyses and evaluations to address these concerns and provide a more precise understanding of our findings.
W1: Issue of Novelty
We acknowledge that embodied mid-training and multimodal co-training have been explored in prior work. We reuse ShareRobot for Bridge and VLA-OS for LIBERO, while constructing VLABench ETC data ourselves. Regarding the closest related works, Du et al. (2026) study embodied mid-training to improve VLA initialization. Fanqi Lin et al. (2026) systematically investigate multimodal co-training data modalities and training strategies. Weifeng Lin et al. (2026) study embodied VQA and representation preservation for VLA initialization. These works provide important insights into individual stages of VLA training. However, the interaction between embodied mid-training and subsequent action co-training remains less explored. We study the interaction between embodied initialization and subsequent action co-training, rather than treating them as independent training stages. In particular, how embodied representations acquired before action learning influence subsequent policy optimization, and how continued embodied supervision preserves these representations during action learning, are not systematically established.
Our contribution is to investigate this interaction through a unified ETC-based VLM-to-VLA adaptation pathway, with three aspects:
(a) Interaction between embodied initialization and action co-training. We investigate how Distribution Bridging and Objective Bridging reinforce each other. On VLABench Track 1, the SR gain from Stage 2 ETC co-training over action-only training increases from 5.2 to 20.5 points with Stage 1 initialization. This substantial interaction shows that the benefits of embodied initialization and subsequent ETC co-training are complementary rather than independent.
(b) Role and alignment of embodied supervision. We systematically study different ETC supervision categories through category ablations (Figure 3b) and comparisons with generic multimodal supervision (Table 1). We further compare ETC from different trajectories and show that supervision from disjoint trajectories still benefits action learning, while trajectory-coupled ETC achieves the best performance. These experiments clarify how embodied scene understanding, task-oriented reasoning, and action-aligned trajectory supervision contribute to policy learning.
(c) Compositional induction through OOD supervision. We show that task-relevant OOD ETC, combined with in-domain action data, enables policies to generalize to novel visual-language compositions without corresponding OOD action demonstrations (Table 2). This extends the use of embodied supervision beyond initialization and action co-training to capability expansion under new deployment conditions.