acceptodds
Under review as a conference paper at ICLR 2027

Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data

Abstract

Vision-language models (VLMs) are powerful general-purpose reasoners, yet converting them into robot control policies (VLAs) is surprisingly difficult. The root cause is a two-fold gap: VLMs are trained on internet-scale images with language-understanding objectives, while VLAs must perceive robot scenes and predict motor actions. Fine-tuning a VLM directly on robot action data forces the model to cross both gaps at once—the learning curve is steep and the rich generalizations learned during pretraining tend to degrade rather than transfer. We argue that this gap can be bridged gradually with the right intermediate data. We introduce embodied trajectory-coupled (ETC) data—vision-language supervision derived from the same robot scenes and trajectories used for action learning. Because ETC data shares the visual context of robot operation while retaining familiar language-understanding objectives, it provides a natural stepping stone between VLM pretraining and VLA fine-tuning. Building on this, we design a three-stage training recipe. Distribution Bridging first adapts the VLM to embodied visual-language semantics. Objective Bridging then gradually shifts the model toward action prediction while preserving the acquired representations. Retentive Adaptation finally specializes the policy to the target deployment domain. We further show that mixing task-relevant out-of-distribution ETC data with a small amount of in-distribution action data enables the model to generalize to novel visual-language conditions without requiring additional robot demonstrations. Simulation and real-robot experiments confirm that this gradual bridging strategy is the key to transferring VLM generalization into robust, deployable robot policies.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot foundation models, general-purpose knowledge systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Mxf2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper frames VLM-to-VLA adaptation as two separate gaps. One is an input-distribution gap: web images versus robot scenes. The other is an objective gap: next-token text versus action prediction. To bridge them, the authors propose Embodied Trajectory-Coupled (ETC) data: VQA-style supervision built from the same scenes and trajectories as the robot action data. It covers scene grounding, task-oriented affordance and planning, and image-plane waypoint forecasting. The data is organized into a three-stage recipe:

  1. Distribution Bridging: ETC-only fine-tuning of the VLM, including the ViT.
  2. Objective Bridging: action and ETC co-training.
  3. Retentive Adaptation: downstream action and ETC co-training.

The paper also reports compositional induction. Adding ETC for out-of-distribution visual-language conditions, with no OOD action training, improves real-robot performance on held-out compositions. Experiments use PaliGemma/π0-FAST on SimplerEnv (Bridge), LIBERO, VLABench, and two WidowX tasks, with CKA, t-SNE, attention, and probing analyses.

Main contributions and key innovations:

  • A unifying framing. The “dual gap” decomposition is clean and intuitive. It motivates aligning the input distribution first and the objective second, all under one next-token objective.
  • ETC as a data category. The paper argues that auxiliary supervision helps because it is coupled to the action data’s scenes and trajectories, not merely because it is multimodal. The ETC vs. generic MM comparison (Table 1) is the most useful empirical takeaway.
  • A factorial ablation. Table 1 crosses Stage 1 (with or without bridging) with the Stage 2 signal (action / +MM / +ETC / +MM+ETC). This lets the authors report an interaction effect: ETC co-training helps far more after Distribution Bridging (+20.5 vs. +5.2 on VLABench T1).
  • Cheap OOD supervision. The real-robot results suggest that OOD ETC produced automatically, using Gemini labels or GPT-Image-2 synthesized scenes, can improve held-out compositions without OOD action training.
Critical strengths
  1. The paper addresses an important, timely question with a clear thesis and readable exposition.
  2. The ETC vs. MM comparison under matched ratios is informative. Its gains are consistent across SimplerEnv and both VLABench tracks.
  3. The ETC category ablation (Fig. 3b) gives an interpretable picture. Trajectory-like supervision helps execution most, while affordance and planning supervision helps instruction understanding most.
  4. The real-robot experiments include a stage-wise ablation of where ETC is applied (Fig. 5a).
  5. The ETC construction pipeline, question templates, and scoring rules are documented in useful detail (Appendix C–D).
Critical weaknesses
  1. Novelty relative to existing embodied VQA co-training and mid-training.

In the main simulation experiments, “ETC” is largely existing datasets: ShareRobot for Bridge and VLA-OS for LIBERO. Using embodied VQA built from robot trajectories, both before policy training and as a co-training signal, is close to prior work the paper cites, including Robo2VLM, RoboBrain, embodied mid-training (Du et al., 2026), Hancock et al. (2025), ChatVLA, π0.5-style co-training, and Lin et al. (2026a). The contribution is mostly in the organization and naming of these ingredients. The paper should say more sharply what is new relative to Du et al. (2026) and Lin et al. (2026a/b) specifically.

  1. Compute and data are not matched.

The ETC conditions get an extra 20k-step Stage 1 and additional ETC gradient signal at every Stage 2 step. Action-only baselines get neither. The MM control matches the ratio but changes the distribution, so it cannot separate “in-domain images” from “trajectory-coupled semantics.” Several controls would make the central claim much more convincing:

  • Compute-matched action-only training (more steps).
  • In-domain captioning without trajectory-derived targets.
  • ETC with shuffled or corrupted answers.
  1. Stage 1 is benchmark-specific, which narrows the generality claim.

Table 4 shows a separate Distribution-Bridged VLM per benchmark (ShareRobot, VLA-OS, VLABench VQA). This is domain-matched mid-training on the target environment’s own trajectories, not a general embodied bridge. This should be stated plainly. It also makes the Fig. 4b comparison with π0/π0.5, which the authors already caveat, less meaningful.

  1. Results that cut against the narrative are under-discussed or moved to the appendix.
  • LIBERO: ETC co-training lowers SR after Distribution Bridging (90.7 → 86.8). The best LIBERO number in Table 1 is SigLIP–Gemma action-only (91.5). The paper sets LIBERO aside as “less diagnostic,” which reads as post hoc.
  • SigLIP+Gemma (Table 5): ETC co-training drops Bridge SR from 38.5 to 17.7.
  • Qwen2.5-VL-3B (Table 6): Distribution Bridging hurts LIBERO in every condition (e.g., 74.3 → 66.6 for action-only, 78.2 → 73.2 for +ETC). This contradicts the claim that Stage 1 gives a better initialization, at least beyond PaliGemma.
  • Frozen-ViT bridging (Fig. 2b): this variant falls well below no bridging (67.3 vs. 79.2), so Stage 1 can substantially hurt when configured differently.
  • Real robot (Fig. 5a): adding Stage 1 ETC gives no benefit. ✗✓✓ and ✓✓✓ both score 95.83, and ✓✗✗ equals the no-ETC baseline (66.67). The real-robot evidence therefore supports Stages 2–3 but not Stage 1.

These results deserve explicit discussion. Taken together, they suggest the recipe is more backbone- and benchmark-sensitive than the abstract implies.

  1. The “preservation” evidence is partly tautological or format-driven.

Held-out ShareRobot/VLA-OS scores of 91.92/60.00 with ETC co-training, versus 0.00/0.05 with action-only training (Table 13), are measured on held-out splits of the same ETC data used for co-training. Near-zero scores after action-only training most likely reflect output-format collapse (the model emits action tokens or malformed text) rather than lost representations. Probing-based or format-robust evaluations would support the preservation claim better.

Table 13 also contains entries that need explanation:

  • Zero MM scores despite MM training: PaliGemma-ETC + MM+ETC on Bridge scores 0.00 on both A-OKVQA and COCO, even though MM data was in the training mix. This suggests an evaluation or formatting issue.
  • MM+ETC beats ETC alone: on VLA-OS, MM+ETC (72.46) outperforms ETC alone (60.00), which conflicts with the claim that MM competes with ETC.
  • Loss of general ability: PaliGemma-ETC action-only loses nearly all general MM ability (LaTeX-OCR 2.66 vs. 61.91 without Stage 1). This sits awkwardly with Appendix F.1’s claim that Distribution Bridging preserves pretrained VLM structure.
  1. Compositional induction is weakly supported.
  • Simulation: on VLABench T2, adding OOD ETC decreases SR (41.4 → 38.9) and PS (49.8 → 48.2), improving only IS (+2.2). In simulation, then, OOD ETC improves understanding metrics but not task success.
  • Real robot: there are two tasks with small evaluation counts, and the ID/OOD trade-off is under-reported. On the flower task, OOD ETC reduces in-domain score from 47.22 to 37.50/38.89.
  • “Without action demonstrations” needs qualification. Section 4.3 states that OOD trajectories were used to construct ETC. If the human-derived OOD ETC includes trajectory-waypoint targets, it carries image-space action information from OOD teleoperation. The paper should specify which ETC categories were used for OOD, and whether OOD robot trajectories were collected.
  • Data volume is a confound. Gemini OOD ETC has more records than human OOD ETC (1540 vs. 1120), so the “Gemini beats human” conclusion is confounded with quantity.
  1. Statistical rigor.

No simulation result reports seeds or confidence intervals, and several conclusions rest on 1–4 point differences (e.g., LIBERO, SimplerEnv +2.0). Real-robot results use 18 ID / 12 OOD rollouts with partial-credit scoring. One rollout-count inconsistency should be checked. With 12 rollouts and scores in 25-point increments, every attainable average is a multiple of about 2.08. The coffee-bag OOD scores 69.44 and 80.56 are not; they are consistent with 36 rollouts instead. Please clarify the evaluation counts.

  1. Missing details.
  • What MM data was used for the VLABench MM ablations? A.3 specifies only Bridge and LIBERO.
  • Is VLABench T2 ETC data excluded from the Table 1 / Section 4.2 runs? Table 10 lists T2 ETC, which appears to be the 6% OOD mixture in Section 4.3. If it is included anywhere in Section 4.2, it would leak into the cross-category evaluation.
  • How many episodes per simulation benchmark, and what are the definitions of VLABench T3/T4?
  • How much manual verification did the Gemini labels require? The paper claims “without additional human annotation.”
  • Sensitivity to the 1:4 ETC-to-action ratio and to Stage 1 length.
  • Fig. 1c presents an optimization landscape that appears purely conceptual. It should be labeled as illustrative so readers do not read it as an empirical result.
Questions for authors
  • Does the gain from ETC persist against an action-only baseline trained for the same total number of samples and steps?
  • Why does Distribution Bridging hurt Qwen2.5-VL-3B, and why does ETC co-training hurt SigLIP+Gemma on Bridge?
  • For real-robot OOD ETC, were OOD trajectories teleoperated? Does the OOD ETC include trajectory waypoints?
  • How many OOD rollouts were run for the coffee-bag task?
  • Can you explain the 0.00 A-OKVQA/COCO scores for PaliGemma-ETC + MM+ETC in Table 13?
7 comments
AuthorsOpenReview ↗

Response part1

We sincerely thank the reviewer for the detailed and constructive feedback, which helped us further examine our experiments and clarify the contributions of our work. We have conducted additional analyses and evaluations to address these concerns and provide a more precise understanding of our findings.

W1: Issue of Novelty

We acknowledge that embodied mid-training and multimodal co-training have been explored in prior work. We reuse ShareRobot for Bridge and VLA-OS for LIBERO, while constructing VLABench ETC data ourselves. Regarding the closest related works, Du et al. (2026) study embodied mid-training to improve VLA initialization. Fanqi Lin et al. (2026) systematically investigate multimodal co-training data modalities and training strategies. Weifeng Lin et al. (2026) study embodied VQA and representation preservation for VLA initialization. These works provide important insights into individual stages of VLA training. However, the interaction between embodied mid-training and subsequent action co-training remains less explored. We study the interaction between embodied initialization and subsequent action co-training, rather than treating them as independent training stages. In particular, how embodied representations acquired before action learning influence subsequent policy optimization, and how continued embodied supervision preserves these representations during action learning, are not systematically established.

Our contribution is to investigate this interaction through a unified ETC-based VLM-to-VLA adaptation pathway, with three aspects:

(a) Interaction between embodied initialization and action co-training. We investigate how Distribution Bridging and Objective Bridging reinforce each other. On VLABench Track 1, the SR gain from Stage 2 ETC co-training over action-only training increases from 5.2 to 20.5 points with Stage 1 initialization. This substantial interaction shows that the benefits of embodied initialization and subsequent ETC co-training are complementary rather than independent.

(b) Role and alignment of embodied supervision. We systematically study different ETC supervision categories through category ablations (Figure 3b) and comparisons with generic multimodal supervision (Table 1). We further compare ETC from different trajectories and show that supervision from disjoint trajectories still benefits action learning, while trajectory-coupled ETC achieves the best performance. These experiments clarify how embodied scene understanding, task-oriented reasoning, and action-aligned trajectory supervision contribute to policy learning.

(c) Compositional induction through OOD supervision. We show that task-relevant OOD ETC, combined with in-domain action data, enables policies to generalize to novel visual-language compositions without corresponding OOD action demonstrations (Table 2). This extends the use of embodied supervision beyond initialization and action co-training to capability expansion under new deployment conditions.

AuthorsOpenReview ↗

Response part2

W2&Q1: Issue of ETC-specific Gains and Experimental Controls.

(a) Longer action-only training. For VLABench, all runs start from the same 20k-step Stage 1 checkpoint. With an action-to-ETC sample ratio of 4:1, 100k Stage 2 updates with Full ETC process the same total number of training samples as 125k action-only updates. Extending action-only training from 100k to 125k updates improves SR by only 1.9 percentage points.

(b) In-domain ETC without trajectory-derived targets. Co-training on Track 1 action data with Scene-grounding and Task-oriented ETC improves all three metrics over action-only training. Full ETC adds a further 6.7 SR points.

(c) Shuffled supervision. Shuffling VLABench ETC question–answer pairings drops all three metrics below both action-only baselines.

(d) Same domain, disjoint trajectories. Co-training Track 1 action data with Track 2 ETC improves the success rate by 18.9 percentage points over the 100k-step action-only baseline.

(e) Cross-domain ETC. Co-training on Track 1 action data and LIBERO ETC improves all three metrics over action-only training, suggesting that embodied supervision from one environment can support action learning in another.

SettingSRPSIS
Action-only (100k)44.353.153.7
Action-only (125k)46.251.854.6
Co-train without trajectory-derived targets58.172.175.7
Co-train with shuffled or corrupted answers42.147.949.6
Only OOD ETC63.273.674.8
LIBERO ETC62.472.973.1
Co-train with Full ETC64.877.377.8

Conclusion. On VLABench Track 1, Full ETC still outperforms action-only training with the same Stage 1 checkpoint, total samples processed, and Stage 2 micro-steps. Correctly paired embodied questions and answers improve action learning. Compared with Full ETC, success rates are lower when ETC comes from different scenes, when 2D end-effector trajectory targets are removed, or when question–answer pairs are shuffled.

W3: Issue of Benchmark-Specific Bridging and Baseline Comparability

(a) Benchmark-specific Stage 1, shared training recipe. Thanks for pointing that Stage 1 is benchmark-specific. For each benchmark, we train a separate Distribution-Bridged VLM using ETC derived from that benchmark’s training trajectories. Our claim is that the same training recipe can be applied across benchmarks, with a separately trained checkpoint for each.

(b) Pretraining differences in Fig. 4b. We initialize our model from the pretrained PaliGemma VLM, with no prior robot-action pretraining. The π-series baselines undergo extensive robot-action pretraining. π₀.₅ also trains on subtask prediction, object detection, and multimodal web data alongside action data from diverse robots and environments. Fig. 4b therefore serves as a benchmark comparison with robot-pretrained baselines under their respective training setups.

AuthorsOpenReview ↗

Response part3

W4&Q2: Issue of Bridging Effects

(a) Bridging effects on LIBERO

Effect of Distribution Bridging on Qwen2.5-VL-3B. Stage 1 is designed to help VLA policies use task-relevant cues as visual scenes and language instructions change. On standard LIBERO, Qwen’s success rate decreases with Stage 1 pretraining (Table 6). We hypothesize that Stage 1 helps Qwen learn task representations that generalize across changes in scenes and instructions. To assess robustness, we evaluate Qwen on LIBERO-Plus, which perturbs object layouts, lighting, language instructions, and background textures while preserving the underlying tasks. Adding Stage 1 improves success rates across all four perturbation types under both Stage 2 training settings. The policy trained with both bridging stages achieves the highest success rate in each category. We report these results and the accompanying analysis in Appendix B.

Stage 1 ETC pretrainingStage 2 trainingObject layout (%)Lighting conditions (%)Language instructions (%)Background textures (%)
wo/Action only36.3654.307.1831.10
wo/Action + VLA-OS ETC54.8167.0118.2167.91
w/Action only62.6065.2913.8564.76
w/Action + VLA-OS ETC68.0576.2919.7475.98

Benchmark-dependent effects on LIBERO. On standard LIBERO, adding ETC after Distribution Bridging reduces PaliGemma’s SR from 90.7 to 86.8. The SigLIP–Gemma action-only baseline achieves 91.5. These results show that ETC co-training can reduce in-domain success rates. Together, these results suggest that the benefits of bridging depend on the backbone and evaluation conditions, and may not be reflected in standard in-domain success rates.

(b) Why Objective Bridging may hurt SigLIP–Gemma. SigLIP and Gemma are pretrained separately and enter Stage 2 without joint vision–language pretraining. We hypothesize that ETC co-training is harder to optimize in this setting because the model must learn visual–language alignment and action prediction simultaneously. Distribution Bridging provides a dedicated phase for aligning visual features with embodied language before action learning begins.

(c) Effect of freezing the ViT during Distribution Bridging. Freezing the ViT prevents ETC supervision from directly adapting visual features to embodied scenes and action-relevant geometry. ETC supervision covers spatial relations, object grounding, and action-relevant geometry. We compare Stage 1 training with the ViT unfrozen and frozen to assess the contribution of visual adaptation. The unfrozen setting achieves 81.2 SR, compared with 67.3 for the frozen setting and 79.2 for the raw PaliGemma baseline. The 13.9-point gap between the two Stage 1 settings shows that updating the visual encoder improves downstream performance in this experiment.

(d) Limited additional gains from Stage 1 on the real robot. On the Coffee task, both variants with Stages 2 and 3 enabled score 95.83, with or without Stage 1. The task’s narrow deployment setting may expose fewer of the visual distribution shifts that Stage 1 is designed to address.

Reviewer_igiBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to turn a VLM into a VLA using what the authors call embodied trajectory-coupled (ETC) vision-language data, which covers scene grounding, task/affordance understanding and trajectory prediction. The recipe has three steps. The VLM is first adapted on ETC data. Action prediction is then co-trained with ETC, and the same joint supervision is kept during adaptation to the deployment domain. The experiments compare supervision and initialization choices on SimplerEnv, LIBERO and VLABench, with additional results on other backbones and two WidowX real-robot tasks. In one more setting, only the ETC data comes from the target (OOD) condition while action training stays in-domain. This setting tests whether non-action supervision alone can help the policy reuse manipulation skills it already has.

Critical strengths
  1. For me, the most informative part of the paper is the crossed initialization × supervision study in Table 1. On VLABench, the SR gain from ETC co-training goes from 5.2 to 20.5 points on T1 and from −1.0 to 13.1 on T2 once Stage 1 is added. That said, generic MM data shows a similarly large interaction. I therefore read this as a result about stage ordering and supervision content in general, not as something specific to ETC. Still, with the same Stage 1, ETC beats MM by 3.8/2.1 points on T1/T2 and by 11.5 on SimplerEnv. I think this is practically useful for anyone building a VLA from a VLM.

  2. I also liked the target-condition experiments. Here the target condition appears only in the supervision, not in action training, and the supervision includes first-frame annotations and synthesized images. The coffee Gemini OOD score of 91.67 is encouraging. If it comes from a single group of 12 rollouts under the stated rubric, it requires eleven complete successes. I also appreciate that the authors report the negative execution result in simulation and the weaker synthesized-flower result, because these show where the approach stops working.

Critical weaknesses
  1. My main concern is how the real-robot results are aggregated. Appendix E describes twelve OOD rollouts with coffee rewards of 0/25/100. Under this rubric, I could not reproduce the Table 2 means of 69.44 and 80.56 from a single equally weighted group. Multiple groups or per-condition averaging could explain these numbers, but the paper does not say which was used. For flower insertion, a mean of 50 is possible even if no insertion is ever completed, i.e., when only the object/vase selection is correct. Without counts per reward level or complete-task success rates, I cannot tell how much of the compositional-execution claim is actually supported.

  2. The mechanism and retention claims go further than the experiments show. In Table 2, OOD ETC raises IS in simulation but lowers SR/PS. Stage 3 is evaluated only with deployment-domain scores, and I did not find a direct measurement of old-domain retention. Table 1 compares ETC with generic MM, so it does not isolate whether it matters that the data comes from actual trajectories. As noted above, the large stage interaction is not unique to ETC either. Table 6 also shows Stage 1 hurting on Qwen. None of this undermines the PaliGemma recipe, which seems useful, but the "bridging", "coupling" and knowledge-preservation interpretation should be scoped more carefully.

Questions for authors
  1. How many OOD rollouts were run per variant, and with how many groups/seeds? How are conditions weighted and aggregated? Could you report the counts at each reward level and the number of complete-task successes, especially for the two coffee means above and the flower variants? This would clear up the numbers and show whether the flower gains include completed insertions, not just correct selection.

  2. What does each OOD variant actually need: full trajectories, trajectory-derived targets, static first frames, or synthesized images? How are the examples used to build ETC kept separate from the closed-loop evaluation? Please separate the human-derived data requirements from those of the first-frame and synthetic variants. I am not assuming that the latter need new action demonstrations.

  3. What is the most specific retention/bridging claim that the current experiments support? Is there any Stage 3 evaluation on the old domain or on understanding, or any evidence that separates trajectory provenance from supervision content? Pointing to existing results, or narrowing the claim, would substantially improve my assessment. I am not asking for a broad new model comparison.

8 comments
AuthorsOpenReview ↗

Response part1

We thank the reviewer for the thoughtful feedback. The comments on bridging, retention, and trajectory coupling prompted us to further examine the scope of our claims. While we appreciate that a broad model comparison is not requested, we provide targeted analyses and additional evaluations using existing experimental settings to clarify the evidence behind our claims and better establish their limitations.

W1&Q1: Issue of Real-Robot Evaluation

1. Coffee-bag scoring and corrected results

We apologize for the mismatch between the ID and OOD scoring rubrics for the coffee-bag task. ID rollouts used 0/25/100, whereas OOD rollouts used 0/33⅓/100. Appendix E documented the ID rubric but omitted the different OOD rubric. We have rescored the OOD rollouts so that both splits use 0/25/100. Under the unified rubric, the two OOD scores highlighted by the reviewer change from 69.44 to 66.67 and from 80.56 to 79.17, respectively. The table reports the corrected scores, counts at each reward level, and full-task successes.

Training variantID process scoreID counts (0, 25, 100)ID full successesOOD process scoreOOD counts (0, 25, 100)OOD full successes
Action only91.67(0, 2, 16)16/1839.58(5, 3, 4)4/12
+ ID ETC95.83(0, 1, 17)17/1866.67(1, 4, 7)7/12
+ OOD ETC (Human)91.67(0, 2, 16)16/1879.17(1, 2, 9)9/12
+ OOD ETC (Gemini)91.67(0, 2, 16)16/1891.67(1, 0, 11)11/12

All training variants were evaluated under the same OOD protocol. After rescoring, both OOD ETC variants still achieve higher process scores and full-task success counts than the action-only and ID ETC variants. This supports our original conclusion that OOD ETC helps the policy generalize coffee-bag placement to unseen visual–language compositions without action demonstrations for those compositions. We have revised the coffee-bag OOD process scores in Table 2 and Section 4.3.1.

2. Flower target selection and completed insertions

Flower insertion remained difficult: every variant completed 0 of 12 OOD insertions. The higher process scores reflect partial progress, mainly selecting the correct flower and vase and, in some rollouts, grasping the correct flower before dropping it. This is the compositional generalization we observed: OOD ETC helped the policy use what it learned about the targets to guide its actions in unseen flower–vase combinations.

That transfer was limited by the underlying manipulation skill. For example, when the grasped stem shifted out of alignment with the vase opening, the policy could not reliably correct it. The action-only baseline also completed 0 of 18 ID insertions, suggesting that the policy had yet to learn reliable insertion, possibly because of limited action demonstrations. We have revised Section 4.3.1 to clarify the scope of the observed generalization and report the insertion success counts.

Training variantID process scoreID counts (0, 25, 50, 75, 100)ID full successesOOD process scoreOOD counts (0, 25, 50, 75, 100)OOD full successes
Action only47.22(3, 0, 11, 4, 0)0/1827.08(5, 2, 4, 1, 0)0/12
Action + ID ETC45.83(1, 4, 10, 3, 0)0/1827.08(5, 2, 4, 1, 0)0/12
Action + OOD ETC (Human)37.50(2, 7, 7, 2, 0)0/1850.00(0, 3, 6, 3, 0)0/12
Action + OOD ETC (GPT-Image-2)38.89(3, 4, 9, 2, 0)0/1837.50(3, 2, 5, 2, 0)0/12

3. Rollout counts, scene variation, and aggregation

For each task and training variant, we evaluated one trained policy on 18 ID rollouts and 12 OOD rollouts. We varied object types and positions across rollouts within each split. Each variant had one evaluation set per split, with no additional groups or repeated training seeds. Within each task and split, all variants were evaluated on the same scene configurations. Every rollout received equal weight within its split. We computed each process score as the arithmetic mean over the 18 ID or 12 OOD rollout scores, with no intermediate averaging by scene condition. We have updated Appendix E.2 to document the rollout protocol, equal-weight aggregation, and the unified 0/25/100 coffee-bag rubric, and to report the reward-level counts for both tasks and splits.

AuthorsOpenReview ↗

Response part2

Q2: Issue of OOD ETC Construction

Data requirements. Human-derived ETC uses recorded robot trajectories from a dedicated data collection to extract 2D trajectory targets. The other two variants use only images from that same collection, including initial or intermediate frames. Gemini ETC generates 2D trajectory annotations directly from the real images. GPT-Image-2 ETC uses the images to synthesize OOD scenes, which Gemini then annotates. Both image-based variants generate their trajectory targets without using the recorded robot motion.

Evaluation and action supervision. The data used to construct ETC were collected separately from the scored closed-loop evaluation rollouts. Across all variants, the dedicated collection was used exclusively for ETC construction, and action supervision used only ID demonstrations.

AuthorsOpenReview ↗

Response part3

W2: Issue of Bridging and Retention Evidence

1. Retention of Old-Domain Understanding and Task Performance

(a) Retention of Old-Domain Understanding. Old-domain understanding after Stage 3. We evaluate Stage 3 checkpoints on the in-domain VLABench ETC benchmark, comparing action-only adaptation and Action + Expanded OOD ETC from the same PaliGemma-VLABench initialization. We also report results for both the original PaliGemma backbone and its counterpart pretrained on VLABench ETC. These results demonstrate that continued OOD ETC co-training substantially mitigates the forgetting of previously acquired embodied understanding during Stage 3 adaptation. We have added the results to Appendix D.5.

InitTraining mixOverallAffordanceGoal DescriptionSpatial UnderstandingTask PlanningTrajectory
PaliGemma—9.5524.677.1311.044.920.00
PaliGemma-VLABench—56.3847.6347.5489.6984.0912.94
PaliGemma-VLABenchAction only0.000.000.000.000.000.00
PaliGemma-VLABenchAction + Expanded OOD ETC55.2472.2239.2981.2542.8660.00

(b) In-domain performance trade-off with OOD ETC. We evaluate the Stage 2 policies on VLABench Track 1 to measure how adding OOD ETC affects performance on the original tasks, with the Track 1 action data held fixed. Both settings that use OOD ETC score lower than ID ETC alone on SR, PS, and IS. This illustrates the trade-off in in-domain performance when co-training with OOD ETC. We have added the results to Table 2.

SettingTrack 1 SRTrack 1 PSTrack 1 ISTrack 2 SRTrack 2 PSTrack 2 IS
ID ETC only64.877.377.841.449.851.1
ID ETC + OOD ETC60.264.471.638.948.253.3
Expanded OOD ETC63.273.674.846.6752.0053.96

(c) OOD execution with expanded ETC coverage. In the original Table 2 setting, OOD ETC accounted for only 6% of the ETC mixture. This setting improved Track 2 IS from 51.1 to 53.3, while SR and PS declined. To test whether broader OOD ETC coverage could also improve execution, we expanded the Track 2 ETC dataset to match the size of the ID ETC dataset and used it as the sole source of ETC supervision, keeping the Track 1 action data fixed. The result is in the table above. Compared with ID ETC alone, this setting improved all three Track 2 metrics. SR increased from 41.4 to 46.67, PS from 49.8 to 52.00, and IS from 51.1 to 53.96.

Reviewer_oph8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes embodied trajectory-coupled (ETC) data: vision-language supervision about robot scenes, affordances, task plans, and future gripper trajectories. It uses this data in three stages: adapting a pretrained VLM to robot scenes, jointly training language understanding and action prediction, and continuing this joint training during deployment-task adaptation. Experiments on three simulation benchmarks and two real-robot tasks show that ETC initialization and co-training can reinforce one another. The paper also explores learning from supervision about new visual-language conditions without directly training on their action demonstrations, with promising hardware results but mixed simulation results.

Critical strengths
  • Addresses an important training problem through a practical approach: ETC reuses robot data to provide supervision that is closer to manipulation than generic image-language data, making the proposed intermediate training stage well motivated.
  • Useful ablations of the training stages: The paper compares initialization and co-training choices jointly, rather than evaluating each in isolation. The substantial interaction between these stages on VLABench is one of its strongest findings.
  • Evaluates both understanding and execution: Policy performance is complemented by embodied-understanding evaluations and representation analyses, helping examine what the additional supervision preserves.
  • Promising use of supervision without new action labels: The coffee-bag and flower experiments suggest that annotations or generated images can help adapt existing manipulation skills to new conditions.
Critical weaknesses
  • The experiments do not isolate the importance of trajectory coupling. ETC differs from generic multimodal supervision in several ways: it contains robot scenes, task-relevant labels, and information derived from trajectories. The current comparisons establish that this combination is useful, but not which aspect is responsible. Please compare against equally sized, domain-matched robot VQA data drawn from different trajectories or scenes. This would clarify whether coupling to the action-training trajectories is essential or whether relevant embodied supervision is sufficient.
  • The compositional-generalization claim is stronger than the results support. In simulation, adding OOD ETC improves the understanding score but reduces task success from 41.4 to 38.9. On the flower task, OOD performance improves while in-domain performance declines. The evidence therefore supports condition-specific gains with possible trade-offs, rather than a consistently more generalizable policy. Please evaluate additional held-out conditions and report both in-domain and OOD execution performance prominently. Alternatively, clarify or adapt the language in the paper.
Questions for authors

My questions are incorporated into the weaknesses section as actionable steps the authors can take. I don't have any further questions for the authors beyond that.

4 comments
AuthorsOpenReview ↗

Response part1

We thank the reviewer for the thoughtful and constructive feedback, and for recognizing the practical value of ETC. We have conducted additional experiments to further examine trajectory coupling and compositional generalization, and refined our claims to better reflect the evidence.

W1: Issue of Trajectory Coupling in ETC.

We clarify what the category ablation in Figure 3b establishes about the role of action-aligned supervision, and introduce a new comparison using ETC from different trajectories to examine whether exact trajectory coupling is necessary or whether broader embodied supervision can also benefit action learning. 1. The contributions of different ETC labels

We apologize for the ambiguity in Figure 3b. Each of the three category rows reports a separate Stage 2 co-training run using only that ETC category. The Full ETC row uses all three categories together.

(a) Scene grounding already provides substantial gains. VQA about objects and spatial relations raises Track 1 SR from 44.3 to 56.2 and IS from 53.7 to 73.6. This shows that supervision about the robot scene is useful even in the absence of task-planning and waypoint labels.

(b) Task-oriented labels yield the strongest Track 2 understanding among individual categories. Affordance and task-planning labels raise Track 2 IS from 36.5 to 50.7, the highest IS among the three single-category settings.

(c) Trajectory waypoint labels yield the strongest execution among individual categories. Action-aligned planning, which predicts future gripper waypoints from recorded trajectories, achieves the highest single-category SR and PS on both tracks (62.1/74.4 on Track 1 and 43.8/49.9 on Track 2). Combining all three categories gives the highest scores on all Track 1 metrics (64.8/77.3/77.8) and the highest Track 2 IS (51.1).

The gains are therefore spread across several types of supervision, with different labels helping different outcomes. Figure 3b varies label content within robot scenes. The effect of robot-scene exposure itself remains open, as does whether the VQA and action data need to share trajectories. We have revised Figure 3b to mark the ETC categories used in each row and clarify these controls in the caption and main text.

SettingTrack 1 SRTrack 1 PSTrack 1 ISTrack 2 SRTrack 2 PSTrack 2 IS
Action only44.353.153.728.335.136.5
Action + Scene-grounding ETC56.269.673.640.748.147.3
Action + Task-oriented ETC59.371.974.831.442.350.7
Action + Action Planning ETC62.174.477.143.849.948.2
Action + Full ETC64.877.377.841.449.851.1

2. Robot ETC from different trajectories improves Stage 2 performance

We fix the Stage 2 action data to VLABench Track 1 and vary the source of ETC. VLABench Tracks 1 and 2 contain similar manipulation tasks involving different target objects. This lets us compare ETC from the same domain with and without overlap with the action-training trajectories. We expanded Track 2 ETC to a size comparable to Track 1 ETC. Both VLABench ETC runs used the original co-training configuration for 100k steps. All results are evaluated on Track 1.

SettingSRPSIS
Action-only44.353.153.7
Only OOD ETC63.273.674.8
LIBERO ETC62.472.973.1
VLABench ETC64.877.377.8

(a) Same domain, different trajectories. We use Track 2 ETC, which covers similar manipulation tasks with different target objects, as the sole VQA source while keeping Track 1 action data fixed.

(b) Different domain and scenes. We use LIBERO ETC as the VQA source while keeping Track 1 action data fixed, introducing supervision from a different embodied environment.

Overall, ETC can improve policy learning even without exact trajectory overlap with action data. Nevertheless, trajectory coupling provides additional benefits. Track 1 ETC achieves the best performance, highlighting the importance of scene and task alignment as well as action-aligned trajectory supervision. We have added the full results and analysis to Appendix G and summarized the key findings in Section 4.2.2 of the main paper, with a cross-reference to Appendix G.

AuthorsOpenReview ↗

Response part2

W2: Issue of In-domain trade-offs

We apologize that our original compositional generalization claim was too broad. The original VLABench experiment improved OOD understanding while reducing task success. The new VLABench results and the real-robot experiments show that OOD ETC can improve execution on the conditions it covers, with a trade-off in in-domain performance. We will narrow the claim and report in-domain and OOD execution results side by side.

1. Co-training with Track 2 ETC alone improves Track 2 execution. In the original experiment, Track 2 ETC made up about 6% of the ETC mixture. We expanded the Track 2 ETC dataset to a size comparable to Track 1 ETC and used it as the sole ETC source for Stage 2 co-training. All action demonstrations came from Track 1.

SettingTrack 1 SRTrack 1 PSTrack 1 ISTrack 2 SRTrack 2 PSTrack 2 IS
ID ETC only64.877.377.841.449.851.1
ID ETC + OOD ETC60.264.471.638.948.253.3
Only OOD ETC63.273.674.846.6752.0053.96

2. OOD ETC improves execution on targeted OOD conditions, with an in-domain trade-off. In the real-robot evaluation (18 ID and 12 OOD rollouts per task), OOD ETC improved OOD execution scores by 22.92 points on the flower task and 11.12 points on the coffee-bag task, while ID scores fell by 8.33 and 4.16 points, respectively. Together with the VLABench results, these findings show that OOD ETC can improve execution on the conditions it covers, with an observed trade-off in ID performance. We have revised Section 4.3.1 to clarify the condition-specific gains and in-domain trade-offs, and updated Table 2 to report the new VLABench results for both tracks.

I wish to thank the authors for their thorough and complete response to my review. The response answers some of my questions and feedback (but these were mostly clarification rather than significant weaknesses), and as such, I'll maintain my score since I feel it is a fair reflection of my view of the work's quality.

AuthorsOpenReview ↗

Thank You for Your Thoughtful Feedback

Thank you very much for your thoughtful follow-up and for acknowledging the completeness of our response. We appreciate your careful assessment and are glad that our clarifications addressed some of your questions. We sincerely appreciate your time and constructive feedback, which has helped us improve the clarity and presentation of our work.

Comments on the paper

2 comments

Dear Reviewers,

The authors have submitted their rebuttal and tried to address the concerns raised in your reviews. If you haven't already, we would appreciate it if you could take the time to read their responses and engage in the discussion, particularly to assess whether your concerns have been adequately addressed.

Thank you for your time and valuable contributions to the review process!

Best, AC

AuthorsOpenReview ↗

General Response.

We sincerely thank all reviewers for their thoughtful and constructive feedback, and for recognizing the importance and practical value of our work. We have carefully considered the concerns raised and conducted additional experiments and analyses to strengthen the empirical evidence, clarify the methodological contributions, and better characterize the applicability and limitations of ETC. Since several concerns overlap across reviewers, we summarize our major revisions by theme below. Detailed results and analyses are provided in our individual responses (Parts 1–5).

1. ETC Contribution and Trajectory Coupling (Reviewers oph8, igiB, Mxf2)

We conducted additional cross-trajectory and cross-dataset experiments to examine the role of trajectory coupling. Using expanded Track 2 ETC and LIBERO ETC while keeping Track 1 action data fixed, we achieved SRs of 63.2% and 62.4%, respectively, compared with 44.3% for action-only training. Track 1 ETC achieved the highest SR of 64.8%, suggesting that exact trajectory overlap is not necessary for ETC to benefit action learning, while better scene and task alignment provides additional gains. We further added compute-matched and supervision-controlled experiments, including trajectory-target removal and shuffled ETC, to isolate the contributions of supervision content and trajectory-derived information.

2. OOD Adaptation and Compositional Generalization (Reviewers oph8, igiB, Mxf2)

We expanded the VLABench and real-robot experiments to better evaluate OOD adaptation. On VLABench, using OOD ETC alone increased Track 2 SR from 41.4% to 46.67%, while the real-robot experiments demonstrated improved execution under targeted OOD conditions without directly training the policy on OOD action demonstrations. We also clarified the construction of human-derived, first-frame, and synthetic OOD ETC, and matched the training data volume across variants. Importantly, we acknowledge the observed ID/OOD trade-off and have refined our compositional-generalization claims to emphasize condition-specific improvements rather than universal generalization gains.

3. Bridging Mechanism and Knowledge Retention (Reviewers igiB, Mxf2)

We added LIBERO-Plus evaluations to further investigate the backbone-dependent effects of Distribution Bridging, particularly for Qwen2.5-VL-3B. We also expanded our representation analysis and clarified the evidence for embodied understanding retention. Accordingly, we narrowed our preservation claims to embodied knowledge and representations supported by our experiments, rather than claiming preservation of general multimodal capabilities. We further clarified the benchmark-specific nature of Stage 1 adaptation and the conditions under which the three-stage recipe is effective.

4. Experimental Rigor and Reproducibility (Reviewers igiB, Mxf2)

We corrected the coffee-bag OOD scoring inconsistency by applying a unified 0/25/100 rubric to both ID and OOD evaluations. We additionally reported reward-level counts and complete-task success rates, clarified rollout numbers and simulation evaluation protocols, and provided further details on ETC construction, manual verification, training ratios, and Stage 1 training duration. These revisions make the experimental methodology and results more transparent and reproducible.

5. Novelty and Manuscript Clarifications (Reviewer Mxf2)

We expanded the related work discussion to more clearly distinguish our approach from existing embodied VQA and multimodal co-training methods, particularly in terms of the dual-gap formulation and the three-stage training framework. We also clarified the contributions and limitations of benchmark-specific bridging, discussed backbone-dependent results, and improved the presentation of experimental settings and figures, including explicitly identifying the conceptual illustration in Figure 1(c).

We sincerely appreciate the reviewers' feedback, which has helped us strengthen the empirical evidence, sharpen the methodological contributions, and present a more rigorous and balanced account of ETC. We hope these additional experiments and clarifications address the concerns raised and provide a clearer understanding of both the contributions and limitations of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.