acceptodds
Under review as a conference paper at ICLR 2027

EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training

Abstract

Vision-Language-Action Models (VLAs) inherit their visual and linguistic capabilities from Vision-Language Models (VLMs), yet most VLAs are built from off-the-shelf VLMs that are not adapted to the embodied domain, limiting their downstream performance. In this work, we propose EmbodiedMidtrain to bridge the gap between VLMs and VLAs. We first characterize the data distribution gap between them, showing that VLA data occupy compact regions that are largely separated from the broader VLM distribution, while the degree of alignment varies substantially both across and within VLM data sources. Then, we build a mid-training data engine that leverages a lightweight learnable proximity estimator to select the most VLA-aligned candidates from a large VLM pool, and produce EmbodiedMidtrain-1M, a VLM dataset that we further use to mid-train the VLM before downstream VLA fine-tuning. Experiments on both simulated robot manipulation benchmarks and real-world tasks show consistent VLA performance gains over off-the-shelf checkpoints across VLM backbones and VLA action heads. These gains extend to downstream tasks not used to guide data selection. Further analysis reveals that mid-training provides a stronger initialization for VLA fine-tuning, with gains emerging from the earliest steps and widening throughout training. Moreover, the data engine captures both dataset-level and sample-level alignment signals, favoring spatial reasoning over text-centric tasks while preserving the diversity of the VLM data. We will release all code, data and models for future research.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, vision language models, mid-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_txKYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EmbodiedMidtrain, a data selection and mid-training pipeline for improving VLM initialization for VLA models. A lightweight classifier trained on frozen VLM features scores candidate vision-language samples by their similarity to robot training data. The method selects 1.28 million samples for VLM mid-training, followed by robot action fine-tuning. Experiments on CALVIN, SimplerEnv, LIBERO-10, and real-world tasks show improvements over the original VLM initializations, with additional analyses of transfer across backbones and action heads, selection methods, and selected data.

Critical strengths

1.The paper studies a meaningful research problem. Improving the transfer from VLMs to VLAs is an important direction in embodied intelligence. The proposed data selection and mid-training pipeline offers a practical way to improve VLM initialization for robot action learning.

2.The experimental analysis is broadly thorough, combining quantitative evaluations with visualizations. Results across multiple benchmarks, together with ablation studies and qualitative examples, provide useful evidence for the effectiveness of the proposed pipeline.

Critical weaknesses

1.The reduction in the data distribution gap is not directly verified after selection. The paper uses MMD and t-SNE to show the gap between the original vision-language data and robot data, but does not report the corresponding comparison after selection. The higher classifier scores in Figure 4 are an expected result of selecting high-scoring samples and do not independently show that the gap has narrowed. I suggest comparing MMD before and after selection in the same feature space, with t-SNE visualizations as supporting evidence.

2.The data sizes and sampling procedures in the ablation experiments are unclear. Although the paper reports the same mid-training compute budget, it does not specify whether the random-selection baseline uses a fixed subset of 1.28 million distinct samples, matching the selected dataset. Please also clarify whether the alternative selection methods in Table 2 each select 1.28 million samples, and report the number of distinct samples used in each setting and how repeated sampling is handled.

Questions for authors

Could the authors respond to the two concerns raised in the weaknesses section?

4 comments

Thanks for your feedback on our paper. For your concerns and questions, we would like to respond as follows.

Q1. Distribution gap after data selection.

A1. We computed the MMD between each VLM source and each VLA dataset before and after selection. The table reports the change in MMD after selection, where a negative value means a narrower gap.

VLA datasetCC-12MLAION-400MLLaVA-Instruct-665kRefSpatialRoboPoint
LIBERO-0.022-0.031-0.0720.000-0.001
Bridge-V2-0.025-0.040-0.079-0.018-0.001
Calvin-0.018-0.028-0.061-0.002-0.002
Collected real-world data-0.015-0.019-0.0450.000-0.001

The gap narrows for most pairs, showing that selection moves the VLM data toward the VLA domain under a measure that does not depend on the classifier score. The reduction is largest for the general sources LLaVA-Instruct-665k, LAION-400M, and CC-12M, consistent with Appendix K, where selection within general sources gives larger downstream gains than selection within embodied-oriented sources. We have added these results and the t-SNE visualization after selection to the revision.

Q2. Data sizes and sampling in the ablation experiments.

A2. All settings in Table 2 use the same data size. Random selection and the three alternative proximity measurements each use a fixed subset of 1.28M distinct samples, drawn without repetition from the same candidate data as our method. The settings differ only in the 1.28M samples they select. Mid-training runs for 5,000 steps at a batch size of 256, which is one pass over the 1.28M samples, and no sample is repeated. We will state these details in the revision.

AuthorsOpenReview ↗

We look forward to hearing from you

As the discussion period is coming to an end, we would greatly appreciate any further feedback you may have. We would be happy to address any further questions or concerns about our work.

Thank you for the detailed responses and for taking the time to address my concerns. The additional MMD analysis and clarification of the data sizes and sampling procedures have largely resolved my questions.

We sincerely thank the reviewer for the feedback and for taking the time to engage with our paper. We are glad that the additional experiments and clarifications addressed your concerns.

Reviewer_vHvZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EmbodiedMidtrain, a mid-training framework that bridges the gap between general-purpose VLMs and VLAs by aligning their data distributions. The authors first characterize the VLM-VLA distribution gap and then introduce a lightweight "learnable proximity estimator" to select VLA-aligned samples from a large VLM data pool for mid-training. Experiments on simulated and real-world manipulation tasks demonstrate consistent performance improvements across different VLM backbones, action heads, and downstream tasks.

Critical strengths

1."Well-motivated problem": The paper provides both quantitative and qualitative analyses of the distribution gap between VLM and VLA data, revealing differences across and within datasets and motivating sample-level data selection. 2."Effective data selection and dataset contribution":The proposed sample-level selection strategy identifies VLA-aligned samples from diverse VLM sources, resulting in the curated EmbodiedMidtrain-1M dataset that improves downstream VLA performance. 3."Comprehensive experiments and analysis ":The paper demonstrates consistent improvements across multiple VLM backbones, benchmarks, and action heads. Extensive ablations on proximity measurements, selection strategies, and data source contributions, together with training dynamics analysis, provide valuable insights into the effectiveness of mid-training.

Critical weaknesses

1."Potential contribution of data diversity":The selected dataset exhibits a higher diversity score than the original VLM data pool (1.93 vs. 1.85). Could the downstream performance gains be partially attributed to increased data diversity, rather than solely to improved proximity to the VLA distribution? 2."Robot data efficiency":The paper demonstrates consistent improvements in downstream VLA performance, but it remains unclear how these benefits vary with the amount of available robot training data. Would EmbodiedMidtrain provide greater advantages in few-shot settings? Experiments with different fractions of robot demonstrations could further clarify whether mid-training improves data efficiency. 3."Lack of quantitative behavioral analysis":While the paper provides qualitative examples of improved manipulation behaviors, it lacks a systematic quantitative analysis of which specific capabilities benefit from mid-training. A phase-level evaluation of approach, grasping, transport, and placement could provide deeper insights into how the selected VLM data contribute to downstream VLA performance.

Questions for authors

Questions: Please see Weaknesses 1–3 for my main questions and concerns.

2 comments

Thanks for your feedback on our paper. For your concerns and questions, we would like to respond as follows.

Q1. Potential contribution of data diversity.

A1. To separate diversity from proximity, we built a diversity selector that does not use the proximity score. It samples evenly across k-means clusters of the VLM features and selects 1.28M samples with a diversity of 1.93, matching the size and diversity of our selected data. We mid-trained InternVL3.5-1B on these data with the same setup and three training seeds.

InitializationCalvinSimplerEnvLIBERO-10
InternVL3.5-1B3.097 ± 0.16035.4 ± 4.839.1 ± 5.5
+ Diversity selector3.131 ± 0.10137.9 ± 3.743.0 ± 2.4
+ EmbodiedMidtrain3.646 ± 0.06551.4 ± 4.354.6 ± 2.8

With matched diversity, the diversity selector stays close to the off-the-shelf model, with differences within the seed variation on all three benchmarks. EmbodiedMidtrain outperforms the diversity selector on all three benchmarks, showing that the gains come from proximity to the VLA domain and not from higher diversity.

Q2. Robot data efficiency.

A2. We fine-tuned the VLA on 100%, 50%, and 25% of the Calvin robot demonstrations, with and without EmbodiedMidtrain, using three training seeds for each setting.

InitializationCalvin with 100% robot demonstrationsCalvin with 50% robot demonstrationsCalvin with 25% robot demonstrations
InternVL3.5-1B3.097 ± 0.1602.756 ± 0.0822.356 ± 0.132
+ EmbodiedMidtrain3.646 ± 0.0653.052 ± 0.1022.695 ± 0.173

EmbodiedMidtrain improves the VLA at every data fraction, and the gain holds in the low-data settings and is largest with the full data, showing that mid-training keeps its benefit as robot data grows. We will add this experiment to the revision.

Q3. Quantitative phase-level analysis.

A3. We labeled each of the 100 real-robot rollouts in Sec. 5.3 by the furthest phase it reached, using the recorded camera views and the gripper state. The rollouts cover five tasks with 10 rollouts per task for each model. The table reports the number of rollouts that reached each phase.

Phase reachedInternVL3.5-1B+ EmbodiedMidtrain
Approach the target object50/5050/50
Grasp the object43/5047/50
Transport it to the destination42/5047/50
Complete the placement27/5036/50

Both models approach the target object in every rollout, and a grasped object is carried to the destination in all but one rollout, showing that the two models do not differ in approach or in transport after a grasp. The gain comes from grasping and placement. The mid-trained model grasps the object in more rollouts, and it completes the placement in 77% of the rollouts that reach the destination, against 64% for the off-the-shelf model. The gains are largest on the two tasks that need precise placement. Placing the cube into the cup improves from 3 to 6 successes out of 10, and stacking the cubes improves from 2 to 4. These phases depend on locating the object and the target precisely, consistent with the preference of our selected data for spatial grounding and reasoning shown in Sec. 6.3. We will add this analysis to the revision.

AuthorsOpenReview ↗

We look forward to hearing from you

As the discussion period is coming to an end, we would greatly appreciate any further feedback you may have. We would be happy to address any further questions or concerns about our work.

Reviewer_fwwdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the distributional gap between general VLM data and VLA robot trajectories and proposes EmbodiedMidtrain to improve VLM initialization for robot-policy learning. The method trains a linear domain classifier on frozen VLM features to select vision-language examples that are more closely aligned with VLA data for mid-training before downstream VLA fine-tuning. Experiments with two VLM backbones report improvements on three simulation benchmarks and five real-world manipulation tasks. Additional studies examine data-selection strategies, transfer across benchmarks and action heads, and comparisons under matched computational budgets.

Critical strengths
  • The experiments hold the downstream training recipe and robot data fixed while varying VLM initialization. Comparisons with random selection, alternative selection strategies, and FLOP-matched extended fine-tuning provide evidence that the selected data contribute to the gains beyond additional computation alone.
  • The selected data improve a second VLM backbone without re-selection and benefit three action-head designs and real-world manipulation tasks. Excluding LIBERO from the selector’s target data retains about 95% of the downstream gain on that benchmark, suggesting that the selection captures properties useful beyond the benchmarks used to guide it.
Critical weaknesses
  • Limited methodological novelty. Domain-classifier-based data selection and its density-ratio interpretation are established techniques, with related work such as DSIR cited by the paper. The contribution is mainly their careful application to VLM-to-VLA adaptation. The distribution-gap analysis is largely expected and offers limited explanation of which data properties improve robot learning. Together, these limit the conceptual advance for a selective venue such as ICLR.
  • Insufficiently validated selection signal and uncertain practical significance. The last-token classifier may exploit differences in language, formatting, or image style rather than action-relevant content, and the analyses do not distinguish these explanations. With no training-seed uncertainty reported, the reliability of smaller gains is hard to assess. Most results use the lightweight VLM4VLA pipeline, where absolute performance remains modest. Although alternative-head experiments demonstrate transfer, evidence in stronger settings has limited benchmark coverage and lacks reported training-seed uncertainty. The robustness of the selection advantage and its incremental value in stronger VLA pipelines therefore remain uncertain.
Questions for authors
  • How are VLA and VLM samples formatted when extracting classifier features? Does selection remain effective when text-format differences are controlled?
  • How many independent training runs support the reported results? Can you provide uncertainty estimates for the gains over the original initialization and the strongest alternative selector?
  • Beyond the reported alternative-head experiments, is there additional evidence that the gains persist across more benchmarks with flow-matching heads or after large-scale robot pretraining? If not, please clarify the scope of the conclusions.
6 comments

Thanks for your feedback on our paper. For your concerns and questions, we would like to respond as follows.

Q1. The methodological novelty of EmbodiedMidtrain.

A1. Our novelty lies in data selection across heterogeneous multimodal data, which is a different problem from data selection for language models. In DSIR, the candidate and target data are both text in the same format, and they can be compared directly with n-gram features. In our setting, VLM data are images paired with questions and text answers, while VLA data are robot observations paired with instructions and continuous actions. The two differ in format, output space, and training objective. EmbodiedMidtrain introduces a way to compare them, with the goal of bridging the data distributions of VLMs and VLAs for better downstream robot manipulation performance. Bridging this gap is not straightforward. Table 2 in the paper shows that direct alternatives such as feature distance and perplexity-based scores do not give consistent gains. Our design is a learned estimator that scores each sample in the joint vision-language representation space of the frozen VLM, and it is the one that works consistently.

Moreover, Our distribution analysis drives this design and identifies the data that help bridge the gap between VLM and VLA. Sec. 3 shows that the degree of alignment differs within each VLM source. This finding motivates sample-level selection, and Appendix K also shows that it outperforms dataset-level selection. Sec. 5 and Appendix H demonstrate that the gains transfer across VLM backbones, action heads, and target tasks.

Q2. Validation of the selection signal.

A2. For feature extraction, VLA and VLM samples use the same format. Each VLA sample is built from its image observation and language task goal, and written in the same VQA template as the VLM datasets. The estimator therefore sees no template difference between the two domains. Selection remains effective when the text format is controlled. We restrict the candidate data to the samples that share the same prompt, and compare proximity-based selection with random selection of the same number of samples, under the same mid-training and fine-tuning setup on InternVL3.5-1B with three training seeds. The table reports the mean and standard deviation.

Mid-training data from the same-prompt subsetSimplerEnvLIBERO-10
Random selection38.5 ± 3.941.6 ± 4.7
Proximity-based selection44.8 ± 4.147.8 ± 3.6

Proximity-based selection outperforms random selection on both benchmarks while the prompt is the same for all candidates, showing that the gains do not come from prompt format. We will add the feature-extraction format and this experiment to the appendix in the revision.

Q3. Training-seed uncertainty analysis.

A3. We ran 3 independent training seeds on InternVL3.5-1B for the off-the-shelf initialization, our method, and delta perplexity, the strongest alternative selector. The table reports the mean and standard deviation.

InitializationCalvinSimplerEnvLIBERO-10
InternVL3.5-1B3.097 ± 0.16035.4 ± 4.839.1 ± 5.5
+ Delta perplexity selector3.238 ± 0.30939.9 ± 2.652.7 ± 1.9
+ EmbodiedMidtrain3.646 ± 0.06551.4 ± 4.354.6 ± 2.8

EmbodiedMidtrain outperforms the off-the-shelf initialization on every training seed, with gains much larger than the seed variation. Compared with delta perplexity, our estimator is better on Calvin and SimplerEnv and comparable on LIBERO-10, showing that it is the only selector that is strong on all three benchmarks. We will add the multi-seed results to the revision.

Q4. Robustness of the gains in flow-matching head VLA pipelines.

A4. We ran three independent training seeds with the flow-matching action head on all three benchmarks, using InternVL3.5-1B. The table reports the mean and standard deviation.

Initialization with flow-matching headCalvinSimplerEnvLIBERO-10
InternVL3.5-1B3.347 ± 0.07028.1 ± 5.588.8 ± 0.9
+ EmbodiedMidtrain3.648 ± 0.12141.0 ± 6.392.5 ± 1.4

EmbodiedMidtrain improves the flow-matching VLA on all three benchmarks, with gains larger than the seed variation, showing that the gains hold with a flow-matching action head. Our conclusions concern the VLM initialization under direct VLA fine-tuning, across VLM backbones, action heads, and benchmarks. We will update these results and state this scope in the revision.

AuthorsOpenReview ↗

We look forward to hearing from you

As the discussion period is coming to an end, we would greatly appreciate any further feedback you may have. We would be happy to address any further questions or concerns about our work.

Reviewer_fwwdOpenReview ↗

One more clarification question

Thank you for the thorough response and the new multi-seed and prompt-controlled experiments, which address most of my technical concerns. One brief clarification remains regarding A1 and A2: since VLA samples are represented only by image & instruction pairs without actions, the proximity selector appears to operate on the visual language input distribution rather than on output spaces or training objectives. Could you clarify how this aligns with the novelty positioning in A1? With this clarified, I would be glad to raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.