acceptodds
Desk rejected at ICLR 2027

Rethinking Vision-Language-Action Model Scaling: Alignment, Mixture, and Regularization

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision–Language–Action (VLA) models show strong promise for generalist robot control. However, it remains unclear whether—and under what conditions—the standard “scale data" recipe translates to robotics, given that robotic training data is inherently heterogeneous across embodiments, sensors, and action spaces. We present a systematic, controlled study of VLA scaling that revisits core training choices for pretraining across diverse robots. Using a representative VLA framework combining a vision–language backbone with flow-matching, we ablate key design decisions under matched conditions in extensive simulation and real-robot experiments. To improve the reliability of real-world evaluation, we introduce a Grouped Blind Ensemble protocol that blinds operators to model identity and randomizes evaluation order within groups. Our analysis targets three dimensions of VLA scaling: (1) Physical alignment: Using SE(3) as a shared action space, we find that end-effector (EEF)-relative actions support positive cross-embodiment transfer. (2) Embodiment mixture: Adding heterogeneous robot datasets can reduce downstream transfer performance, even with the same training budget. (3) Training regularization: We observe that common strategies, such as sensory dropout and multi-stage training, do not consistently improve performance at scale. Together, this study challenges common assumptions about embodied scaling and provides practical, evidence-based guidance for training large-scale VLA policies.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Cross-Embodiment Transfer, Action Space Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HN3TOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a systematic empirical study of Vision-Language-Action (VLA) model scaling with heterogeneous robotic data, focusing on three key aspects: physical alignment, embodiment mixture, and training regularization. Using a representative VLA architecture combining a vision-language backbone with flow-matching-based action generation, the authors investigate how action representations, cross-embodiment data mixtures, and training strategies affect downstream transfer performance under controlled training budgets. The paper also introduces a Grouped Blind Ensemble evaluation protocol to reduce operator bias in real-robot experiments. Experiments on LIBERO, RoboCasa, LIBERO-Plus, RoboTwin, and a Franka Panda robot suggest that EEF-relative action representations facilitate cross-embodiment transfer, while adding heterogeneous datasets does not consistently improve performance and can introduce negative transfer. The results further indicate that common regularization techniques, including sensory dropout and multi-stage training, provide limited or inconsistent benefits. Overall, the study offers empirical insights into the importance of physical alignment and data composition in VLA pretraining.

Critical strengths
  1. Systematic and practically relevant empirical study. The paper investigates several important design choices in large-scale VLA pretraining, including action representations, heterogeneous data mixtures, and training regularization. The controlled comparisons provide useful empirical evidence that cross-embodiment transfer depends not only on data diversity but also on how actions are represented and datasets are combined. These findings offer practical guidance for designing generalist VLA training pipelines.

  2. Comprehensive evaluation with real-robot validation. The authors conduct experiments across multiple simulation benchmarks, including LIBERO, RoboCasa, LIBERO-Plus, and RoboTwin, and further validate their findings on a Franka Panda robot. The proposed Grouped Blind Ensemble protocol, which anonymizes model identities and randomizes evaluation order, is a valuable effort to reduce operator bias in real-world evaluations. The combination of simulation and physical experiments strengthens the empirical contribution of the work.

Critical weaknesses
  1. Insufficient causal evidence for the scaling conclusions. Although the paper investigates VLA scaling, the data-mixture experiments primarily compare different dataset compositions under a fixed training budget rather than systematically varying data scale or compute. Moreover, adding heterogeneous datasets also changes the sampling distribution and potentially the data quality, making it difficult to attribute the observed performance degradation specifically to embodiment heterogeneity. This limits the strength and generalizability of the paper's central conclusions about VLA scaling.

  2. Limited real-robot evidence for the main claims. While the inclusion of real-robot experiments is valuable, the evaluation is limited to a single Franka Panda platform, four tasks, and ten trials per task per model. The reported results also do not clearly establish the superiority of EEF-relative actions over world-relative actions in real-world settings (44% vs. 45% average score). Without stronger statistical analysis or broader physical validation, it remains unclear how robustly the findings generalize across robot embodiments and real-world conditions.

Questions for authors
  1. Scaling conclusions versus data-mixture effects. The paper argues that simply adding heterogeneous robotic data can negatively affect VLA scaling. However, the experiments in Section 5.2 keep the number of training steps, batch size, and total sampled examples fixed while changing the dataset composition. Could the authors clarify how these results distinguish negative transfer caused by embodiment heterogeneity from effects of data distribution, sampling ratios, or data quality? In particular, would the main conclusions still hold under a controlled scaling experiment that increases the amount of training data and compute, rather than replacing part of the training distribution? This distinction is important for determining whether the findings reflect a general VLA scaling limitation or a specific data-mixture effect.

  2. Evidence for the superiority of EEF-relative actions. The paper identifies EEF-relative actions as a preferred representation for cross-embodiment transfer. However, Figure 4 shows that EEF-relative and world-relative actions achieve nearly identical average scores in real-robot evaluation (44% versus 45%). Could the authors explain this discrepancy between the simulation transfer results and real-world performance? What evidence establishes that the advantage comes specifically from the end-effector coordinate frame rather than from using start-relative actions instead of step-wise delta actions? A clearer decomposition of these factors would substantially strengthen the physical alignment claim.

  3. Robustness and statistical reliability of real-robot experiments. The real-world evaluation is conducted on a single Franka Panda platform with four tasks and ten trials per task per model. Although the Grouped Blind Ensemble protocol is a valuable effort to reduce operator bias, could the authors provide uncertainty estimates, such as confidence intervals or paired statistical comparisons, for the key differences between action representations and pretraining mixtures? In particular, are the performance differences between D1 and the other mixtures statistically distinguishable? This would help determine whether the reported real-world trends are robust rather than attributable to evaluation variability.

  4. Scope of the regularization conclusions. The paper reports that sensory dropout and two-stage training do not consistently improve performance. Could the authors clarify whether these conclusions are supported across different pretraining mixtures, training budgets, or independent runs, rather than primarily under the selected training configuration? This would help establish whether these are general findings about VLA pretraining or configuration-dependent observations.

Overall assessment and importance of the questions.

I consider Questions 1–3 particularly important because they directly concern the validity and empirical support of the paper's main conclusions. In particular, distinguishing genuine scaling limitations from data-mixture effects, establishing the specific contribution of EEF-relative action representations, and demonstrating the statistical reliability of the real-robot results are essential for assessing the significance of this study.

I appreciate the practical relevance and breadth of the experiments, and I would welcome careful, evidence-based responses to these questions. The authors' responses, particularly to Questions 1–3, will substantially influence my final recommendation regarding acceptance.

1 comment

Thank you for the thorough and constructive review.

Q1. Heterogeneity vs. data distribution, sampling and quality. We compare data compositions at the same budget (240k steps × batch 256) so that the effect of the data is not mixed with that of longer training. Under this budget, each mixture sees the following amount of OXE data (Table 3, Fig. 5):

MixtureOXE samples seenLIBERO (frozen)LIBERO (unfrozen)Real robot
D1 (OXE only)61.4M77.386.448
D2 (+ Real EEF)27.0M73.884.043
D3 (+ Sim EEF)19.8M72.183.739
D4 (+ Joint)16.1M75.184.545

Two observations separate heterogeneity from dilution and data quality. First, adding joint-space data (D3 → D4) further reduces the OXE samples seen but does not lower performance; D4 also scores higher than D3 on RoboCasa (50.0 vs. 49.6). Because joint-space data does not share the SE(3) action space with OXE, this suggests that the gap to D1 comes mainly from conflicts within the SE(3) space rather than from dilution alone. For your scaling question, this means the gap is unlikely to disappear simply by training longer, since less exposure to OXE did not by itself lower performance. Second, the largest drop comes from adding real-world EEF data from other robots (D2: AgiBot, RoboMind), so it is not caused by the quality of simulated data; Sec. 5.2 discusses possible sources, such as differing TCP frame definitions and control frequencies (3–30 Hz) across robots.

Q2. Frame vs. reference point. Thank you for suggesting this decomposition. The four parameterizations form a 2×2 design, so the two factors can be separated with existing results. The table shows the pretraining gain over training from scratch (Tables 1–2):

WorldEEF
LIBERO, Rel−0.9+2.6
LIBERO, Delta−0.5+2.4
RoboCasa, Rel+2.5+4.9
RoboCasa, Delta−0.1+3.4

Moving from the World frame to the EEF frame increases the gain by 2.4–3.5 points in every row, whereas switching between Rel and Delta within a frame changes it by −0.4 to +2.6 points, with no consistent pattern. The consistent transfer benefit therefore comes from the EEF frame. The reference point matters for a different reason: on the real robot, start-relative actions run stably while step-delta actions do not (Sec. 5.1). The similar real-robot scores of World-Rel and EEF-Rel (45 vs. 44) match what we observe on LIBERO, which, like our real-robot setup, uses a single arm with a fixed base and cameras: there the two also reach similar success after pretraining (83.5 vs. 84.5), because World-Rel already performs well when trained within a fixed setup (Sec. 5.1). The advantage of EEF-Rel lies in how much it gains from heterogeneous pretraining, as the table shows.

Q3. Statistical reliability. D1 outperforms the full mixture D4 in all 14 downstream comparisons we report:

Success (%)ScratchD4D1
LIBERO 5-shot, frozen / unfrozen VLM66.9 / 81.975.1 / 84.577.3 / 86.4
RoboCasa45.150.054.7
LIBERO-Plus, 5 / 50-shot16.0 / 22.113.7 / 29.122.8 / 46.4
RoboTwin4.114.115.2
LIBERO, 1 / 5 / 10 / 25 / 50-shot27.9 / 55.4 / 58.6 / 78.5 / 81.831.9 / 64.5 / 71.3 / 85.9 / 90.537.8 / 72.8 / 76.6 / 87.6 / 92.2
LIBERO, co-training / LoRA59.0 / 5.561.2 / 16.972.2 / 33.7
Real robot314548

A two-sided sign test over these 14 paired comparisons gives p = 1.2 × 10⁻⁴, so the difference is statistically distinguishable. D1 also outperforms D2 and D3 on LIBERO and on the real robot (Q1 table). For the action representations, the frame effect holds in all four rows of the Q2 table.

Q4. Scope of the regularization findings. They were measured in our main configuration with the full data mixture, which is the regime of interest: whether these strategies still help once the corpus is large and diverse. In this setting, removing a regularizer never lowers the average (no state masking +0.7, no view dropout +1.1, Stage 2 Only +1.3), and strengthening it never raises it (state masking 0.5: −0.5; view dropout 0.5: ±0.0).

We hope these responses address your concerns. Thank you again for your thoughtful review.

Reviewer_M5YYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies heterogeneous VLA pretraining through a shared flow-matching testbed. It compares world-frame and end-effector (EEF) action representations, data mixtures under a fixed pretraining budget, sensory dropout and staged optimization. Evaluation covers LIBERO, RoboCasa, additional adaptation settings, and four real-robot tasks with anonymized policies, randomized trial order and subtask scores. EEF-relative actions support positive transfer, while adding data sources or training heuristics does not consistently improve performance in these settings.

Critical strengths
  1. The shared testbed provides useful evidence about action representations and fixed-budget data mixtures. EEF-relative pretraining improves RoboCasa from 45.1 to 50.0, while OXE-only pretraining outperforms the full mixture, 54.7 versus 50.0. The mixture ordering also holds in additional adaptation settings.

  2. The randomized, anonymized real-robot protocol gives other groups a concrete evaluation procedure to reuse. It specifies policy aliases, trial order and subtask scoring, although the reduction in operator bias is not directly measured.

Critical weaknesses
  1. The mixture results do not isolate alignment as the main cause of performance changes. Reduced OXE exposure, sampling and source quality remain plausible explanations. This limits the concluding causal claim, while leaving the fixed-budget comparison useful.

  2. The 0.7–1.3-point advantages in Table 6 lack variability estimates, and the update budget for "Stage 2 Only" is unclear (p. 9; Appendix C, p. 15). The step-delta execution details also need clarification before interpreting both variants' zero real-robot scores (Fig. 4, p. 6).

Questions for authors
  1. How much OXE data does each mixture actually see during pretraining?

  2. How are step-delta predictions converted into robot commands in the real-robot experiments?

2 comments

Thank you for the precise questions, and for noting that the fixed-budget comparison and the evaluation protocol are useful.

Q1. OXE data seen by each mixture. All mixtures are trained for 240k steps at batch size 256 (61.4M samples). The table lists the OXE data seen by each mixture together with its downstream performance (Table 3, Fig. 5):

MixturePoolOXE shareOXE samples seenLIBERO (frozen)LIBERO (unfrozen)Real robot
D1 (OXE only)47.7M100%61.4M77.386.448
D2 (+ Real EEF)108.5M44%27.0M73.884.043
D3 (+ Sim EEF)148.1M32%19.8M72.183.739
D4 (+ Joint)182.5M26%16.1M75.184.545

W1. Alignment vs. OXE exposure, sampling and source quality. Thank you for raising this; two observations in the table bear on it. First, adding joint-space data (D3 → D4) further reduces OXE exposure but does not lower performance; D4 also scores higher than D3 on RoboCasa (50.0 vs. 49.6). Since joint-space data does not share the SE(3) action space with OXE, this points to conflicts within the SE(3) space, rather than reduced OXE exposure alone, as the main cause of the gap to D1. Second, the largest drop comes from adding real-world EEF data from other robots (D2: AgiBot, RoboMind), so it is not caused by the quality of simulated data; Sec. 5.2 discusses possible sources, such as differing TCP frame definitions and control frequencies (3–30 Hz) across robots.

W2a. Table 6. Thank you for pointing this out. We agree that these small differences should not be read as improvements, and we have revised the wording in Sec. 5.3 accordingly. Our conclusion only requires that the regularizers do not help, which Table 6 shows consistently:

Variant (baseline: 84.5)Avg.Change
No state masking85.2+0.7
No view dropout85.6+1.1
Stage 2 Only (no Stage 1)85.8+1.3
State masking 0.584.0−0.5
View dropout 0.584.5±0.0

Removing a regularizer never lowers the average, and strengthening it never raises it.

W2b. "Stage 2 Only" budget. It is trained for 240k steps, the same total as the two-stage schedule (40k + 200k). This is now stated in Appendix C.

Q2. Execution of step-delta predictions. All four parameterizations share the same pipeline: the predictions are converted into absolute end-effector pose targets, the first 8 steps of each chunk are executed before re-planning, and a Cartesian impedance controller tracks the targets. The only difference is the reference. Start-relative targets are all anchored to the chunk start, whereas step-delta targets accumulate the per-step increments, so an error in one step carries over to the rest of the chunk. These details are now in Appendix C.

We hope these responses address your questions. Thank you again for your careful review.

Thank you for the clarifications. Q1, Q2, and the Stage 2 budget are now addressed.

Reviewer_psWqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates when scaling heterogeneous robot data improves vision-language-action (VLA) models through controlled experiments on LIBERO, RoboCasa, and several real-world tasks (on Franka). Using a pretrained vision-language backbone coupled with a flow-matching action expert, the authors systematically vary action representations, pretraining dataset mixtures, and regularization strategies. They find that end-effector-relative actions support cross-embodiment transfer, adding heterogeneous datasets can reduce performance under a fixed training budget, and regularization approaches like sensory dropout and staged training offer no consistent benefit. The paper also introduces a Grouped Blind Ensemble evaluation protocol that anonymizes policies and randomizes trial order to reduce operator bias. Its main contribution is empirical guidance on action alignment, data composition, and training choices for VLA scaling.

Critical strengths
  • Controlled, practically relevant ablations: Systematically varies action representations, dataset mixtures, and training strategies, keeping training steps and batch size fixed across mixture experiments.
  • Evaluation across simulation and hardware: Tests on LIBERO, RoboCasa, and four real-world manipulation tasks, providing complementary evidence about transfer and control performance
  • Useful findings that challenge common assumptions: Shows that broader data mixtures and additional regularization do not automatically improve performance, offering actionable guidance for VLA training.
Critical weaknesses
  • Unclear coverage of generalization evaluation: The LIBERO-Plus results lack details about the tasks and perturbations evaluated. Clarifying this coverage and providing broader, disaggregated results would help establish whether the findings generalize beyond standard LIBERO. Also, since LIBERO is quite overfit, consider doing all tasks of LIBERO-Plus if not done already or LIBERO-Pro (as an alternative)
  • Dependence on one VLA architecture: The conclusions come from a single backbone and flow-matching architecture. Reproducing the central findings with at least one additional VLA would strengthen their broader applicability.
  • Dependence on training budget: The dataset mixtures are compared under a single fixed training budget. Evaluating across multiple budgets would help determine whether larger mixtures consistently underperform or simply require more training to realize their benefits.
  • Small real-world evaluation budge: Each real-world task uses only ten trials. Increasing this to at least twenty trials per task and model would make the real-world comparisons more convincing.
Questions for authors

My weaknesses are presented as very clear potential avenues for the authors to improve the scores. I have no further clarification requests beyond potentially strengthening the paper in the ways I suggested above.

1 comment

Thank you for the constructive review and the concrete suggestions.

W1. LIBERO-Plus coverage and saturation. We use the full LIBERO-Plus benchmark under its standard setting (Fei et al., 2025), which covers seven perturbation dimensions: camera viewpoints, robot initial states, language instructions, lighting, background textures, sensor noise and object layout. The protocol is now described in Appendix E. Our LIBERO ablations use a 5-shot protocol, under which LIBERO is far from saturated: success rates range from 66% to 86%, compared with 94–98% when fine-tuning with 50 demonstrations. The mixture findings also hold beyond LIBERO (Tables 4–5, Fig. 5):

Success (%)LIBERO-Plus 5-shotLIBERO-Plus 50-shotRoboTwinRoboCasaReal robot
Scratch16.022.14.145.131
D4 (full mixture)13.729.114.150.045
D1 (OXE only)22.846.415.254.748

The action-space finding also holds on RoboCasa, where EEF-Rel gives the largest pretraining gain among the four parameterizations (+4.9, Table 2).

W2. Single architecture. Thank you for the suggestion. Since each ablation requires its own large-scale pretraining run, we focused on one architecture and kept it fixed so that every difference can be attributed to a training choice. We chose the mainstream design of a pretrained VLM with a flow-matching action expert, as used in π0/π0.5 and GR00T N1, and our model is competitive with these policies (Table 7, 50-shot fine-tuning):

Avg. success (%)GR00T-N1GR00T-N1.6†π0π0.5Ours
LIBERO93.997.094.496.997.8
RoboCasa36.066.642.441.450.0

† GR00T-N1.6 includes RoboCasa in its pretraining data. We therefore believe the findings are representative of this widely used family of VLAs.

W3. Training budget. We compare the mixtures at the same pretraining budget on purpose: this answers the practical question of whether adding datasets helps for a given amount of compute, without mixing in the effect of longer training. Within this budget, adding joint-space data (D3 → D4) further reduces the OXE samples seen but does not lower performance:

OXE samples seenLIBERO (frozen)LIBERO (unfrozen)RoboCasaReal robot
D161.4M77.386.454.748
D319.8M72.183.749.639
D416.1M75.184.550.045

This suggests that the gap to D1 is not simply due to less training per sample. The ordering is also stable across downstream data budgets: in separate fine-tuning runs with 1 to 50 demonstrations per task, D1 > D4 > Scratch at every budget (Table 10):

LIBERO success (%)1-shot5-shot10-shot25-shot50-shot
Scratch27.955.458.678.581.8
D431.964.571.385.990.5
D137.872.876.687.692.2

W4. Number of real-robot trials. Our real-robot study covers many models: four action parameterizations and five pretraining settings, each evaluated on four multi-step tasks under the blinded protocol, which amounts to several hundred rollouts in randomized, anonymized order. To keep this feasible, we used ten trials per task and model and scored each trial by sub-task (3–5 graded steps per task, Appendix D) to capture partial progress. The real-robot results agree with simulation (Fig. 5):

Score (%)Stack BowlsPick to DrawerWipe BoardWater PlantAvg.
Scratch2724254831
D42324557745
D13030557748

All pretrained models outperform training from scratch on average, D1 scores highest, and D1 outperforms Scratch on every task.

We hope these responses address your concerns. Thank you again for your helpful suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.