- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a systematic empirical study of Vision-Language-Action (VLA) model scaling with heterogeneous robotic data, focusing on three key aspects: physical alignment, embodiment mixture, and training regularization. Using a representative VLA architecture combining a vision-language backbone with flow-matching-based action generation, the authors investigate how action representations, cross-embodiment data mixtures, and training strategies affect downstream transfer performance under controlled training budgets. The paper also introduces a Grouped Blind Ensemble evaluation protocol to reduce operator bias in real-robot experiments. Experiments on LIBERO, RoboCasa, LIBERO-Plus, RoboTwin, and a Franka Panda robot suggest that EEF-relative action representations facilitate cross-embodiment transfer, while adding heterogeneous datasets does not consistently improve performance and can introduce negative transfer. The results further indicate that common regularization techniques, including sensory dropout and multi-stage training, provide limited or inconsistent benefits. Overall, the study offers empirical insights into the importance of physical alignment and data composition in VLA pretraining.
Critical strengths
-
Systematic and practically relevant empirical study. The paper investigates several important design choices in large-scale VLA pretraining, including action representations, heterogeneous data mixtures, and training regularization. The controlled comparisons provide useful empirical evidence that cross-embodiment transfer depends not only on data diversity but also on how actions are represented and datasets are combined. These findings offer practical guidance for designing generalist VLA training pipelines.
-
Comprehensive evaluation with real-robot validation. The authors conduct experiments across multiple simulation benchmarks, including LIBERO, RoboCasa, LIBERO-Plus, and RoboTwin, and further validate their findings on a Franka Panda robot. The proposed Grouped Blind Ensemble protocol, which anonymizes model identities and randomizes evaluation order, is a valuable effort to reduce operator bias in real-world evaluations. The combination of simulation and physical experiments strengthens the empirical contribution of the work.
Critical weaknesses
-
Insufficient causal evidence for the scaling conclusions. Although the paper investigates VLA scaling, the data-mixture experiments primarily compare different dataset compositions under a fixed training budget rather than systematically varying data scale or compute. Moreover, adding heterogeneous datasets also changes the sampling distribution and potentially the data quality, making it difficult to attribute the observed performance degradation specifically to embodiment heterogeneity. This limits the strength and generalizability of the paper's central conclusions about VLA scaling.
-
Limited real-robot evidence for the main claims. While the inclusion of real-robot experiments is valuable, the evaluation is limited to a single Franka Panda platform, four tasks, and ten trials per task per model. The reported results also do not clearly establish the superiority of EEF-relative actions over world-relative actions in real-world settings (44% vs. 45% average score). Without stronger statistical analysis or broader physical validation, it remains unclear how robustly the findings generalize across robot embodiments and real-world conditions.
Questions for authors
-
Scaling conclusions versus data-mixture effects. The paper argues that simply adding heterogeneous robotic data can negatively affect VLA scaling. However, the experiments in Section 5.2 keep the number of training steps, batch size, and total sampled examples fixed while changing the dataset composition. Could the authors clarify how these results distinguish negative transfer caused by embodiment heterogeneity from effects of data distribution, sampling ratios, or data quality? In particular, would the main conclusions still hold under a controlled scaling experiment that increases the amount of training data and compute, rather than replacing part of the training distribution? This distinction is important for determining whether the findings reflect a general VLA scaling limitation or a specific data-mixture effect.
-
Evidence for the superiority of EEF-relative actions. The paper identifies EEF-relative actions as a preferred representation for cross-embodiment transfer. However, Figure 4 shows that EEF-relative and world-relative actions achieve nearly identical average scores in real-robot evaluation (44% versus 45%). Could the authors explain this discrepancy between the simulation transfer results and real-world performance? What evidence establishes that the advantage comes specifically from the end-effector coordinate frame rather than from using start-relative actions instead of step-wise delta actions? A clearer decomposition of these factors would substantially strengthen the physical alignment claim.
-
Robustness and statistical reliability of real-robot experiments. The real-world evaluation is conducted on a single Franka Panda platform with four tasks and ten trials per task per model. Although the Grouped Blind Ensemble protocol is a valuable effort to reduce operator bias, could the authors provide uncertainty estimates, such as confidence intervals or paired statistical comparisons, for the key differences between action representations and pretraining mixtures? In particular, are the performance differences between D1 and the other mixtures statistically distinguishable? This would help determine whether the reported real-world trends are robust rather than attributable to evaluation variability.
-
Scope of the regularization conclusions. The paper reports that sensory dropout and two-stage training do not consistently improve performance. Could the authors clarify whether these conclusions are supported across different pretraining mixtures, training budgets, or independent runs, rather than primarily under the selected training configuration? This would help establish whether these are general findings about VLA pretraining or configuration-dependent observations.
Overall assessment and importance of the questions.
I consider Questions 1–3 particularly important because they directly concern the validity and empirical support of the paper's main conclusions. In particular, distinguishing genuine scaling limitations from data-mixture effects, establishing the specific contribution of EEF-relative action representations, and demonstrating the statistical reliability of the real-robot results are essential for assessing the significance of this study.
I appreciate the practical relevance and breadth of the experiments, and I would welcome careful, evidence-based responses to these questions. The authors' responses, particularly to Questions 1–3, will substantially influence my final recommendation regarding acceptance.
Thank you for the thorough and constructive review.
Q1. Heterogeneity vs. data distribution, sampling and quality. We compare data compositions at the same budget (240k steps × batch 256) so that the effect of the data is not mixed with that of longer training. Under this budget, each mixture sees the following amount of OXE data (Table 3, Fig. 5):
Two observations separate heterogeneity from dilution and data quality. First, adding joint-space data (D3 → D4) further reduces the OXE samples seen but does not lower performance; D4 also scores higher than D3 on RoboCasa (50.0 vs. 49.6). Because joint-space data does not share the SE(3) action space with OXE, this suggests that the gap to D1 comes mainly from conflicts within the SE(3) space rather than from dilution alone. For your scaling question, this means the gap is unlikely to disappear simply by training longer, since less exposure to OXE did not by itself lower performance. Second, the largest drop comes from adding real-world EEF data from other robots (D2: AgiBot, RoboMind), so it is not caused by the quality of simulated data; Sec. 5.2 discusses possible sources, such as differing TCP frame definitions and control frequencies (3–30 Hz) across robots.
Q2. Frame vs. reference point. Thank you for suggesting this decomposition. The four parameterizations form a 2×2 design, so the two factors can be separated with existing results. The table shows the pretraining gain over training from scratch (Tables 1–2):
Moving from the World frame to the EEF frame increases the gain by 2.4–3.5 points in every row, whereas switching between Rel and Delta within a frame changes it by −0.4 to +2.6 points, with no consistent pattern. The consistent transfer benefit therefore comes from the EEF frame. The reference point matters for a different reason: on the real robot, start-relative actions run stably while step-delta actions do not (Sec. 5.1). The similar real-robot scores of World-Rel and EEF-Rel (45 vs. 44) match what we observe on LIBERO, which, like our real-robot setup, uses a single arm with a fixed base and cameras: there the two also reach similar success after pretraining (83.5 vs. 84.5), because World-Rel already performs well when trained within a fixed setup (Sec. 5.1). The advantage of EEF-Rel lies in how much it gains from heterogeneous pretraining, as the table shows.
Q3. Statistical reliability. D1 outperforms the full mixture D4 in all 14 downstream comparisons we report:
A two-sided sign test over these 14 paired comparisons gives p = 1.2 × 10⁻⁴, so the difference is statistically distinguishable. D1 also outperforms D2 and D3 on LIBERO and on the real robot (Q1 table). For the action representations, the frame effect holds in all four rows of the Q2 table.
Q4. Scope of the regularization findings. They were measured in our main configuration with the full data mixture, which is the regime of interest: whether these strategies still help once the corpus is large and diverse. In this setting, removing a regularizer never lowers the average (no state masking +0.7, no view dropout +1.1, Stage 2 Only +1.3), and strengthening it never raises it (state masking 0.5: −0.5; view dropout 0.5: ±0.0).
We hope these responses address your concerns. Thank you again for your thoughtful review.