- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes StarVLA, a unified framework/infrastructure VLA training and evaluation. The key point is allowing interchangability between models, data methods, training methods, etc. They provide both numerous models and benchmarks, and support many types of training.
Critical strengths
Unifying VLA research into one framework is a good idea. Presently, each research project is an ad-hoc implementation or built on someone else's ad hoc implementation. They may help speed up research, similar to stable baselines for RL. This may help the VLA community build more reproducible and reliable research.
They also provide evidence that their work is already aiding research, which is encouraging.
Critical weaknesses
This is more of an engineering project than a research project. I don't see any contributions other than a well-designed framework. I do see value in this work, but I am not sure if this is the right venue. I guess this is a question for the AC, if the paper needs a research contribution or just engineering contributions to the research community. I have not reviewed for the area "infrastructure, software libraries, hardware, systems, etc." before, so I do not know whats expected here. Once I have feedback from the AC I will update my score.
Questions for authors
-
What is the purpose of Figure 3? If I understand correctly, StarVLA in this figure is just some combination of configs, but they arent even all the same. Is the point of this to illustrate that mixing and matching models, training methods, etc can perform close to state of the art?
-
What is the research contribution? What did you discover? A new algorithm, training method? Please just clarify that the sole contribution is the infrastructure.
-
Not a question, but a nice experiment would be to show that you can come up with novel research simply within your framework, ie perform ablations on common methods and find a nice contribution that would not be possible without your infra. For example, maybe some insight into how training data affects all of these methods and is biasing our interpretation of each method, since different models use different datasets even if their inputs/outputs are identical.
-
How necessary are these unified packages in the age of Claude/codex? 10 years ago this would have been great for accelerating research, but now people just use AI to handle interfacing between models, training methods, datasets. Make a case for why this is still necessary.
Response 1/2: Research Enabled by StarVLA and Measured Training Efficiency
Thank you for the careful reading, and apologies for the late reply. We answer the two weaknesses and two questions together.
W1 / Q1. Research enabled beyond component composition.To be explicit: we do not claim "component composition" as a novelty; every VLA codebase composes components in some form. What matters is what sits under one stack. Our contribution is the infrastructure and the research workflows it supports. In StarVLA, VLM backbones and video world-model backbones share the same action heads, dataloaders, trainers, and evaluation interfaces, and the same stack supports vision-language co-training and 13 benchmarks. To our knowledge, StarVLA is the first framework to combine these: in Table 1, RoboVLMs, Dexbotic, and VLA Foundry have no world-model policies, FluxVLA Engine comes with each policy's own pipeline, and LeRobot integrates policies as plugins rather than as composable backbones and heads.
You ask for new findings or measured savings; we address each below.
1. On research findings: a VLM and a video world model under one pipeline. Section 5.3 (Table 5) brings together the backbone and action-head results from Appendix C.5 (Table 10). The LIBERO and SimplerEnv grid share a data pipeline, training budget, and evaluation protocol.
At 30k steps on LIBERO, the eleven backbone and action-head configurations span 95.2–96.6%. The SimplerEnv results extend the comparison to a second environment, far from saturation, with the same three continuous action heads. With the action head held fixed, the VLM and the world-model backbone differ by at most 1.3 points on LIBERO and 3.7 points on SimplerEnv WidowX. To our knowledge, the two backbone families had not previously been compared with the action head and the evaluation protocol held fixed.
2. On "no seeds or variance": evaluation protocol and training cost. Our evaluation follows the practice of assessing trained policies through repeated rollouts, as in OpenVLA-OFT, π0, and GR00T N1 [1–3]. Each LIBERO entry aggregates 2,000 rollouts across four suites, and each SimplerEnv entry uses five repetitions of the official evaluation. Given the substantial cost of repeating training runs for 2–9B models, we use one training run per configuration and evaluate each policy across multiple tasks and repeated rollouts.
3. On "counts and adoption only": efficiency evidence already in the submission. The submission already reports quantitative training-efficiency measurements in Section 6.2 and Figure 4. On 8×H200 at batch 32 per GPU, the reported step times are 0.78 s for StarVLA, 1.12 s for LingBot-VLA, and 1.98 s for Dexbotic (measurement details in Appendix E.3). Our A100 series scales from 8 to 256 GPUs at batch eight per GPU, reaching 2,200 samples/s and 79.1% scaling efficiency.
To make the research findings more prominent, we have brought the key comparisons and findings from Appendix C.5 into Section 5.3 of the revised main text. We did not foreground the comparison in point in the original submission because we submitted to the infrastructure track, where the contribution is the system. We believe the absence of a new learning algorithm should not, by itself, be grounds for rejection in that track.
[1] Kim et al. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success. RSS 2025.
[2] Black et al. π0 RSS 2025.
[3] Bjorck et al. GR00T N1.5