- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SimVLA 2.0, a VLA model that drops all pretrained language components, and trains them from scatch end2end with action loss. The method is evaluated on multople sim benchmarks and achieves good performance. The ablations show that the instruction itself is important, but initializing from pretrained text encoders does not help, hence concluding that current robotics sim benchmarks do not require pretrained text encoders.
Critical strengths
- The ablation is clean and clealry supports that pretrained text encoders are not needed for current sim robotics benchmarks
- The evaluation is broad, covering seven sim benchmarks with, and the seed variance study in the appendix is appreciated.
Critical weaknesses
- The main motivation is not tested. The paper uses LIBERO-Para to argue that language-pretrained VLAs are fragile under paraphrase, but it does not evaluate on LIBERO-Para, while LIBERO-PRO is not discrimitive at all among the baselines tested. Also, since the embedding table is random, unseen words remain random vectors at test time. I suspect the robustness comes mostly from overlapping tokens and the visual context
- A simple yet very important baseline is missing: one learned embedding per unique instruction (a task ID). On benchmarks with a fixed set of tasks, such as LIBERO, CALVIN and BEHAVIOR, I would expect this to perform similarly. If it does, the conclusion is more about the benchmarks than about the proposed design
- On the LIBERO-PRO object-attribute perturbation, SimVLA 2.0 is clearly worse than pi-0.5 and openvla (80% vs. 96% and 93%), but the paper does not mention this. For the position perturbation, the improvement is presented against SimVLA, but pi-0.5 performs about the same. It seems results are selectively discussed.
- The reported LIBERO success rate in the main paper appears to be the best of three seeds rather than the mean, and the ablations also use a single seed. The SigLIP2 text encoder result is surprisingly bad, given that all open-sourced paligemma-based VLAs rely on it and can achieve good performance, which makes me wonder whether the baseline was tuned properly. There are no ablations on CALVIN, where the role of language might be more important.
Questions for authors
See weaknesses
4 comments
Response to Reviewer eqBy (Part 2)
(W1b & Q2) Whole-Instruction Task-ID Baseline
“The paper is missing the obvious baseline: a whole-instruction task-ID embedding.”
“How does a whole-instruction task-ID baseline do on LIBERO, LIBERO-PRO Sem, CALVIN, and the WidowX Eggplant task?”
Thank you for suggesting this baseline. We added a whole-instruction Task-ID representation, assigning each registered training instruction string a learned 1,024-dimensional vector. This replaces the token sequence while retaining the vision encoder and action Transformer. Unseen complete strings map to a shared fallback representation, rather than being mapped to their underlying task identities.
| Evaluation | SimVLA 2.0 token-level model | Whole-instruction Task-ID |
|---|---|---|
| Standard LIBERO, average success (%) | 96.4 | 94.8 |
| LIBERO-Para, success (%) | 60.12 | 5.74 |
| LIBERO-PRO Sem, success (%) | 96.5 | 30.5 |
| CALVIN ABC$\rightarrow$D, average chain length | 4.24 | 0.03 |
| WidowX, average success (%) | 91.7 | 50.0 |
| WidowX Eggplant, success (%) | 87.5 | 4.17 |
Standard LIBERO supports your observation that task identifiers can be effective: Task-ID reaches 94.8%, close to the token-level model’s 96.4%.
Under instruction rewording, however, exact-string lookup is insufficient: on LIBERO-Para, the token-level model achieves 60.12%, versus 5.74% for whole-instruction Task-ID. On CALVIN, all 34 evaluation strings are absent verbatim from the training inventory, explaining the baseline’s 0.03 average chain length. These results demonstrate a practical advantage of token-level conditioning over exact-string lookup, without establishing systematic compositional understanding.
We will include these results to clarify that whole-instruction task IDs support strong control on familiar instructions, while token-level conditioning handles instruction rewording more effectively than exact-string lookup.
(W1c) Composition, Visual Context, and Task-Perturbation Results
“The recombination claim rests on a single WidowX task at 24 episodes.”
“You can’t conclude from floor-level numbers that language pretraining ‘buys no compositional goal grounding.’”
Thank you for highlighting this distinction. We will replace “composition over action-trained token embeddings rather than whole-instruction memorization” with “successful execution under an unseen instruction string composed of training-seen tokens.”
The no-instruction results—6.4% on LIBERO-Goal and 20.8% on WidowX Eggplant—support the usefulness of instruction conditioning (Table 9 in the manuscript). They do not, however, determine whether the policy uses keywords or more structured relationships. Likewise, the new LIBERO-Para results demonstrate tolerance to rewording rather than replacing a systematic held-out recombination test.
We will remove the inference that low task-perturbation scores establish an absence of compositional benefits from language pretraining. Our central finding concerns control sufficiency in the evaluated task-bounded settings, which is compatible with reliance on familiar lexical cues.
(W2) Balanced Reporting of LIBERO-PRO Robustness
“The robustness results are selectively reported.”
“SimVLA 2.0 averages 80.0% (54% on Long) vs. 96.0% for and 93.0% for OpenVLA.”
Thank you for highlighting this limitation. Although Table 8 reports these results, we will discuss the object-attribute weakness more explicitly and clarify the comparison on position robustness.
One possible explanation is sensitivity to changes in object appearance relative to the training demonstrations, with errors potentially accumulating over longer task sequences. Differences in visual representations, pretraining data, and augmentation may also contribute; the current cross-system comparison does not isolate these factors. Improving robustness to object-attribute changes is an important direction for future work, including controlled studies of visual augmentation and representation choices. We will revise the discussion to reflect this limitation.
Response to Reviewer eqBy (Part 3)
(W3b & Q4) Text-Tower Optimization and Three-Seed Ablations
“Was the tower frozen or fine-tuned, and was any tuning done for it? Can you run the ablations with 3 seeds?”
Thank you for highlighting the importance of text-specific optimization and repeated evaluation.
1. Training settings and additional tuning.
In Table 9, the full 24-layer SigLIP2 text tower and pretrained embedding tables were fine-tuned under the action objective. We remove the tower’s pooled projection head and pass its final-layer, 1,024-dimensional token features directly to the joint action Transformer, without an additional adapter.
We use AdamW with and zero weight decay. Only the action heads are trained during the first 1,000 updates. Thereafter, the pretrained variants use the same text learning rate as the default random-embedding model on each benchmark:
The original settings were selected for the default model () and reused without separate tuning for the pretrained variants.
To address this concern, we subsequently swept , keeping other training settings fixed within each sweep; zero freezes the text component. We also extended component ablations to CALVIN (please see W3c below).
2. Three-seed results.
Following the learning-rate sweep over , we repeated the selected configuration for each text variant on LIBERO and WidowX using seeds 0, 1, and 2. The tables report mean ± standard deviation.
(a) LIBERO (%)
| Configuration | Spatial | Object | Goal | Long | Avg. |
|---|---|---|---|---|---|
| Default | 98.4±0.2 | 99.1±0.4 | 96.6±0.6 | 91.8±0.3 | 96.5±0.3 |
| BERT embedding table | 99.6±0.2 | 99.6±0.4 | 96.7±0.6 | 88.3±2.3 | 96.1±0.7 |
| SigLIP2 embedding table | 97.8±0.8 | 98.1±1.7 | 94.1±1.7 | 85.3±3.2 | 93.8±0.7 |
| SigLIP2 text encoder | 97.7±0.8 | 98.7±0.7 | 94.8±1.3 | 88.3±1.6 | 94.9±1.0 |
(b) WidowX (%)
| Configuration | Carrot | Eggplant | Spoon | Stack | Avg. |
|---|---|---|---|---|---|
| Default | 100.0±0.0 | 86.1±2.0 | 100.0±0.0 | 83.3±5.9 | 92.4±1.8 |
| BERT embedding table | 97.2±2.0 | 88.9±4.0 | 93.1±1.9 | 73.6±2.0 | 88.2±1.7 |
| SigLIP2 embedding table | 97.2±2.0 | 54.2±15.6 | 100.0±0.0 | 83.3±6.8 | 83.7±6.0 |
| SigLIP2 text encoder | 93.1±1.9 | 44.4±10.4 | 100.0±0.0 | 84.7±8.5 | 80.6±4.7 |
BERT embeddings perform close to the default on LIBERO (96.1 ± 0.7% versus 96.5 ± 0.3%). Across these benchmarks, the pretrained variants do not show a consistent improvement over the default, and we avoid interpreting small numerical differences as definitive advantages.
3. Eggplant performance.
The lower SigLIP2 scores persist in the three-seed averages: 54.2 ± 15.6% for the embedding table and 44.4 ± 10.4% for the text encoder, versus 86.1 ± 2.0% for the default. BERT embeddings achieve 88.9 ± 4.0%, so the decrease is not a general consequence of pretrained initialization.
One plausible explanation is the change in how instruction cues share supervision. The SigLIP2 variants replace the 30,522-token WordPiece vocabulary with a 256k multilingual SentencePiece vocabulary that segments “eggplant” differently, increase the embedding-table size from approximately 31M to 262M parameters, and change the representation geometry. These factors may affect transfer to the unseen complete instruction, although the current experiments do not isolate their individual effects.
We will incorporate these repeated results and optimization details into the revised manuscript. Our conclusion is no consistent advantage for pretrained text components under the evaluated settings, while competitive instruction-conditioned control remains achievable without pretrained language-side parameters.
Response to Reviewer eqBy (Part 4)
(W3a & Q5) Three-Seed Means and Headline Reporting
“Please report the headline LIBERO/WidowX numbers as 3-seed means and clarify how the reported seed was picked.”
Thank you for requesting clearer reporting. Our main tables consistently report seed-0 results rather than the best result across seeds, following the single-run reporting convention used by baselines such as X-VLA and VLA-Adapter. This keeps the reporting format consistent with the published comparisons. For example, we report 91.7% on WidowX from seed 0, rather than seed 2’s higher 94.8%. The corresponding LIBERO result is 96.4%, while 96.9% corresponds to seed 1 (this reporting error has now been corrected). We will ensure that the manuscript’s values and seed labels consistently reflect this convention.
To complement these single-run comparisons, Appendix A.7 reports multi-seed variability. We have now extended this analysis to CALVIN and the key text-component ablations. To directly address your request, we report the three-seed results below and will include the means and standard deviations alongside the headline scores.
| Benchmark | Seed 0 | Seed 1 | Seed 2 | Mean±std |
|---|---|---|---|---|
| LIBERO average success (%) | 96.4 | 96.9 | 96.2 | 96.5±0.3 |
| WidowX average success (%) | 91.7 | 90.6 | 94.8 | 92.4±1.8 |
| CALVIN average chain length | 4.24 | 4.19 | 4.28 | 4.24±0.04 |
The rebuttal time budget allowed us to extend the default-model seed study only to CALVIN. We will complete and report multi-seed evaluations across remaining datasets.
(W3c) SimVLA Comparison, Parameter Accounting, and Longer-Horizon Ablations
“The SimVLA comparison isn’t a matched experiment.”
“SimVLA 2.0 is ~639M total … despite being labeled ‘0.3B’.”
Thank you for requesting clearer parameter accounting and controlled evidence.
1. Backbone size versus total policy size.
The 0.3B label refers to the pretrained SigLIP2 vision backbone, not the complete policy. SimVLA 2.0 has approximately 0.639B total parameters, including its action Transformer, instruction embeddings, and associated projections. We will explicitly label backbone and total-policy parameter counts in the main tables.
2. A more direct language-side removal experiment.
SimVLA and SimVLA 2.0 share a hyperparameter search space, but their backbones and selected configurations differ. The original published scores are therefore system-level references rather than isolated estimates of language pretraining’s contribution.
To address this concern, we additionally evaluated SimVLA with its language side removed on Google Robot. We retain the pretrained vision tower, visual connector, and action Transformer, but remove the language model and its output head. Visual features bypass the language model and enter the action Transformer alongside randomly initialized, action-trained WordPiece embeddings. The two variants use the same training data, optimization settings, training budget, action chunking, and evaluation protocol.
| Configuration | Average success (%) |
|---|---|
| SimVLA | 77.0 |
| SimVLA w/o LM | 75.4 |
Removing the language side produces an observed average decrease of 1.6 percentage points. This provides more direct evidence that strong control can be retained without the pretrained language pathway in this setting, while evaluating pathway removal as a whole rather than pretrained initialization alone.
3. Additional CALVIN ablations.
We extended the component analysis to CALVIN ABC$\rightarrow$D:
| Configuration | SR@1 (%) | SR@2 (%) | SR@3 (%) | SR@4 (%) | SR@5 (%) | Avg. len. |
|---|---|---|---|---|---|---|
| Default | 99.4 | 93.0 | 85.8 | 78.6 | 67.6 | 4.24 |
| No instruction | 34.2 | 7.7 | 2.3 | 0.6 | 0.0 | 0.45 |
| Whole-instruction Task-ID | 3.0 | 0.1 | 0.0 | 0.0 | 0.0 | 0.03 |
| BERT embedding table | 91.3 | 78.9 | 64.6 | 54.2 | 43.6 | 3.33 |
| SigLIP2 embedding table | 97.6 | 90.5 | 81.5 | 73.3 | 64.4 | 4.07 |
| SigLIP2 text encoder | 96.4 | 90.9 | 82.6 | 75.5 | 67.1 | 4.13 |
| DINOv3 vision backbone | 82.4 | 60.8 | 44.4 | 31.4 | 20.0 | 2.39 |
| Randomly initialized SigLIP2 vision tower | 55.1 | 27.2 | 12.0 | 4.1 | 1.8 | 1.00 |
These results extend the evidence to long-horizon control: instruction conditioning and visual pretraining are important, while the selected pretrained text configurations do not show a consistent improvement over the default.
(Q6) Provenance of the BEHAVIOR-1K Public Validation Score
“Was the BEHAVIOR-1K score from an official challenge submission, or evaluated post hoc on the public validation set?”
Thank you for requesting this clarification. Our score comes from a post-challenge evaluation on the Public Validation split. The evaluation records supporting this score and representative rollout videos are available on our anonymous project page (link).
Response to Reviewer eqBy (Part 1)
Thank you for your detailed and insightful review. We sincerely hope our clarifications and new experimental results below will address your concerns.
(W1a, Q1 & Q3) LIBERO-Para, Training-Unseen Tokens, and Inference-Time Handling
Thank you for suggesting this evaluation. We have added LIBERO-Para experiments for the default model and pretrained text variants, together with token-coverage analysis.
1. Evaluation and definition of training-OOV.
We evaluate each configuration using the LIBERO-Para protocol. Here, training-OOV means a token ID absent from the corresponding action-training instruction vocabulary, rather than a token outside the tokenizer vocabulary. Neither WordPiece nor SigLIP2 SentencePiece produces tokenizer-level
[UNK]tokens on the LIBERO-Para prompts.For context, the default’s 60.12% approaches the published X-VLA result (62.1%) and exceeds VLA-Adapter (46.3%).
2. Handling and prevalence of unseen tokens.
The default model uses a fixed WordPiece tokenizer and randomly initialized, action-trained embeddings. An unfamiliar word is first segmented into subwords. IDs encountered during training use their updated embeddings; IDs never encountered retain their individual initial random vectors under our zero-weight-decay setup. They are not mapped to a shared zero representation.
The training instructions contain 112 unique strings and 87 unique content WordPiece IDs. Coverage differs across evaluations:
3. Performance under reduced lexical coverage.
Grouping LIBERO-Para episodes by the per-instruction share of training-unseen content tokens gives:
The default maintains approximately 80% success when at most 25% of content tokens are training-unseen and achieves 34.14% when the proportion exceeds 50%. Thus, untrained token embeddings do not necessarily prevent successful execution, although higher unseen-token proportions are associated with lower success.
With the same WordPiece tokenizer and bucket assignments, BERT embeddings perform better in the zero-unseen-token group, while the default performs better in all three groups containing unseen tokens. The evaluated pretrained representations therefore do not provide a consistent advantage under these settings.
These results extend our evidence beyond entirely training-seen vocabulary. Our claim remains competitive instruction-conditioned control without pretrained language-side parameters within the evaluated task-bounded settings, rather than general open-vocabulary understanding.