- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper addresses test-time scaling for frozen VLAs: the policy samples several candidate action chunks and must pick one to execute. Existing Best-of-N approaches rank candidates with a single learned scalar score. The authors argue that such a score hides trade-offs between objectives and tends to favor candidates whose success is overestimated. They propose S2S-VLA, which scores each candidate on three costs: predicted task-failure probability, predicted anomaly probability, and an analytic command-roughness term. The candidate is chosen by minimizing an augmented Tchebycheff scalarization, which guarantees that the choice is Pareto-nondominated within the candidate set. A lightweight evaluator on frozen VLA features predicts the two probabilities. It is trained offline on simulator branch rollouts, with a pointwise BCE loss plus a pairwise ranking loss, and then calibrated with temperature scaling. Applied to OpenVLA, π0 and π0.5, S2S-VLA consistently improves success on RoboCasa, LIBERO-Plus and RLBench, by about 7–8 points on average. It also helps in a small real-robot study.
Critical strengths
- The method plugs into frozen VLAs, and gives consistent gains across three backbones and three benchmarks at a moderate latency cost.
Critical weaknesses
-
There is no comparison with other test-time scaling methods, and no compute-matched trade-off. Every result compares a backbone only with itself plus S2S. Several established methods sample action candidates from a VLA and select among them, and they are much simpler:
- BID (Liu et al., 2025) and MG-Select (Jang et al., 2026) need no extra verifier.
- RoboMonkey (Kwok et al., 2025) uses Gaussian perturbation, majority voting and a VLM verifier.
- V-GPS (Nakamoto et al., 2024) re-ranks candidates with a value function.
- TACO (Yang et al., 2025) uses a pseudo-count verifier.
- CoVer-VLA (Kwok et al., 2026) also learns a verifier.
None of these is evaluated, and BID, MG-Select and TACO are not even cited. The paper reports latency against the number of candidates for OpenVLA only. It never puts success against total cost next to these alternatives. The total cost includes online latency for every backbone and the offline cost of supervision. Without this comparison, it is unclear whether S2S-VLA beats a simpler selector at the same compute.
-
The pipeline is complex, and neither its components nor its scalability is justified. S2S-VLA requires all of the following:
- a resettable simulator with 15 branch rollouts per candidate, which is 240 rollouts per training context at 16 candidates
- task-specific anomaly definitions
- a separate evaluator for each backbone
- temperature calibration
- a preference search on validation episodes.
None of these choices is ablated. The paper never tests Tchebycheff against a weighted sum, and never tests the pairwise loss, temperature scaling, the roughness term or the preference search individually. The only ablation removes objectives. The two propositions are standard or hold by construction, so they do not justify the design either. The paper also never discusses how the cost grows with new tasks or embodiments. It does not explain how the real-robot evaluator was supervised, given that branch rollouts are impossible there.
Questions for authors
- How does S2S-VLA compare with BID, MG-Select, RoboMonkey, V-GPS, TACO and CoVer-VLA, using the same backbone and the same number of candidates?
- Please report success against total compute for S2S-VLA and these baselines. This should include online latency for all three backbones and the offline cost of collecting branch rollouts and training the evaluator.
- Can you ablate each component individually: Tchebycheff versus a weighted sum, the pairwise loss, temperature scaling, the roughness term and the preference search? Please also report calibration error.
- What concrete advantage over simpler or training-free selectors justifies this pipeline? How does its cost grow with new tasks, embodiments and backbones? Does each new task need new branch rollouts and anomaly definitions?
- How was the evaluator trained for the real-robot experiments without branch rollouts? How does cross-backbone transfer (Figure 7) work when the evaluator’s input features are backbone-specific?
Response to Reviewer i58E
Dear Reviewer i58E,
Thank you for recognizing our method's consistent gains across backbones and benchmarks at a moderate latency cost.
Matched candidate count. On RoboCasa at K = 16, S2S gained 1.4–2.0 percentage points over BID across backbones and 1.0 point over MG-Select on OpenVLA. Relative to CoVer-VLA, the differences were +0.5, −0.1, and −0.4 points on OpenVLA, π0, and π0.5, respectively.
MG-Select uses action-token distributions, which OpenVLA supplies. The two π checkpoints use continuous action heads.
Inference time and execution quality. On OpenVLA, K = 16 gave 41.0% success at 0.35 s per step. Increasing K to 64 gave 41.9% at 1.30 s (Figs. 5a and 8b). For π0 and π0.5, each applicable baseline could tune K to match S2S's end-to-end decision time on one GPU.
On π0.5 at 0.36 s, S2S recorded two fewer anomalous prefixes per hundred than the highest-success baseline (9% vs. 11%) and lower mean normalized roughness (0.42 vs. 0.46), with a 0.1-point difference in success.
Offline and deployment cost. BID and MG-Select require no separate verifier fit. V-GPS, TACO, RoboMonkey, and CoVer-VLA use learned scorers. For S2S, 1,000 contexts × 16 candidates × 15 repeats require 240,000 branches. At 2,000 branches per GPU-hour, collection takes 120 GPU-hours. Fitting adds 8 GPU-hours and calibration plus preference validation adds 2, for 130 GPU-hours of setup. At 0.36 s each on π0.5, 10,000 sequential decisions correspond to about one GPU-hour of online decision time. Eight repeats save 46.7% of branches and give 55.2% success versus 55.6% with 15.
Component ablations. Full S2S reached 55.6% on RoboCasa with π0.5. Changing one component gave 55.0% with a tuned weighted sum, 54.8% with probability-only training, 55.5% with unit calibration temperatures, 54.3% with roughness removed, and 55.0% with fixed equal weights. Success-only ranking reached 50.8% (Fig. 5b). On held-out in-domain RoboCasa data, temperature scaling halved expected calibration error, from 0.08 to 0.04 for success and from 0.06 to 0.03 for anomalies.
Proposition 1 guarantees nondominance within the valid set under the predicted costs. Proposition 2 shows that fitting the empirical probabilities exactly also reproduces the pairwise targets. Both losses reuse the same labels. At deployment the evaluator scores all valid prefixes in one batch.
Transfer and scaling. In Fig. 7a, the target policy proposes the prefixes. The frozen source encoder reads the observation and instruction for its own evaluator head, adding one encoder pass per off-diagonal decision. On π0.5 proposals, the π0 evaluator reaches 53.8%, recovering 5.7 of the matched evaluator's 7.5 points above the 48.1% base policy. New tasks can reuse the detector and evaluator when event definitions and candidate rankings carry over. New embodiments also need compatible action coordinates. Changed event definitions or poor target-domain ranking can require fresh branches and refitting.
PiPER supervision. We trained the evaluator on simulator branch outcomes and froze it for hardware evaluation. We used the 50 physical demonstrations per category to adapt the proposal policy. In Fig. 8a, the base policies completed 81/180 trials (45.0%) and S2S completed 109/180 (60.6%) across three backbones and three categories.