acceptodds
Under review as a conference paper at ICLR 2027

Look Before You Leap: Distilling Tree Search into Action Evaluation for Frozen VLA Models

Abstract

Vision-Language-Action (VLA) models acquire broad embodied capabilities through large-scale pretraining, yet their generalization remains far more fragile than that of LLMs and VLMs. The prevailing remedy, post-training via supervised fine-tuning or reinforcement learning, improves task-specific performance but narrows the generalist capability that makes pretraining valuable. We identify a key bottleneck: VLA failures stem not only from action *generation* but also from action *evaluation*. A diagnostic pass@k study confirms that frozen VLAs already contain competent behaviors in their output distribution, with overall success rates rising from 33% at pass@1 to 92% at pass@32. Inspired by this, we propose **SVA** (**S**earch, **V**alue, and **A**ct), a simple framework that equips frozen VLA policies with long-term consequence awareness. SVA first uses Monte-Carlo tree search in simulation to sample and explore the VLA's output distribution under a finite search budget, collecting diverse trajectories annotated with empirical returns; this knowledge is then distilled into a lightweight Q-value model that predicts the expected consequence of candidate actions; at deployment, the frozen VLA proposes multiple candidates and the evaluator selects the one with the highest uncertainty-regularized Q-value, requiring no simulator access. By decoupling action proposal from consequence evaluation, SVA preserves the generalization capacity of the VLA backbone while improving task success. Across embodied benchmarks, SVA consistently improves generalization on unseen tasks and exhibits strong test-time scaling; across real-robot manipulation tasks, it improves average success from 42.8% to 55.6% (**+12.8 points**). Strikingly, SVA enables a 9B VLA to outperform a 27B VLA by **7 points** at **27% lower inference latency**, suggesting that scaling test-time evaluation is more cost-effective than scaling model size.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, action evaluation, test-time scaling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_68KqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SVA (Search, Value, and Act), a test-time action-selection framework for VLA models. The authors first use MCTS in simulation to explore candidate trajectories and obtain return targets, then distill these targets into a lightweight Q-value evaluator. At deployment, the VLA policy remains fixed, generates multiple candidate actions, and the evaluator selects the candidate with the highest uncertainty-regularized value. Experiments cover embodied reasoning, simulated manipulation, test-time scaling, and five real-robot tasks.

Critical strengths

1.The paper addresses an important problem: improving VLA performance without modifying the policy backbone. Separating action generation from action evaluation is a sensible and practically relevant direction.

2.The experimental coverage is strong. The method is evaluated across several benchmarks, action spaces, proposer backbones, and model scales, together with real-robot experiments.

3.The controlled comparisons and ablations are useful. The paper examines MCTS supervision, learned value evaluation, multi-candidate selection, search-data scale, and candidate-budget scaling.

4.The test-time scaling results are interesting. In particular, the comparison between a smaller model with SVA and a larger single-shot model suggests that additional evaluation compute can be an effective alternative to parameter scaling.

Critical weaknesses

1.The pass@k analysis does not fully establish an “action evaluation bottleneck.” Higher pass@k only shows that some complete rollouts succeed under repeated sampling; it does not demonstrate that failures primarily arise from selecting the wrong action among locally available candidates.

2.The methodological novelty is somewhat limited relative to recent work on verifier-guided VLA inference, value-guided action selection, and test-time sampling. The main distinction is using MCTS-generated return targets to train the evaluator, but the conceptual gap from related approaches could be sharper.

3.The claim that SVA preserves generalist capability is not directly evaluated. Freezing the VLA parameters prevents parameter forgetting, but does not by itself show that the overall SVA system preserves performance across the policy’s broader pretrained capabilities.

4.SVA relies on substantial simulator-based search supervision and benchmark-specific evaluators. This limits the generality of the approach in settings without resettable simulators or task reward signals, and the cost of collecting MCTS supervision is not directly compared with alternative post-training approaches.

5.The real-robot results are encouraging, but both the proposer and evaluator are adapted using 500 real-robot trajectories. Therefore, these experiments demonstrate deployment-time frozen selection after adaptation rather than direct zero-shot transfer of the simulation-trained evaluator.

Questions for authors

1.How does the pass@k experiment distinguish an action-evaluation bottleneck from stochasticity or compounding execution errors across complete rollouts?

2.What is the main technical distinction between SVA and existing value- or verifier-guided VLA action-selection methods beyond the use of MCTS-generated supervision?

3.Can the authors clarify what empirical evidence supports the claim that SVA preserves generalist capability, beyond keeping the VLA parameters frozen?

4.How much simulator interaction and compute is required to generate the MCTS training data, and how does this cost compare with the post-training alternatives discussed in the paper?

5.In the real-robot study, how much of the improvement depends on adapting the Q-model with the 500 real trajectories rather than transferring the simulation-trained evaluator directly?

4 comments
AuthorsOpenReview ↗

Author Response (Part 1/3)


W1 & Q1: Interpreting pass@k and establishing the action-evaluation bottleneck.

A1. Pass@k shows that the frozen policy can succeed, not why it fails. We agree that a higher pass@k cannot distinguish a wrong action choice from rollout stochasticity or accumulated execution errors, and we do not claim that most failures stem from action selection. Our claim is narrower: choosing among the proposer's own candidates at a given state is a substantial and addressable source of failure.

State-level analysis. In the RoboTwin π0.5\pi_{0.5} search data, a candidate is successful if executing it and then rolling out the frozen policy reaches task success. Of the 360 initial states, 35% contain both successful and failed candidates. Choosing one of eight candidates at random succeeds 73.33% of the time, while at least one candidate succeeds in 88.33% of states (a 15-point gap); across all 3,357 states with eight evaluated candidates the figures are 79.91% and 88.62%. Each label comes from a single stochastic rollout, so this gap is an optimistic estimate of what selection can achieve. It motivates an evaluator but does not by itself show that selection is the bottleneck; the evidence is the downstream gain of the learned Q-evaluator over other selectors on the same frozen proposer (A2).

Same-state case studies. In EmbodiedBench (Table 6, Fig. 10), the goal-directed plan has a lower prior than the distractor-driven plan (0.0625 vs. 0.4375) but a higher Q-value (0.2948 vs. −0.1282); SVA selects it and succeeds. On the real robot (Fig. 7), at step 48 of pick_dual_bottles SVA scores 16 action chunks from the same state and selects the one with the highest Q-value (0.80); the episode succeeds at step 95, while the baseline reaches the 200-step limit. These cases are illustrative, not statistical evidence.


W2 & Q2: Technical distinctions beyond MCTS-generated supervision, and novelty beyond recent work on verifier-guided VLA inference.

A2. SVA bridges training-time search and test-time scaling through learned action evaluation. It distills the outcomes of simulation-based search into a lightweight Q-evaluator, enabling frozen embodied policies to select actions based on their downstream consequences without online simulator search. This places SVA between two lines of prior work. Learned selectors score candidates cheaply at test time but are trained without search: TACO [1] ranks candidates by learned pseudo-counts, V-GPS [2] learns a Q-function through offline RL, and RoboMonkey [3] trains an action preference model on synthetic preference data. Search-at-deployment methods, such as value-guided tree search with V-VLAPS [4] or rollouts in a learned world model, look ahead at test time but pay the search cost at every decision. SVA moves search to training time. Because the search branches over the frozen proposer's own candidates, sibling candidates from the same state receive Monte Carlo return labels that include both successful and failed outcomes. The Q-evaluator regresses these returns without temporal-difference bootstrapping, so it learns the same comparison it performs at deployment, where compute scales only with the number of scored candidates.

On RoboTwin, SVA outperforms both learned selectors with the same frozen proposer and candidate budget (mean ± SD of the 12-task average over three evaluation runs of 50 episodes per task, fixed checkpoints).

MethodAverage SR (mean ± SD, %)
π0.5\pi_{0.5}62.72 ± 4.58
π0.5\pi_{0.5} + TACO [1]65.33 ± 2.24
π0.5\pi_{0.5} + V-GPS [2]64.72 ± 1.06
π0.5\pi_{0.5} + SVA67.78 ± 0.98

A controlled ablation isolates the effect of search supervision. Rollout-Q and MCTS-Q use the same 30 training episodes per task, the same number of training samples (92,728), the same Q-evaluator architecture and hyperparameters, and the same frozen proposer, and are evaluated on identical test episodes and candidate settings. Only the supervision differs: returns from independent sequential rollouts versus MCTS-backed returns. Rollout-Q reaches 64.17%, close to TACO (65.33%) and V-GPS (64.72%), whereas MCTS-Q reaches 67.78% (+3.61 points) and is higher on 11 of 12 tasks. Search adds same-state success/failure contrast among sibling candidates and concentrates the budget on promising branches; the ablation measures their combined effect.

[1] Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach. arXiv:2512.02834, 2025.

[2] Steering your generalists: Improving robotic foundation models via value guidance. CoRL 2024.

[3] RoboMonkey: Scaling test-time sampling and verification for vision-language-action models. CoRL 2025.

[4] V-VLAPS: Value-guided planning for vision-language-action models. arXiv:2601.00969, 2026.

AuthorsOpenReview ↗

Author Response (Part 2/3)


W3 & Q3: Multi-task evidence and the scope of capability-preservation claims.

A3. We assess capability preservation by comparing the full SVA system with the same frozen proposer. The proposer’s existing capabilities provide a common starting point, allowing us to assess how evaluator-guided selection changes performance across tasks. Our empirical support therefore comes from evaluating the full proposer-plus-evaluator system, not from parameter freezing alone.

Shared Q-evaluators improve average multi-task performance in the evaluated settings. A single Q-evaluator is jointly trained across all 12 RoboTwin tasks; each EmbodiedBench suite likewise uses one Q-evaluator across its task categories. On RoboTwin (Table 2), SVA improves π0.5\pi_{0.5} on 11 of 12 tasks, raising average SR from 62.72% to 67.78%. On both EmbodiedBench suites (Table 1), the Q-evaluator is reused across five proposer backbones, with positive average gains for all five.

We further evaluate SVA on three RoboTwin tasks that the Q-evaluator never saw. place_shoe, place_object_scale, and shake_bottle are excluded from all search and Q-evaluator training data. We use the same frozen π0.5\pi_{0.5} checkpoint, the 12-task Q-evaluator without further training, and the Table 2 protocol (three runs × 50 episodes per task, 150 trials per task and method). On these unseen tasks, SVA leaves the frozen proposer’s average success unchanged (86.7% for both). The Q-evaluator therefore does not degrade the proposer on tasks outside its training data.

Unseen Taskπ0.5\pi_{0.5}π0.5\pi_{0.5} + SVA
Place Shoe87.3%88.7%
Place Object Scale73.3%71.3%
Shake Bottle99.3%100.0%
Average86.7%86.7%

W4(a) & Q4: Offline supervision cost and comparison with post-training.

A4(a). SVA’s full offline cost on RoboTwin is about 13 hours on 4 × RTX 4090. On simulator interaction, MCTS runs one search per training episode with 128 simulations. Each simulation expands one node and evaluates it with a frozen-policy rollout. Appendix D.2 give the full search settings; the measured resource usage is below. Q-evaluator training fits the Qwen3.5-0.8B backbone with LoRA and five value heads on all of these samples. Both stages are fully automated, and neither updates the VLA backbone. GPU memory is the peak usage per GPU.

StageHardwareWall-clock timeGPU memory
MCTS data collection4 × RTX 4090≈ 12 h≈ 22 GB
Q-evaluator training4 × RTX 4090≈ 1 h≈ 18 GB

Published RL post-training methods report larger training setups, although the settings are not directly comparable. The table below lists the training configurations reported by recent RL post-training works. None of them reports total GPU-hours, and they differ from our setting in hardware, backbone, task set, and initial demonstrations. The table is therefore not a controlled comparison, and we do not derive a cost ratio from it.

MethodBackbone / trained parametersReported hardwareReported training data and interaction
SimpleVLA-RL [5]OpenVLA-OFT, full-parameter training8 × A800 80GBRoboTwin 2.0, 12 tasks: SFT on 1,000 demonstrations per task, then RL on 1,000 simulation scenarios per task with 8 sampled rollouts each
RLinf-VLA [6]OpenVLA-OFT, LoRA (GRPO)8–32 × H100 80GBRoboTwin, 6 tasks with one task-specific model each: 1,000 fixed scene seeds per task, 128 parallel environments, group size 8
πRL\pi_{RL} [7]π0\pi_0 / π0.5\pi_{0.5}, 300M action expert (VLM frozen)8 × H100 80GBLIBERO: 500 training epochs, 64 parallel environments, 8 rollout epochs per training epoch
SVA (ours)Q-evaluator only (LoRA + heads)4 × 4090 24GBRoboTwin 2.0, 12 tasks: 30 search episodes per task, no additional demonstrations; ≈ 13 h in total

These RL methods update the policy using about 1,000 training scenarios per task on data-center GPUs, whereas SVA’s offline stage uses 30 search episodes per task on four consumer GPUs and leaves the proposer unchanged. The two approaches are complementary rather than interchangeable: RL changes what the policy generates, SVA changes how its candidates are selected, and an RL-fine-tuned policy can serve as an SVA proposer.

[5] SimpleVLA-RL: Scaling VLA training via reinforcement learning. ICLR 2026.

[6] RLinf-VLA: A unified and efficient framework for reinforcement learning of vision-language-action models. arXiv:2510.06710, 2025.

[7] πRL\pi_{RL}: Online RL fine-tuning for flow-based vision-language-action models. arXiv:2510.25889, 2025.

AuthorsOpenReview ↗

Author Response (Part 3/3)


W4(b): Applicability without resettable simulators.

A4(b). SVA's search stage requires a resettable simulator and a task-success signal; deployment does not. On RoboTwin the signal is a sparse success indicator from the built-in checker, without reward shaping. Deployment uses only candidate generation and Q-evaluator scoring, without simulator access.

When a simulator is available, search supervision is inexpensive. Collection is fully automated, with no human teleoperation or physical resets: RoboTwin MCTS data (30 episodes per task) took about 1 hour per task on 4 RTX 4090 GPUs, whereas teleoperating 100 real demonstrations took about 1.5 hours of operator time per task. For the physical experiments we reused existing RoboTwin 2.0 tasks and aligned only the camera intrinsics/extrinsics and the spacing between the two arms with our platform, once per robot. A task outside RoboTwin would require building a simulated task with a success checker.

Without a simulator, the Q-evaluator can be trained or adapted on independently collected rollouts that contain both successes and failures. The regression objective only needs observed returns for executed state–action pairs, so real-robot rollouts labelled with task success or failure can supervise the Q-evaluator without tree search or intermediate-state restoration. On RoboTwin, Rollout-Q trained on such independent rollouts already improves success over the base policy (64.17% versus 62.72%), and MCTS raises it further to 67.78% when a simulator is available.


W5 & Q5: Quantifying Q-evaluator adaptation versus direct transfer.

A5. Table 4 evaluates frozen-policy selection after adaptation, not zero-shot transfer. With the same real-adapted proposer, the adapted Q-evaluator raises SR from 42.8% to 55.6% (+12.8 points); this alone does not isolate the contribution of real-data adaptation.

Most of the gain comes from direct transfer of the simulation-trained Q-evaluator. All conditions use the same frozen, real-adapted proposer, the same candidate settings, and the same protocol (five tasks, 25 trials per task under each of the target-only and with-distractors conditions). MCTS-only denotes an evaluator with no real-data fine-tuning; the proposer is still real-adapted. Without adaptation it raises SR from 42.8% to 50.4% (+7.6 points); adaptation on the 500 real demonstrations adds 5.2 points, reaching 55.6%. With 250 trials per condition, these are point estimates with substantial sampling uncertainty.

ConditionQ-evaluator trainingSR (%)
Base policyNo reranking; execute the common real-adapted proposer42.8
MCTS-onlyTrained on simulation MCTS data, without real-data adaptation50.4
MCTS pretraining + real adaptationThe same Q-evaluator adapted on the 500 real demonstrations55.6

Adaptation reuses existing demonstrations without additional real-robot collection. We reuse the 500 expert demonstrations (100 per task) already collected for the common proposer, assign reward 1 at the final step of each trajectory with returns discounted backward by 0.99, and fine-tune the LoRA adapters and value heads of the simulation-trained Q-evaluator by Smooth-L1 regression (learning rate 5×10−65\times10^{-6}, batch size 64, 5 epochs).


Summary Response

We thank you for your constructive review. (A1) At 35% of RoboTwin initial states some of the eight candidates succeed and others fail; a random choice succeeds 73.33% of the time versus 88.33% for at least one candidate, an optimistic gap that motivates the evaluator but does not by itself establish the bottleneck. (A2) SVA outperforms TACO and V-GPS with the same proposer and candidate budget, and MCTS adds 3.61 points over rollout returns with the same number of training samples. (A3) On three tasks unseen by the Q-evaluator, average success is unchanged (86.7%). (A4) The offline cost is about 13 hours on four consumer GPUs; the simulator-free rollout route gives a smaller gain and is untested on a real robot. (A5) On the real robot, the simulation-trained Q-evaluator alone raises SR from 42.8% to 50.4%. We hope these clarifications and new evidence fully address your concerns, and we would sincerely appreciate your reconsideration of our work.

The authors’ rebuttal has addressed most of my questions and clarified several aspects of the evaluation and methodology. While some limitations remain, the response improves my understanding of the paper. I therefore keep my original rating unchanged.

Reviewer_txi8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SVA (Search, Value, and Act), an embodied test-time scaling framework for frozen VLA models. Its core idea is to leverage the resettable nature of simulation environments to perform Monte Carlo Tree Search (MCTS), exploring diverse candidate actions and collecting trajectories annotated with empirical returns to train a lightweight Q-value model. At inference time, without requiring online MCTS or simulator access, the frozen VLA proposes multiple candidate action chunks, and the learned Q-model directly selects the best chunk for execution. Experiments are conducted across EmbodiedBench, RoboTwin 2.0, SimplerEnv, and real-world manipulation tasks.

Critical strengths

1.The problem formulation is well-motivated by a compelling pass@k diagnostic study, and the idea of distilling training-time MCTS into a lightweight Q-evaluator is a clever way to achieve consequence-aware evaluation without incurring tree-search overhead at test time.

2.The overall structure of the paper is well-organized, with intuitive method pipelines and clear visualizations. The experimental evaluation covers multiple simulation benchmarks, diverse model backbones, and real-robot tasks, accompanied by a detailed analysis of test-time scaling latency.

Critical weaknesses
  1. Overall, this is a well-executed paper. However, in my understanding, the proposed method suffers from a fundamental practical limitation regarding real-world deployment (please correct me if I have misunderstood, and I would be very willing to adjust my score accordingly). Specifically, SVA relies heavily on training-time MCTS, which fundamentally requires a simulator that can be arbitrarily reset and branched from intermediate states. This makes the core training paradigm almost impossible to apply directly in the real world. In fact, the real-robot experiments in Section 5.2 and Appendix C largely confirm this concern: the authors had to construct physical tabletop scenes that closely mirror the tasks in RoboTwin 2.0, pre-train the Q-model on MCTS data collected in the RoboTwin 2.0 simulator, and then adapt it using 500 real-world demonstrations to bridge the Sim2Real gap (moreover, the paper does not clearly explain how the Q-model is adapted on these 500 real demonstrations without MCTS return labels or sibling candidates). Consequently, the practical applicability of SVA to general real-world tasks appears severely limited. Given that current VLA models almost always require task-specific fine-tuning to work reliably on real hardware, deploying SVA on any new real-world task would require first building a corresponding high-fidelity Real2Sim twin environment, running MCTS in simulation, and then transferring back to the real robot. In my view, this pipeline is prohibitively expensive; a practical method should ideally be directly applicable in real-world settings. Even real-world RL only requires sequential rollouts rather than MCTS that depends on arbitrary state resets.

  2. There are some minor formatting/LaTeX compilation issues—for example, the caption of Figure 5 contains an unresolved table reference (Table ??). While this is a minor issue, a thorough proofreading is recommended.

Questions for authors

1.Referring to Weakness 1, could the authors clarify how the proposed MCTS-based Q-value learning framework can be applied to novel real-world manipulation tasks where corresponding high-fidelity, resettable simulators are unavailable? Additionally, how exactly is the Q-model adapted on the 500 real-robot demonstration trajectories in Section 5.2?

2.Building upon Figure 6 (which only reports test-time scaling on EmbodiedBench) and Figure 9 (which only reports latency for π0.5\pi_{0.5} ), could the authors provide the test-time scaling curves showing how the task Success Rate (SR) changes as the number of candidate action chunks NN increases on the manipulation benchmarks (e.g., RoboTwin 2.0 and SimplerEnv)?

3.Referring to Figure 5, the w/o MCTS ablation (training the Q-model using only standard policy rollout returns) is currently only reported on EB-Navigation. Could the authors clarify the performance difference between MCTS returns and standard policy rollout returns when training the Q-model on manipulation benchmarks (such as RoboTwin 2.0 or SimplerEnv)? In my view, if training the Q-model with standard policy rollout returns can still achieve competitive performance on manipulation tasks, the method would have much greater potential for direct real-world deployment without relying on resettable simulators.

2 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)


W1 & Q1(a): Does SVA require a task-specific resettable simulator for real-world application?

A1(a). SVA moves search to training time: it distills the outcomes of simulation-based search into a lightweight Q-evaluator, so a frozen policy can select actions by their downstream consequences without online search. The resettable simulator is needed only by this offline search stage; the Q-model itself only needs state–action–return samples. At deployment, SVA uses only candidate generation and Q evaluation, without online search or simulator access (Sec. 4.3, App. C.4).

Simulation data is cheap to collect, and our real-robot pipeline needed only platform-level alignment. Simulator data collection is fully automated: environments reset programmatically, run in parallel, and label outcomes with built-in success checkers. Collecting RoboTwin MCTS data (30 episodes per task) took about 1 hour per task on 4 RTX 4090 GPUs with no human involvement, whereas teleoperating real demonstrations (100 per task) took about 1.5 hours of operator time per task. For the physical experiments, we reused existing RoboTwin 2.0 tasks and aligned only the camera intrinsics/extrinsics and the spacing between the two arms with our platform; we did not build per-task assets or tune physical parameters. This alignment is done once per robot and shared by all its tasks. The evaluator was then adapted on the 500 demonstrations already collected to fine-tune the proposer, so no additional real-robot data was needed.

MCTS extracts more informative supervision than independent rollouts. The search evaluates several candidate actions from the same intermediate state, spending more of its budget on promising or uncertain branches. These sibling candidates come with both successful and failed outcomes, which directly supervises the ranking decision the evaluator makes at deployment. With the same number of training samples, MCTS-Q outperforms Rollout-Q by 3.61 percentage points on RoboTwin (A3). The search cost is paid once offline and shared by all deployment episodes.

On the physical robot, simulation-MCTS supervision provides most of the gain, and the real demonstrations adapt it. All conditions use the same frozen, real-adapted proposer and the same physical evaluation protocol. The simulation-only evaluator raises success from 42.8% to 50.4%, and adaptation on the 500 real demonstrations raises it further to 55.6%.

Q-supervision routeEvaluator training dataSuccess rate (%)
Base policyNo Q reranking42.8
MCTS onlySimulation-MCTS data, no real-data adaptation50.4
MCTS pretraining + real adaptationSimulation-MCTS data, then adapted on the 500 real demonstrations55.6

When no suitable simulator exists, the evaluator can be trained from sequential rollouts labelled with success or failure. Such data requires no state restoration or branching. In our RoboTwin ablation (A3), Rollout-Q, trained only on sequential rollouts, already improves over the base policy. On a real robot, this data comes from ordinary policy execution with automatic or human success labels, the same kind of data that real-world RL collects. We have not yet evaluated this fully real-world route and will discuss it in the limitations (App. A).

Q1(b): How is the Q-evaluator adapted on real-robot demonstrations without MCTS labels or sibling candidates?

A1(b). Return targets are computed directly from the recorded demonstrations. We use 100 successful expert trajectories for each of the five physical tasks (500 in total). Each trajectory receives reward 1 at its final step and 0 elsewhere, and each state–action sample’s target is this terminal reward discounted by 0.99 per remaining step. No MCTS or unexecuted candidates are involved.

Adaptation is supervised return regression on individual samples, so it needs no sibling candidates. Each sample contains the observation, instruction, demonstrated action, and its return target. Starting from the simulation-MCTS-trained Q-model, we jointly fine-tune its LoRA adapters and five value heads on all 500 trajectories with Smooth-L1 loss (learning rate 5×10−65\times10^{-6}, batch size 64, 5 epochs). The small learning rate and short schedule keep the model close to the pretrained evaluator, so adaptation recalibrates it to real observations while preserving the ranking learned from simulation. We will add these details to Appendix C.2.

AuthorsOpenReview ↗

Author Response (Part 2/2)


Q2: How does manipulation success scale with the candidate action-chunk budget on RoboTwin?

A2. We sweep the number of candidate action chunks N∈{2,4,8,16,32}N\in\{2,4,8,16,32\} on RoboTwin. At each decision, NN candidates are both sampled and scored. All budgets use the same frozen π0.5\pi_{0.5} proposer and Q-model. Following the Table 2 protocol, each budget is evaluated in three runs of the same 50 initial states per task on 12 tasks (150 trials per task, 1,800 per budget); each reported value is total successes divided by 1,800. Table 2 instead samples 32 candidates and clusters them into 16 for scoring, so the N=16N=16 entry here is not directly comparable to Table 2.

Success shows an overall upward trend with NN at modest latency cost. Average SR rises from 63.50% at N=2N=2 to 67.94% at N=32N=32 (+4.44 points). Over the same range, end-to-end latency per decision grows from 124 ms to 223 ms. This extends the test-time scaling of Figure 6 from high-level reasoning to continuous manipulation.

MetricN=2N=2N=4N=4N=8N=8N=16N=16N=32N=32
Task-averaged success rate (%)63.5064.5664.6166.0667.94
End-to-end Act latency (ms)124.1137.0152.7168.0223.2

We will add this RoboTwin SR–latency trade-off curve to Section 5.4.

Q3: Rollout versus MCTS supervision on RoboTwin and implications for simulator-free real-world training.

A3. We add a controlled RoboTwin ablation in which only the source of supervision changes. Rollout-Q is trained on returns from independent sequential policy rollouts, and MCTS-Q on MCTS-backed returns. Both use the same 30 training episodes per task, the same number of training samples (92,728), the same Q-model architecture and training hyperparameters, and the same frozen π0.5\pi_{0.5} proposer. Evaluation follows the Table 2 protocol: with fixed checkpoints, all methods are evaluated in three runs of the same 50 initial states per task, giving 150 trials per method and task. Each entry is total successes divided by 150 (e.g., 37/150 = 24.67% for Rollout-Q on Click Bell); Base and MCTS-Q are from Table 2.

Rollout supervision already improves success, and MCTS adds a further 3.61 points. Rollout-Q raises average SR from 62.72% to 64.17%, and MCTS-Q reaches 67.78%, outperforming Rollout-Q on 11 of 12 tasks.

TaskBase π0.5\pi_{0.5}Rollout-QSVA (MCTS-Q)
Click Bell20.0024.6731.33
Move Pillbottle Pad69.3366.0068.67
Move Card Away81.3384.0082.67
Pick Dual Bottles83.3383.3386.67
Place Bread Skillet70.6774.0078.00
Place Fan67.3374.6776.00
Place Mouse Pad40.0036.0042.00
Press Stapler76.0078.6782.67
Scan Object50.0053.3354.00
Stack Blocks96.0098.0098.67
Stamp Seal50.6750.6756.67
Turn Switch48.0046.6756.00
Average62.7264.1767.78

When a simulator is available, MCTS supervision accounts for most of SVA’s gain on RoboTwin; sequential rollouts are a weaker fallback. Sequential rollouts, which need no state restoration, yield a modest improvement (+1.45 points over the base policy); with the same number of training samples, MCTS yields a stronger evaluator (+5.06 points). This suggests that simulator-free training needs outcome-labelled sequential data rather than resettable branching, as discussed in A1(a). We will add this ablation to the revision.

W2: Minor formatting and LaTeX issues.

A(W2). We will fix the unresolved “Table ??” reference in the Figure 5 caption and check all cross-references, captions, and the compiled PDF.

Summary Response

We thank you for recognizing our well-motivated problem formulation, clear paper structure, and comprehensive experimental evaluation. Search is run offline in simulation and distilled into the evaluator: simulation data is collected automatically at low cost, the real-robot pipeline needed only platform-level alignment and no extra real data, and the simulation-trained evaluator alone raises real-robot success from 42.8% to 50.4% (W1, Q1(a)). Real-data adaptation is return regression on the existing demonstrations, without MCTS labels or sibling candidates (Q1(b)). On RoboTwin, success shows an overall upward trend with the candidate budget (Q2), and MCTS outperforms rollout supervision by 3.61 points with the same number of training samples (Q3). We will add these results to the revision and fix the formatting issues (W2). We hope these clarifications and new evidence fully address your concerns, and we would sincerely appreciate your reconsideration of our work.

Reviewer_4XpyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Search, Value, and Act (SVA), which improves a frozen embodied policy by learning an external action evaluator. Offline Monte Carlo tree search in a resettable simulator generates state-action examples with backed-up empirical returns. A Qwen3.5-0.8B evaluator with LoRA, continuous-action input modules where needed, and five value heads regresses these fixed targets. At deployment, the proposer samples candidate skill sequences or action chunks, and the evaluator selects a candidate using its mean value, head-disagreement penalty, and optional proposal prior. Deployment requires no simulator search.

Validation covers two EmbodiedBench suites with five proposer backbones, 12 RoboTwin tasks with two proposers, four SimplerEnv tasks, and five physical manipulation tasks under two scene conditions. The reported gains include 62.72% to 67.78% for pi0.5 on RoboTwin and 42.8% to 55.6% over the physical trials. The physical experiment uses a proposer fine-tuned on 500 real demonstrations and an evaluator additionally adapted on those same trajectories; it is not a zero-shot simulation-to-real transfer experiment. Ablations and latency measurements support the utility of learned selection and practical batched evaluation, while important questions remain about the generalization claim and the specific advantage of MCTS supervision.

Critical strengths

The evaluator is tested across substantially different proposers and action interfaces. On each EmbodiedBench suite, the same evaluator is reused across all five proposer models, rather than training a separate evaluator for each backbone (Table 1). RoboTwin additionally holds each proposer checkpoint fixed when adding selection, and shows gains for both pi0.5 and FastWAM (Table 2). This makes the proposal/evaluation separation more convincing than a result limited to one policy or a joint policy update.

The physical evaluation and deployment measurements give the method practical substance. The physical comparison uses the same adapted proposer in both conditions, reports integer success counts, and evaluates both uncluttered and distractor scenes. Aggregating Table 4 gives 107/250 successes for the proposer and 139/250 for SVA, with a larger reported gain in distractor scenes. These results support practical evaluator-guided selection without simulator access during execution.

Critical weaknesses

The evidence for unseen-task generalization and preservation of generalist capability is insufficiently specified. These are central claims in the abstract and introduction, but Tables 8-9 describe training/evaluation separation mainly through episode ranges and seeds. The manuscript does not explain which task identities, instruction compositions, objects, or other distributions are excluded from evaluator training. It trains separate evaluators for the benchmark suites and trains the RoboTwin evaluator jointly on all 12 evaluated tasks. The physical evaluator is adapted on the same 500 demonstrations used for the physical proposer. These are valid settings for held-out-episode performance, but they do not by themselves establish retention of broad capability under an evaluator trained on a narrower distribution. Freezing the proposer preserves its parameters; the deployed composite policy can still reject its good actions on unfamiliar tasks. Clarifying the actual held-out task structure, or narrowing the claim to the demonstrated setting, is important to evaluating the paper's main motivation.

The distinctive benefit of search-derived supervision is not isolated well enough. The complete system outperforms useful baselines, but frozen-policy value reranking is already the premise of V-GPS, and sampling plus learned verification is the premise of RoboMonkey, both cited by the paper. Thus the key empirical question is what MCTS contributes beyond simpler supervision at a comparable resource budget. The reported no-MCTS ablation improves EB-Navigation from 50.83% to 56.11% with MCTS, but the manuscript does not specify matched simulator transitions, rollout count, retained training examples, or collection compute for these variants. It also does not provide this component comparison on the manipulation tasks. Appendix E.1 describes V-GPS and TACO conceptually but omits their training data/checkpoint details and candidate budgets, making the 3.06- and 2.45-point RoboTwin advantages difficult to attribute specifically to better search supervision. The paper establishes promising performance for the combined pipeline more strongly than a controlled advantage for its distinguishing training mechanism.

Questions for authors

What exactly is held out from evaluator training in each benchmark: initial states, instructions, object arrangements, task identities, or task compositions? Please map the “unseen tasks” claim to those exclusions and clarify what evidence supports preservation of generalist behavior beyond the evaluator's training distribution. An explicit task-disjoint analysis would be useful if available; it should not be conflated with the already reported seed/episode split.

For the no-MCTS ablation and the V-GPS/TACO comparisons, what training data, simulator-interaction budgets, candidate sets/counts, and tuning procedures were used? Does the MCTS advantage persist when rollout-only collection and search receive the same simulator-transition or collection-time budget? This bears directly on whether search provides the claimed benefit rather than simply more or different supervision.

How many independent runs and trials underlie Tables 1-3, and how are they averaged? Table 9 states 50 RoboTwin evaluation episodes per task, whereas Table 2 contains success rates such as 69.33% and 81.33%, which require additional averaging or a different denominator. Please identify the repetitions, random seeds, whether initial states are paired, and variability of the main differences. Also specify the return targets and training procedure used to adapt the physical evaluator from demonstrations in Appendix C.2; this is necessary to understand what additional supervision produced the real-world result.

6 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)


W1 & Q1. Generalization scope and preservation of generalist capability in the composite policy.

A1. We evaluate generalization to held-out task instances within each benchmark. Training and evaluation use disjoint instances with shared task objectives; for example, Click Bell instances differ in initial scene configurations and object placements. This follows the within-benchmark evaluation practice used in prior VLA-steering work [1-3]; for example, TACO [1] evaluates each RoboTwin task on 100 randomly generated scenes with distinct seeds. We will specify the held-out factors in Tables 8-9 and use “held-out task instances” consistently throughout the paper.

A single evaluator serves multiple task families within each benchmark. One Q-model is jointly trained across all evaluated RoboTwin tasks; likewise, each EmbodiedBench suite uses one evaluator across its task categories. This supports multi-task action selection without training a separate evaluator or further updating the proposer for each task family. Table 1 additionally demonstrates reuse of the same evaluator across five proposer backbones within each EmbodiedBench suite.

Freezing the proposer alone does not guarantee preservation of the composite policy’s generalist capabilities. We agree that the evaluator can misrank good proposals on unfamiliar tasks even when the proposer is unchanged. Our main results show improved average performance on held-out instances across the evaluated tasks.

We add a task-disjoint check on three RoboTwin tasks the evaluator never saw. place_shoe, place_object_scale, and shake_bottle are excluded from all search and evaluator-training data. We use the same frozen π0.5\pi_{0.5} checkpoint, the 12-task evaluator without further training, and the Table 2 protocol (three runs of 50 episodes per task, 150 trials per task and method). Average success is 86.7% for both π0.5\pi_{0.5} and π0.5\pi_{0.5}+SVA.

Unseen taskπ0.5\pi_{0.5}π0.5\pi_{0.5}+SVA
Place Shoe87.388.7
Place Object Scale73.371.3
Shake Bottle99.3100.0
Average86.786.7

[1] Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach. arXiv:2512.02834, 2025.

[2] Steering your generalists: Improving robotic foundation models via value guidance. CoRL 2024.

[3] RoboMonkey: Scaling test-time sampling and verification for vision-language-action models. CoRL 2025.


W2 & Q2. Budget-controlled MCTS ablation and distinctions from prior action evaluators.

A2. We have conducted a new RoboTwin ablation to isolate the additional benefit of MCTS-derived supervision. Both variants use the same 30 training episodes per task, the same number of retained training samples (92,728), identical Q-model architecture, training hyperparameters, and tuning procedures, and the same frozen proposer checkpoint. Evaluation uses identical test episodes, candidate counts, and action-selection settings. Only the supervision-generation procedure changes: ordinary sequential rollout returns versus MCTS-backed returns. MCTS-Q achieves 67.78% versus 64.17% for Rollout-Q (+3.61 percentage points), with higher success on 11 of 12 tasks. Results average three runs of 50 evaluation episodes per task.

TaskBase policyRollout-QSVA (MCTS-Q)
Click Bell20.0024.6731.33
Move Pillbottle Pad69.3366.0068.67
Move Card Away81.3384.0082.67
Pick Dual Bottles83.3383.3386.67
Place Bread Skillet70.6774.0078.00
Place Fan67.3374.6776.00
Place Mouse Pad40.0036.0042.00
Press Stapler76.0078.6782.67
Scan Object50.0053.3354.00
Stack Blocks96.0098.0098.67
Stamp Seal50.6750.6756.67
Turn Switch48.0046.6756.00
Average62.7264.1767.78

SVA’s core contribution is learning a consequence evaluator to improve frozen-policy action selection. MCTS instantiates the Search stage as an efficient way to collect return-labelled data; the Q-evaluator can also learn from independent rollouts. Rollout-Q’s improvement over Base (64.17% versus 62.72%) supports this flexibility, while the controlled MCTS gain demonstrates the additional value of search supervision.

SVA differs from prior action evaluators in its supervision and training recipe. V-GPS [2] learns a Q-function through offline RL, while RoboMonkey [3] trains an action preference model using synthetic preference data. SVA instead trains a consequence-aware Q-evaluator directly on collected long-horizon returns, providing a simple, policy-agnostic evaluation interface for both discrete skills and continuous action chunks. We will explicitly report the data and interaction budgets and expand Appendix E.1 with the V-GPS/TACO training data, checkpoints, candidate budgets, and tuning details.

AuthorsOpenReview ↗

Author Response (Part 2/2)


Q3. How are the success rates in Tables 1–3 averaged across evaluation runs (evaluation repetitions, random seeds, initial-state pairing, and uncertainty)?

A3. Tables 1-3 average three evaluation runs with fixed model checkpoints. For RoboTwin (Table 2), each run evaluates 50 episodes per task, giving 150 trials per method and task. Success rates are averaged across the three runs, equivalently calculated as total successes divided by 150; for example, 104/150 gives 69.33%. The episode counts in Table 9 are per run: 50 per task for RoboTwin, 20 per split for EB-Habitat, 24 per split for EB-Navigation, and 9 per task for SimplerEnv.

All methods are evaluated on the same initial states per task within each run, and these states are reused across all three runs. Thus, RoboTwin's 150 trials per method and task are repeated executions of the same 50 test episodes. We will list the environment-initialization and action-sampling seeds separately in the appendix.

We report variability as the sample standard deviation across the three evaluation runs. The table below summarizes the RoboTwin macro-average success rates as mean ± SD (%). For each method, we first average success rates over the 12 tasks within each run, then compute the mean and SD of these three run-level averages. The SD therefore measures evaluation variability with fixed checkpoints and fixed test initial states.

MethodSuccess Rate (mean ± SD, %)
π0.5\pi_{0.5}62.72 ± 4.58
π0.5\pi_{0.5}+V-GPS64.72 ± 1.06
π0.5\pi_{0.5}+TACO65.33 ± 2.24
π0.5\pi_{0.5}+SVA67.78 ± 0.98
FastWAM77.72 ± 0.10
FastWAM+SVA79.72 ± 1.67

Paired differences. Because all methods share the same initial states within a run, we also compute each method’s difference from the comparator within the same run, then take the mean and SD of the three run-level differences (percentage points).

ComparisonMean ± SD
π0.5\pi_{0.5}+SVA − π0.5\pi_{0.5}+5.06 ± 5.55
π0.5\pi_{0.5}+SVA − π0.5\pi_{0.5}+V-GPS+3.06 ± 1.67
π0.5\pi_{0.5}+SVA − π0.5\pi_{0.5}+TACO+2.44 ± 2.92
FastWAM+SVA − FastWAM+2.00 ± 1.64

We will report these paired differences, the evaluation protocol, and task-level mean ± SD results in the revised captions and appendix.


Q4. How are real-robot return targets constructed and used to adapt the evaluator?

A4. Return targets are computed directly from successful demonstrations. We use 100 expert trajectories per task across five physical tasks, totalling 500 trajectories. Each trajectory receives reward 1 at its final recorded step and 0 at earlier steps. Starting from the final reward, we work backward through the trajectory, applying the discount factor 0.99 at each step to obtain a fixed return target for each demonstrated state-action sample. This requires neither tree search on the physical robot nor outcome labels for unexecuted candidate actions.

We adapt the simulation-trained evaluator by supervised regression. Each training sample contains the recorded observation, task instruction, demonstrated action, and its precomputed return. Starting from the simulation-MCTS-trained Q-model, we update its LoRA adapters and five value heads by minimizing the Smooth-L1 loss between the predicted Q-values and these returns, using a learning rate of 5×10−6,5 × 10⁻⁶, batch size 64, and 5 epochs. Adaptation is joint across all 500 trajectories.

The π0.5\pi_{0.5} base policy and SVA use the same real-data budget and evaluation protocol. The proposer is fine-tuned on 100 demonstrations per task (500 trajectories in total), and SVA’s evaluator is adapted on the same trajectories. Task settings, scene conditions, and evaluation trial counts are held constant, and all models remain fixed during evaluation. SVA uses no additional real-robot trajectories beyond this common adaptation budget. We will add the reward assignment, return computation, adaptation settings, and matched-data protocol to Appendix C.2.


Summary Response

We thank you for your constructive review. In response, we (1) clarify that evaluation is on held-out task instances, add a task-disjoint check on three unseen RoboTwin tasks (86.7% for both π0.5\pi_{0.5} and π0.5\pi_{0.5}+SVA), and limit the generalist-preservation claims accordingly (W1, Q1); (2) add a controlled RoboTwin ablation showing a 3.61-point gain from MCTS supervision over rollout supervision (W2, Q2); (3) specify the three-run protocol and report paired run-level differences (+3.06 ± 1.67 over V-GPS, +5.06 ± 5.55 over the π0.5\pi_{0.5} base policy; Q3); and (4) describe the return-regression adaptation of the physical evaluator on the same 500 demonstrations used for the proposer (Q4). We will add the detailed protocols, uncertainty estimates, and adaptation settings to the revision, and would appreciate your reconsideration in light of these results and clarifications.

Thank you very much for your response. While some of my concerns have been addressed, several significant questions remain unresolved: How many actual simulator steps or how much data-collection time were consumed by MCTS-Q versus Rollout-Q, respectively? Does the performance advantage persist when this budget is equalized? Please provide the data, checkpoints, candidate budget, and hyperparameter tuning configurations used for V-GPS/TACO.

Specifically, the authors matched the training setup—using 30 training episodes per task and 92,728 stored samples—along with the model, training, and evaluation configurations. The fact that MCTS-Q outperforms Rollout-Q by 3.61 percentage points and performs better on 11 out of 12 tasks certainly strengthens the evidence. However, since MCTS involves repeated branching, rollouts, and resets (or replays) from the same starting state, an identical number of episodes or stored samples does not necessarily imply that the same amount of environment interaction or computational cost was incurred to generate that data.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.