acceptodds
Under review as a conference paper at ICLR 2027

DiVeR: Decision-Critical Verifier Learning for VLA Test-Time Scaling

Abstract

Scaling robot data and model capacity has improved Vision-Language-Action (VLA) policies, but further progress is constrained by the high cost of robotic data. Verifier-guided test-time scaling offers an efficient alternative by sampling multiple action candidates and selecting the one most likely to lead to task success at inference time. Existing classification-based verifiers learn from trajectory-level outcomes but treat all visited states equally, even though their value for candidate discrimination can vary across a trajectory. At many states, plausible actions are similar and provide limited discrimination signal, while only a sparse set of decision-critical states admits meaningfully different actions that can substantially affect downstream outcomes. To address this, we propose DiVeR, which estimates decision criticality from the dispersion of sampled action representations. DiVeR then uses this signal to reweight verifier learning toward states where action selection is most consequential, without requiring step-level annotations or additional environment interaction. Across LIBERO, RoboCasa, and real-world experiments on a Franka Research 3 robot, DiVeR consistently improves task success through more effective verifier-guided action selection, while adding negligible verifier inference overhead.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action model, test-time scaling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PQ9HOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DIVER, an importance-weighted verifier training framework based on decision-critical states to improve the effectiveness of Vision-Language-Action (VLA) models during test-time scaling (TTS). The authors observe that not all states along a trajectory are equally important during execution. Notably, they make the clever observation that at routine states (such as free-space motion), candidate actions exhibit lower variance in their action-expert hidden representations, whereas at decision-critical states, this variance is significantly higher. Leveraging this variance for importance weighting, DIVER encourages the verifier to focus more on decision-critical states during training. Extensive experiments demonstrate that DIVER achieves superior performance, and ablation studies clearly validate the effectiveness of the proposed importance weighting mechanism.

Critical strengths

1.The motivation of this paper is clear, and the proposed method is simple yet effective. The experimental evaluation is comprehensive, and the ablation studies clearly demonstrate the contribution of each component.

2.The idea of leveraging internal action-expert representations to estimate state criticality and discriminate candidate actions is clever and well-motivated.

Critical weaknesses

1.Trajectory-level supervision appears overly coarse. For instance, in a multi-stage manipulation task, if the robot successfully completes the first stage but fails during the second stage, the current training paradigm assigns a terminal label of yt=0y_t = 0 (failure) to the entire trajectory. This penalizes the successful transitions in the first stage and leads to inefficient data utilization and noisy supervision.

2.While the distinction in action-representation variance between routine states and decision-critical states is intuitive for in-distribution scenarios, it remains unclear whether this property still holds in out-of-distribution (OOD) scenarios. Intuitively, under OOD conditions, the variance of sampled actions might be generally elevated across the trajectory due to epistemic uncertainty, even at routine states. However, the paper lacks an analysis of this aspect.

Questions for authors

1.Have the authors considered adopting more fine-grained supervision or training schemes—for example, leveraging expert demonstrations [1, 2] or incorporating sub-task decomposition [3]?

2.The intuition behind decision-critical states shares conceptual similarities with [4]. Since candidate actions at routine states exhibit low variance and barely affect task success, have the authors explored identifying routine states on the fly and adaptively disabling TTS during those phases to accelerate inference?

3.Does the distinction in action-representation variance between routine and decision-critical states still hold in OOD scenarios (where action variance might be generally elevated)? Could the authors provide some analysis or discussion on this?


Reference

[1]RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models

[2]V-GPS: Value-Guided Policy Search / RL with Value-Guided Verification

[3]ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learning

[4]SkiP: When to Skip and When to Refine for Efficient Robot Manipulation

Reviewer_xg1pOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies verifier-guided Best-of-N action selection for test-time scaling of VLA policies. It argues that lightweight classification-based verifiers, trained on trajectory-level success/failure labels, weight all visited states uniformly. In practice most states are routine, where sampled candidates are nearly identical, and only a sparse set of decision-critical states, where candidates diverge, matters for selection. VLM-based verifiers avoid this issue but are too slow for real-time control, and verifier-free methods are not grounded in task outcomes. DIVER estimates per-state criticality as the variance of action-expert representations across candidates sampled from the frozen VLA. It uses this score to reweight the BCE objective of an MLP verifier on those representations, without step-level labels or extra environment interaction. With π0\pi_0 and π0.5\pi_{0.5} on LIBERO-Long, RoboCasa, and a real Franka robot, DIVER generally outperforms verifier-free and learned selection baselines and a uniformly trained verifier, at far lower cost than a VLM-based verifier.

Critical strengths
  1. Simple, well-motivated, and easy to adopt. Outcome-supervised verifiers spend most of their capacity on routine states, and the paper addresses this with a criticality signal taken from the frozen policy’s own candidate dispersion. The signal needs no step-level labels and no additional environment interaction, so it can be added to existing generator–verifier pipelines with little effort. The verifier is a small MLP on action-expert features, which makes its scoring cost negligible relative to candidate sampling. This matters for real-time control and is the main reason I see value in the paper.

  2. Supporting analyses beyond the main tables. The component ablation (Table 4) separates the effect of the input representation from that of the weighting. The input, layer, and token ablations, the correlation with rollout-based oracle criticality (App. C.1), the cross-category transfer results (Fig. 5), and the OpenVLA experiment (App. C.2) make the story more credible than a single benchmark table would.

Critical weaknesses
  • The significance of the performance gains is unclear. The core contribution is reweighting of an existing outcome-based verifier objective, which is essentially SVM in action-expert space with per-state weights. The paper’s value therefore depends on this weighting producing reliable improvements, and the evidence for that is thin in several places.

    • On LIBERO-Long with π0.5\pi_{0.5}, the base policy already reaches 93.5%. DIVER’s margin over the best baseline is only 0.3–0.5 pp across all NN, which is roughly one episode out of 200 evaluation episodes per seed. With π0\pi_0 at N=4N=4, DIVER is below TACO.

    • Standard deviations are reported only for DIVER, not for the baselines, and no significance tests are given. It is therefore hard to tell whether the differences exceed seed and evaluation noise. The same applies to the OpenVLA result (55.5 vs. 55.0 for SVM).

    • On the real robot (24 trials per task, apparently a single run), the gap over the uniform verifier is one to two successful episodes per task, or 69 vs. 63 successes out of 96 in total. No other baselines such as SVM or MG-Select are evaluated. The headline +13.6% is measured against N=1N=1, so it mixes the benefit of Best-of-N sampling itself with the benefit of the proposed weighting.

  • Computational overhead is not meaningfully compared across methods. Test-time scaling is about converting compute into performance, so the cost side needs the same care as the accuracy side.

    • Training time: DIVER draws K=32K=32 additional policy samples at every visited training state to compute utu_t, over datasets of up to hundreds of thousands of steps (Table 8). This cost is not reported. It is also not compared with SVM or TACO, which need no extra sampling, or with spending the same compute on more rollouts for a uniform verifier.

    • Inference time: The paper reports DIVER’s verifier latency and generic candidate-sampling latency (Fig. 8). It does not report end-to-end per-step latency for each selection method at matched NN, even though the methods differ in cost (for example, MG-Select requires an additional masked reference set and SVM requires a CNN image encoder).

    • The only cross-method cost comparison (Table 6) is against a VLM-based verifier operating on simulator-branched trajectories, which is not a deployable competitor. The relevant comparison is against the lightweight baselines in Tables 1–2, ideally as success-vs-compute curves.

Questions for authors

Q1. Statistical significance. Please report mean ± std or confidence intervals for all baselines in Tables 1, 2, and 12, and for Base and Uniform on the real robot. Could you also add at least one external baseline, such as SVM or MG-Select, to the real-robot evaluation?

Q2. Computational cost. (a) What is the wall-clock or GPU-hour cost of constructing the weighted dataset with K=32K=32, compared with training a uniform verifier? (b) What is the end-to-end per-step latency of each selection method at the same NN? (c) Does DIVER still outperform the baselines under matched compute, for example when they use a larger NN or when the uniform verifier is trained on more rollout data?

Q3. Can action-representation dispersion separate behaviorally distinct situations? Distances in the action-expert space may not always reflect behaviorally consequential divergence. Under visual occlusion or perceptual ambiguity, candidates may be dispersed because the policy is uncertain about the scene, not because the choice matters. This would inflate utu_t at states that are not decision-critical. Conversely, at a truly critical state the policy may collapse to a single, possibly wrong, mode and yield a low utu_t. The correlation with oracle criticality is moderate, and the scatter in Fig. 7 appears to include many states with near-zero oracle gain across a wide range of utu_t. How does DIVER distinguish these cases? It would help to (i) analyze failure cases or stratify Fig. 7 by occlusion and ambiguity conditions, and (ii) compare utu_t with simpler proxies such as raw action-chunk dispersion. The isotropic mean per-coordinate variance could be dominated by behavior-irrelevant dimensions.

Q4. Benchmark choice. Why is the LIBERO evaluation limited to LIBERO-Long? The motivation is that critical states are sparse within long trajectories. Results on LIBERO-Spatial, Object, and Goal would show whether the gains persist or shrink on shorter-horizon suites, and would more directly test whether the benefit comes from criticality weighting.

Q5. Hyperparameter tuning and sensitivity. Table 7 states that hyperparameters were tuned on RoboCasa with π0.5\pi_{0.5}, which is also the setting for the main RoboCasa result and all ablations. Was tuning done on a validation split separate from the 20 evaluation episodes per task? Please also show sensitivity to β\beta, where β=0\beta=0 recovers uniform training. With β=0.5\beta=0.5 and clipping at c=2.5c=2.5, the reweighting appears relatively mild.

S1. Fine-grained, contact-rich assembly benchmarks such as FMB or DexMimicGen seem like the setting where criticality weighting should matter most, because success there hinges on short and precise alignment or insertion phases. Showing larger gains in such settings would substantially strengthen the central claim. A per-task breakdown for RoboCasa’s insertion task (coffee setup mug) could serve as an initial indicator.

Reviewer_FqugOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DiVeR improves Best-of-N selection for frozen VLA policies by changing how the verifier is trained. At the training stage, it samples action candidates from a frozen policy, measures their dispersion in the action expert's hidden representation space, and converts within-trajectory standardized dispersion into clipped weights. A lightweight MLP is then trained with weighted binary cross-entropy using only trajectory-level success and failure labels. At inference, it ranks candidate action chunks and executes the highest-scoring candidate. The paper evaluates the proposed method on LIBERO-Long, RoboCasa, and four real-robot tasks, with an additional OpenVLA experiment. The most informative ablation fixes the action representation: weighting improves RoboCasa success from 34.9% to 38.1% and from 50.6% to 54.7%, respectively.

Critical strengths
  1. The central innovation is isolated reasonably well. Table 4 fixes the action-expert features and compares uniform against weighted training. Table 3 provides a corresponding real-robot control with a reasonable gain. These comparisons are consequential because they show that the result is not explained solely by replacing image and raw-action inputs with stronger policy features.
  2. The method is practical and has supporting evidence across settings. It reuses a frozen policy, requires no step-level outcome annotation, and adds a small verifier at inference. The simulation and real-robot results are complemented by the branching-rollout analysis in Appendix C, which directly tests whether the proposed proxy tracks the benefit of choosing among candidates.
Critical weaknesses
  1. Baseline coverage is incomplete. The paper omits recent direct-action verifiers such as CoVer and UF-OPS. According to the review guidance, CoVer is considered concurrent work and therefore should not, in itself, be a reason for rejection. But I still encourage the author to provide more comparison to similar works. Appendix E describes TACO’s RND variant rather than its main CFN estimator, weakening the comparison.
  2. The criticality proxy may inherit single-policy biases. Weights are computed from one frozen policy’s candidates and representations, so their reliability depends on that policy’s sampling and representation biases. The design of reusing the policy's backbone as the verifier also makes the verifier rely on the policy feature extraction. Thus, the entire system relies on having a good initial policy and may not work well with a policy that initially performs poorly.
Questions for authors
  1. Appendix E appears to describe the RND variant rather than TACO’s main CFN estimator. Could you clarify which version was used and why? Comparing against the main estimator would make the baseline comparison more convincing.
  2. Could you compare with a representative direct-action verifier, such as CoVer or UF-OPS, using the same candidate pool and comparable training budgets?
  3. Since the weights come from one frozen policy, how sensitive are they to that policy’s sampling and representation biases? For example, would different candidate samples or policy checkpoints substantially change which states receive high weights?
  4. Could you compare representation variance against raw-action variance and shuffled weights while keeping the verifier features fixed? This would help distinguish meaningful criticality estimation from a more general benefit of sample reweighting.
  5. Could you clarify the validation/evaluation split, what varies across the three seeds, and what the error bars represent? Reporting uncertainty for the main baselines and Uniform ablation would help assess whether the smaller improvements are robust.

Comments on the paper

1 comment

Thank you for the detailed responses and additional clarifications. They have fully addressed my concerns, and I am happy to raise my score.

I have a few follow-up questions.

  1. Regarding W1 and Q1, is there a specific reason why only the N=16 results are reported for LIBERO, while the RoboCasa results are averaged across all settings? Looking at the full results in the paper, the gains still do not seem consistent. It also looks as though the higher aggregated performance comes mainly from a few tasks that flip between zero and non-zero success rates. Could you identify, for each task and each N, the cases where DiVeR does not improve performance, and explain why it fails in those cases? It would also help to see how the aggregated performance changes when these extreme cases are excluded.

  2. Data construction takes about 5 additional hours, which adds compute time and GPU usage on top of inference. MG-Select, in contrast, only uses the GPU at inference time. Are there ways to reduce this data construction cost? For example, could the data be constructed once and shared across multiple tasks, so the cost is amortized?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.