- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on test-time sampling for flow-based VLAs while adaptively deciding when additional sampling is needed. The authors propose VeriVLA, which uses a two-axis trigger based on temporal variation within action chunks and variance across two sampled chunks to decide when to use additional computation. When triggered, the proposed method samples more candidate actions and uses a JEPA-based latent world model to predict future latent states and delta task progress. It then executes the candidate with the highest predicted cumulative progress. The verifier is trained with Robometer-derived progress labels, while the VLA remains frozen at test time. The authors report an increase in average completion score from 50.8% to 65.5% on three real-world tasks, along with success-rate gains on RoboTwin 2.0 and LIBERO benchmarks.
Critical strengths
-
Clear problem setup and well-motivated approach: Flow-based VLAs are inherently stochastic when sampling actions, and sampling a suboptimal action may lead to task failure. The authors observe that failures tend to occur in high-variance segments and propose a trigger to identify these critical moments and allocate additional test-time sampling.
-
Trigger component is easy to understand and straightforward: The authors use variability in the policy’s sampled actions to identify potentially important moments by measuring temporal variation within each of two sampled action chunks and disparity between the chunks. These two axes provide a simple but quite reasonable signal for deciding when to perform additional sampling.
Critical weaknesses
-
The necessity of the trigger component is somewhat unclear: One of my main concerns is that I am not fully convinced why adaptive test-time verification is necessary, as the trade-off between performance gains and latency from additional test-time computation remains vague. Although the authors provide a trigger-rate sweep in Figure 6 and compare latency with always-on verification in Figure 5, they do not show the relationship among trigger rate, success rate, and latency within the same experimental setup.
-
The reliability of the delta-progress labels used to train the verifier is vague: I believe that test-time verification performance depends heavily on verifier quality. This paper uses absolute progress labels automatically generated by Robometer to train the latent world model to predict delta progress. However, the accuracy of these Robometer labels is not independently validated on the evaluated tasks, and it remains unclear whether they provide reliable supervision in various robot tasks and environments.
Questions for authors
Q1. Can the authors provide a performance-latency plot while varying the trigger threshold under the same experimental setup as the main Tables, including always-on verification? Figures 5 and 6 report latency and trigger-rate sweeps separately; a joint plot would help clarify the benefit of adaptive verification.
Q2. Can the authors examine the sensitivity of the verifier to the Robometer-derived progress labels? First, how does performance compare with alternative labels, such as other labeling methods or task-specific heuristics based on gripper states? Second, can the authors compare the Robometer progress labels with human-annotated progress labels on a validation dataset not used for training the verifier? Finally, do these labels remain reliable for higher-DoF embodiments, such as robot arms with dexterous hands or humanoids?
Q3. Can the authors provide an ablation of the progress loss comparing the Huber loss alone, the rank loss alone, and their combination? I would like to know whether the rank loss indeed contributes beyond the Huber loss.