- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper examines whether sampling more actions at test time improves policy performance and whether world models help select successful actions. The authors introduce a diagnostic benchmark across several policy families and simulation environments. They measure the potential gain from perfect action selection, the actual gain achieved by practical selectors, and whether dispersion among sampled actions predicts failure. They evaluate action consensus, value-based selection, and VLM judgments of predicted futures, using simulator-generated futures to diagnose selection failures. First, they find potential gains in some settings, but practical selectors do not consistently realize them. Second, longer ground-truth rollouts improve ranking in some cases, while longer predicted rollouts do not reproduce this benefit in their Motus experiment. Finally, they find that the usefulness of action dispersion for failure prediction varies by setting and execution stage.
Critical strengths
None
Critical weaknesses
- The contribution of this paper is vague. I am unsure exactly what the authors want to claim, because the key takeaways of Sections 3, 4, and 5 do not lead to a clear overarching conclusion. They largely show that sampling potential, selection effectiveness, and uncertainty informativeness depend on the specific experimental setup. This dependence is unsurprising, so the new generalizable insight remains unclear.
- The selection baselines are too narrow, and the proposed future directions do not adequately consider prior work. The authors already discuss several selection approaches beyond VLM-based judging in the related work, including a relative action critic (VLA-ATTC), policy-derived selection signals (MG-Select), and geometric consistency (GeoBoN), but do not evaluate these approaches at all. Simply calling for stronger native value heads or explicitly trained external verifiers is therefore not a sufficiently novel future direction.
Questions for authors
Q1. Can the authors identify any consistent trend across the evaluated settings in selection effectiveness (Section 4) or uncertainty informativeness (Section 5)? The current takeaways seem unsurprising, and it is unclear whether the positive findings extend beyond the specific experimental setups.
Q2. Can the authors evaluate the rollout-refined Cosmos Policy planning model mentioned in Section 4.2 on the proposed benchmark, or explain why it cannot be included? This seems an important reference for value-based selection, since the current experiment only uses native values from the action-proposal checkpoint.
Q3. Can the proposed benchmark be extended to any real-world robot evaluations to test whether its takeaways hold outside simulation?