- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how to allocate model-generated experience in world-model-based VLA policy optimization. It proposes U-GROW, a sampling layer that modifies the starting-state distribution of short-horizon rollouts without changing the policy optimization objective. Policy uncertainty is estimated from action disagreement across different denoising step counts under the same state and initial noise. The method truncates failed trajectories at their first high-uncertainty state when constructing the prioritized pool, then combines sampling from high-uncertainty retained states with uniform sampling over all offline states. The authors investigate the prioritization criterion through policy-switching experiments and evaluate the method on RoboTwin 2.0, LIBERO, and four real-world manipulation tasks. Under the same world model and model-generated transition budget, the reported results show improved sample efficiency and final success rates compared with uniform rollout-start sampling.
Critical strengths
- A practical problem addressed with a simple method. The paper studies how to allocate model-rollout budgets in world-model-based VLA policy optimization. It improves experience allocation by modifying the rollout-start distribution without changing the policy optimization objective. This makes the method straightforward to integrate into existing training pipelines.
- Controlled comparisons and evaluation across settings. The main comparison with VLA-MBPO shares the world model, initial policy, and model-generated transition budget, helping assess the incremental benefit of the sampling strategy. Evaluations in simulation, on real robots, and under distribution shifts provide supporting evidence for its applicability.
Critical weaknesses
-
The connection between the uncertainty measure and training utility is insufficiently established. Action disagreement across denoising step counts may reflect numerical approximation error rather than greater potential for policy improvement. The policy-switching experiment shows that intervention by a stronger policy can be more effective at these states, but does not directly establish that allocating additional training rollouts to them is more valuable. This gap weakens the justification for the central prioritization criterion.
-
The experiments do not sufficiently explain the mechanism behind the gains. The component ablations support the proposed combination, but are concentrated on one task and do not establish whether the uncertainty measure outperforms other reasonable prioritization criteria. It therefore remains unclear whether the gains primarily arise from selecting decision-sensitive states, downweighting failure tails, or a more general redistribution of rollout starts. This limits confidence in the method’s scope of applicability.
Questions for authors
- What theoretical justification or direct empirical evidence distinguishes numerical discretization error from genuine policy-improvement potential in the proposed uncertainty measure?
- The policy-switching experiment measures the benefit of intervention by a stronger policy. Is there evidence that, under a matched training budget, rollouts starting from high-uncertainty states produce greater policy improvement than rollouts starting from other states?
- Have you compared alternative rollout-prioritization criteria, such as value error, world-model uncertainty, or randomized priorities? Can these comparisons establish that the gains depend on the proposed uncertainty measure rather than nonuniform sampling more generally?
- After including the cost of recomputing uncertainty over all candidate states at each iteration, does U-GROW still require less total training time or fewer GPU-hours than uniform sampling to reach the same success rate?
I thank the authors for their detailed response and additional experiments, which address my primary concerns regarding computational overhead, signal suitability, and baseline comparisons. Given that, I am raising my rating.