acceptodds
Under review as a conference paper at ICLR 2027

Prioritized Rollouts for Efficient World Model-based Vision-Language-Action Policy Optimization

Abstract

Vision-Language-Action (VLA) models have emerged as a powerful paradigm for embodied intelligence, but fine-tuning them with reinforcement learning (RL) remains constrained by the cost of real-world robot interaction. Model-based reinforcement learning (MBRL) reduces this cost by using a learned world model to generate rollouts for policy optimization. However, it becomes computationally expensive as VLA policies and world models scale. Existing methods typically treat states equally, overlooking substantial differences in their utility for policy improvement. In this paper, we show that policy uncertainty helps identify states with greater potential for policy improvement. The policy exhibits high uncertainty at only a small subset of states, often during decision-sensitive stages where small action differences can alter task outcomes, suggesting that policy improvements at these states could be particularly valuable. Building on these findings, we introduce U-GROW, a lightweight, plug-and-play sampling layer that directs more model rollouts to these informative states. By modifying only the branched-start distribution, U-GROW can be integrated into existing MBRL pipelines without changing the policy optimization objective. Experiments in both simulated and real-world manipulation tasks demonstrate the efficiency and effectiveness of U-GROW, supporting the use of policy uncertainty to guide experience generation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Intelligence, Model-based Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NE5HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to allocate model-generated experience in world-model-based VLA policy optimization. It proposes U-GROW, a sampling layer that modifies the starting-state distribution of short-horizon rollouts without changing the policy optimization objective. Policy uncertainty is estimated from action disagreement across different denoising step counts under the same state and initial noise. The method truncates failed trajectories at their first high-uncertainty state when constructing the prioritized pool, then combines sampling from high-uncertainty retained states with uniform sampling over all offline states. The authors investigate the prioritization criterion through policy-switching experiments and evaluate the method on RoboTwin 2.0, LIBERO, and four real-world manipulation tasks. Under the same world model and model-generated transition budget, the reported results show improved sample efficiency and final success rates compared with uniform rollout-start sampling.

Critical strengths
  1. A practical problem addressed with a simple method. The paper studies how to allocate model-rollout budgets in world-model-based VLA policy optimization. It improves experience allocation by modifying the rollout-start distribution without changing the policy optimization objective. This makes the method straightforward to integrate into existing training pipelines.
  2. Controlled comparisons and evaluation across settings. The main comparison with VLA-MBPO shares the world model, initial policy, and model-generated transition budget, helping assess the incremental benefit of the sampling strategy. Evaluations in simulation, on real robots, and under distribution shifts provide supporting evidence for its applicability.
Critical weaknesses
  1. The connection between the uncertainty measure and training utility is insufficiently established. Action disagreement across denoising step counts may reflect numerical approximation error rather than greater potential for policy improvement. The policy-switching experiment shows that intervention by a stronger policy can be more effective at these states, but does not directly establish that allocating additional training rollouts to them is more valuable. This gap weakens the justification for the central prioritization criterion.

  2. The experiments do not sufficiently explain the mechanism behind the gains. The component ablations support the proposed combination, but are concentrated on one task and do not establish whether the uncertainty measure outperforms other reasonable prioritization criteria. It therefore remains unclear whether the gains primarily arise from selecting decision-sensitive states, downweighting failure tails, or a more general redistribution of rollout starts. This limits confidence in the method’s scope of applicability.

Questions for authors
  1. What theoretical justification or direct empirical evidence distinguishes numerical discretization error from genuine policy-improvement potential in the proposed uncertainty measure?
  2. The policy-switching experiment measures the benefit of intervention by a stronger policy. Is there evidence that, under a matched training budget, rollouts starting from high-uncertainty states produce greater policy improvement than rollouts starting from other states?
  3. Have you compared alternative rollout-prioritization criteria, such as value error, world-model uncertainty, or randomized priorities? Can these comparisons establish that the gains depend on the proposed uncertainty measure rather than nonuniform sampling more generally?
  4. After including the cost of recomputing uncertainty over all candidate states at each iteration, does U-GROW still require less total training time or fewer GPU-hours than uniform sampling to reach the same success rate?
Reviewer_fU1wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes U-GROW, a lightweight sampling layer for world-model-based reinforcement learning of vision-language-action (VLA) policies. The starting observation is that a self-consistency uncertainty score — the disagreement between flow-matching action chunks decoded with 3 versus 5 denoising steps at fixed initial noise, following Ren et al. (2026) — is concentrated at a small subset of visited states, typically grasping, placement, and release, while remaining low during free-space motion. A policy-switching probe on one RoboTwin 2.0 task supports the interpretation that high-uncertainty states offer more room for improvement: substituting a stronger policy π* at high-U states yields roughly 80% success versus 50% at low-U states at comparable replacement rates.

The method has three components, all operating only on the rollout-start distribution: (i) truncating each failed trajectory at its first high-uncertainty state (dataset quantile threshold with ρ_fail = 0.1) to remove "post-deviation failure tails"; (ii) prioritizing the top ρ_pri = 20% of the retained states by uncertainty; and (iii) mixing this with uniform sampling over all offline states at λ = 0.5. The PPO objective, world model (a fine-tuned Wan2.2-TI2V-5B video diffusion model), reward model (a fine-tuned Qwen3.5-0.8B frame judge), and total model-rollout budget are held fixed against VLA-MBPO, so the only difference between the proposed method and its main baseline is the start-state distribution.

Validation spans four RoboTwin 2.0 tasks, all four LIBERO suites, and four real-world bimanual tasks on an Arx-X5 platform. The reported gains are +4.5 and +1.4 average percentage points over VLA-MBPO on RoboTwin and LIBERO respectively, and 24.0% → 26.0% average success under clean-to-randomized transfer.

Critical strengths

The paper correctly identifies that high uncertainty is not uniformly valuable: in unrecoverable post-deviation segments, high uncertainty is a symptom of being lost rather than an opportunity for improvement, and any uncertainty-driven prior that ignores this is actively misled. This insight generalizes beyond the proposed method to uncertainty-filtered model-based RL and uncertainty-based exploration more broadly. It is also demonstrated in a demanding setting — a 5B video world model and real hardware — spanning rigid-object manipulation, deformable-object folding, and sub-centimeter insertion, which few groups can reproduce.

Critical weaknesses

The policy-switching probe measures the marginal value of substituting a better action at a state; it does not measure the value of starting a short-horizon rollout branch there, which is what the method actually exploits. The two quantities can dissociate, and the paper's own failure-tail analysis shows they do. The probe is also confounded with manipulation phase — high-U states cluster at grasp/place/release, low-U states in free-space transit — with no random-switching arm and no within-phase control, so the 30-point gap in Figure 2a is consistent with "switch during contact phases" having nothing to do with uncertainty. More seriously, failure-tail truncation cuts each failed trajectory at its first high-U state, so every retained failure state has U below the threshold by construction; the top-20% uncertainty pool is therefore likely dominated by states from successful trajectories. The ablation is consistent with this reading: removing the uniform-coverage term produces the largest gap, and "w/o prioritization" is close to the full method. The decisive baseline — prioritized sampling uniformly over successful-trajectory states, with no uncertainty at all — is absent. If that baseline matches U-GROW, uncertainty is a proxy for the outcome label and the paper's title, abstract, and framing are unsupported. This is consequential because it attacks the load-bearing claim rather than a detail.

Questions for authors
  1. What exactly is π* in Section 4, and what is the switching protocol? Does π* control a contiguous segment after switching, or a single action chunk before handing control back, and how is "replacement rate" defined across chunk boundaries? The answer determines how much of Figure 2a is attributable to uncertainty rather than to dwell time or handoff structure.

  2. What is the success/failure composition of each offline dataset (RoboTwin, LIBERO, and the 50 real-world trajectories per task), and what fraction of failed-trajectory states survives truncation? Since ρ_fail is a quantile over all states and truncation is the mechanism that concentrates the pool, these numbers determine how much room the method has and how comparable the threshold is across tasks.

Reviewer_VraEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes U-GROW, an uncertainty-guided rollout-start sampling method for world-model-based reinforcement learning of vision-language-action policies. It estimates policy uncertainty through disagreement between action predictions obtained with different denoising step counts.

Critical strengths
  • Relevant problem and simple intervention. Allocating expensive world-model rollouts more effectively is an important problem for VLA post-training. Modifying the rollout-start distribution makes the method straightforward to integrate into the evaluated pipeline.
  • Empirical motivation and component analysis. The policy-switching experiment provides evidence that high-uncertainty states can be particularly sensitive to action quality. The paper also includes component ablations and hyperparameter sensitivity analysis, acknowledging that uncertainty prioritization needs to be balanced with state coverage.
Critical weaknesses
  • The central computational-efficiency claim is not fully supported by the evaluation. Appendix C.4 states that U-GROW recomputes uncertainty for all candidate offline states at every training iteration using two denoising schedules, introducing additional policy-inference overhead. However, the experiments compare performance against the number of generated transitions rather than elapsed time, and Appendix A does not separate the runtimes of U-GROW and the baseline. Consequently, the reported sample-efficiency gains do not establish whether U-GROW reduces the computational cost of reaching a given performance level.

  • The uncertainty estimator is inherited from prior work, and rollout prioritization relies solely on actor uncertainty. However, the suitability of this signal for identifying useful world-model rollout starts is insufficiently examined. High actor uncertainty may indicate room for policy improvement, but does not establish that the world model can generate reliable experience in those regions. Although the overall results are encouraging, the relationship between actor uncertainty, world-model error, and the learning value of the generated rollouts remains unclear.

Questions for authors
  • How much runtime overhead does uncertainty estimation introduce, and does U-GROW retain its advantage over uniform sampling under the same wall-clock budget?

  • How does U-GROW compare with MBRL-LDN (Ma et al., 2022), which uses the dynamics model’s learning degree to control rollout continuation? Please clarify the benefits of policy-uncertainty-based start selection relative to model-reliability-based rollout control.

2 comments

I thank the authors for their detailed response and additional experiments, which address my primary concerns regarding computational overhead, signal suitability, and baseline comparisons. Given that, I am raising my rating.

Thank you for taking the time to consider our response and additional experiments. We’re glad they addressed your concerns, and we appreciate your decision to raise your rating.

Comments on the paper

1 comment

Dear reviewers,

As the reminded earlier, please check the author's response and provide your feedback before the end of the discuss phase.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.