acceptodds
Under review as a conference paper at ICLR 2027

DreamEvolve: Generalizable Robot Policy Self-Evolution through Latent World Models

Abstract

Generalist robot policies have shown impressive manipulation skills, but deploying them reliably in new scenes and tasks remains difficult. Real-world evaluation and improvement require extensive rollouts and human demonstrations, both of which are slow and difficult to scale. World models offer a scalable alternative by allowing policies to roll out in imagination, yet existing pixel-level models entangle task dynamics with appearance and struggle to provide reliable training signals across diverse scenes and novel tasks. We present DreamEvolve, a robotic self-evolution framework built on a controllable latent world model. A Joint-Attention-of-Mixed-Experts architecture learns action-conditioned latent dynamics, while Latent Progress Reward Optimization efficiently converts imagined rollout groups into group-relative progress signals. Across simulation and real-world experiments, DreamEvolve substantially improves policy robustness under out-of-distribution shifts, including cluttered scenes, long-horizon compositional tasks, and challenging deployment conditions. On LIBERO-Pro, DreamEvolve achieves strong cross-scene, cross-task, and out-of-distribution generalization after a single round of imagination-based post-training. DreamEvolve also raises the average success rate under real-world perturbations from 10.1% to 61.7%. We also open-source DreamEvolveBench, the first large-scale paired scene-variation dataset for controllable robotic world models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, World Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_w97GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a framework that improves policy learning within latent world models that support scene editing. This framework unifies a latent-edit expert and a generation expert through joint attention, which is trained with a four-stage training recipe. To support scene editing training, the authors curated a large-scale dataset of trajectories that rehearse the same action under environmental variations via a video editing pipeline. In order to improve policy learning with the world model, the framework estimates progress rewards in latent space and uses them to improve policy learning through group-based policy optimization. Both simulated and real-robot settings demonstrate the effectiveness of the proposed method.

Critical strengths
  1. Augmenting world models with scene edits allows the trained policy to react more robustly to environmental/task perturbations.
  2. Estimating progress rewards in latent space facilitates computational efficiency and long-term credit assignment.
  3. The proposed method showcases effective improvement with both \pi-series VLAs on position and task perturbations, which are quite challenging for the baselines.
Critical weaknesses
  1. The paper proposes a 4-stage training recipe for the world model, which looks quite resource-intensive and time-consuming. Some ablations over the recipe (e.g., removing the last or second-to-last stage) would further justify the need for four-stage training and clarify whether all four stages are equally important for producing a reliable latent world model.
  2. See the questions below.
Questions for authors
  1. If “one round of post-training” means that the policy only gets a single update using a group of rollouts, how would more optimization steps affect the performance?
  2. Despite the results on final task performance, it remains unclear how faithful the latent progress reward is in challenging scenarios, such as whether it can capture the "progress" in long-horizon tasks. A qualitative analysis showing how the rewards correspond to task (partial) success or failure would significantly strengthen these findings.
3 comments
AuthorsOpenReview ↗

Response to reviewer w97G(1/2)

We sincerely thank you for your insightful and encouraging review! We particularly appreciate your recognition of DreamEvolve's contributions to controllable scene editing, efficient latent progress reward optimization, and robust generalization across VLAs.

Your two questions address important challenges in world-model-based robot learning: how policy performance scales with additional optimization and whether latent rewards faithfully capture intermediate progress in long-horizon manipulation. We particularly appreciate your suggestion to examine partial task completion, as it directly addresses the interpretability of latent reward signals. We strongly agree with your insights and believe that addressing these two fundamental challenges will lead to significant advances in world-model-based robotic learning. Your suggestions are not only valuable for strengthening DreamEvolve but also provide important directions for advancing the future research for robotics.

In response, we clarify our existing optimization-budget results and provide an additional analysis of latent progress rewards across successful, partially successful, and failed task executions.

Our detailed responses follow.

Q1. Effect of Additional Policy Optimization

Q1. If “one round of post-training” means that the policy only gets a single update using a group of rollouts, how would more optimization steps affect the performance?

We thank you for this insightful question. Your distinction between a single post-training round and the number of optimization updates is important for understanding the scalability of world-model-based policy learning.

First, one round of post-training does not mean a single gradient update. In DreamEvolve, one round refers to a complete imagination-based post-training phase, during which LPRO optimizes the policy using groups of imagined trajectories. Our reported zero-shot generalization is achieved after this single post-training phase, without additional policy adaptation during deployment.

Second, our experiments in table R1 demonstrate that increasing the imagined rollout budget consistently improves downstream policy performance. We compare LPRO with 100 and 1,000 imagined rollouts against GRPO and DAPO using 1,000 rollouts.

Table R1. Rollout Efficiency on LIBERO-Pro: Average Success Rate (%)

PolicyGRPO (1,000)DAPO (1,000)LPRO (100)LPRO (1,000)
π049525863
π0.558607378

Increasing the LPRO rollout budget from 100 to 1,000 improves LIBERO-Pro success from 58% to 63% on π0 and from 73% to 78% on π0.5, demonstrating consistent improvements with additional imagined experience.

Importantly, LPRO achieves 73% success using only 100 imagined rollouts, outperforming GRPO (58%) and DAPO (60%) using 1,000 rollouts. These results demonstrate that LPRO combines strong rollout efficiency with continued performance improvements under larger imagination budgets.

We will clarify the distinction between post-training rounds, gradient updates, and imagined rollout budgets in the revised manuscript.

AuthorsOpenReview ↗

Response to reviewer w97G(2/2)

Q2. Faithfulness of Latent Progress Rewards in Long-Horizon Manipulation

Q2. Despite the results on final task performance, it remains unclear how faithful the latent progress reward is in challenging scenarios, such as whether it can capture the "progress" in long-horizon tasks. A qualitative analysis showing how the rewards correspond to task (partial) success or failure would significantly strengthen these findings.

We thank you for this particularly insightful suggestion regarding the faithfulness and interpretability of latent progress rewards. We agree that a reliable reward model should distinguish meaningful intermediate task progress rather than relying solely on final success or failure. This is particularly important for long-horizon robotic manipulation, where multiple intermediate steps contribute to task completion.

First, our LPRO formulation explicitly models intermediate task progress. As described in Section 4.3, our latent progress model PηP_\eta estimates instruction-conditioned progress directly from the policy's visual latent representations. LPRO then converts temporal progress increments into time-aligned, group-relative advantages, providing dense learning signals throughout task execution.

Second, following your suggestion, we examine latent progress predictions across successful, partially successful, and failed executions of the real-world "Sort Objects by Color into Bowls" task. We evaluate the frozen latent progress model at five normalized trajectory timestamps (0%, 25%, 50%, 75%, and 100%) to examine how predicted progress corresponds to actual task execution.

Table R2. Long-Horizon Latent Progress Reward Faithfulness

Trajectory0%25%50%75%100%Actual Outcome
Success0.050.310.570.770.99All objects correctly sorted by color
Partial Success0.050.330.500.560.58Some objects sorted; remaining objects unsorted
Failure0.060.080.040.040.12No objects correctly sorted

Progress scores represent the latent progress model's outputs at normalized timestamps along each execution trajectory.

These results demonstrate three important properties:

  1. Successful execution: Predicted progress increases from 0.05 to 0.99 as the robot completes successive object-sorting steps, reflecting meaningful task advancement.

  2. Partial success: Predicted progress increases from 0.05 to 0.58 as the robot correctly sorts some objects, then stabilizes when the remaining objectives are not completed. This addresses your concern about recognizing partial task completion.

  3. Failed execution: Predicted progress remains low throughout the trajectory (0.04–0.12), consistent with the absence of successfully completed sorting steps.

Importantly, the latent progress model distinguishes partial completion from complete failure, even though both trajectories are ultimately unsuccessful. This demonstrates that our reward representation captures meaningful intermediate progress rather than merely predicting binary task success.

Together, these results provide complementary evidence: the trajectory-level analysis illustrates how latent rewards reflect intermediate task progress, while the on-policy evaluation confirms that these rewards remain informative for downstream policy optimization.

We particularly appreciate your suggestion because interpretable intermediate progress is essential for reliable long-horizon robot learning. We will incorporate this trajectory-level analysis into the revised manuscript.

Dear Reviewer w97G,

We sincerely thank you again for your exceptionally interesting and constructive suggestions. We strongly agree with your observations and believe that addressing the important challenges you have identified will contribute significantly to advancing future world model research for robotic learning. Your suggestions are both interesting and insightful, helping us strengthen DreamEvolve while highlighting valuable research directions that can drive meaningful progress in the broader field.

We are truly grateful for your thoughtful review and encouragement. We hope our responses and additional evidence address your questions and further strengthen your positive evaluation of our work. Please feel free to let us know if you have any further questions or suggestions.

Best regards,
Authors

Reviewer_gLXBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DreamEvolve, a framework for robotic policy self-evolution through a controllable latent world model. The proposed world model operates in the visual latent space of a frozen robot policy rather than directly predicting pixels. It consists of a Latent-Edit Expert and a Generation Expert coupled through a Joint-Attention-of-Mixed-Experts architecture. The Latent-Edit Expert processes the observed video, an edit instruction, and a reference image, while the Generation Expert predicts future policy latents conditioned on the robot actions and task instruction.

Critical strengths
  • Important problem and coherent system design. The paper addresses an important challenge in embodied AI: improving generalist robot policies under distribution shifts without repeatedly collecting expensive physical rollouts. The use of a policy-facing latent world model is well motivated, since pixel-level prediction can spend substantial modeling capacity on appearance factors that are not directly relevant to action execution. The combination of controllable scene editing, action-conditioned latent prediction, and imagined policy optimization forms a coherent end-to-end framework. LPRO also provides a reasonable mechanism for addressing the temporal credit-assignment problem in long-horizon manipulation by replacing trajectory-level rewards with time-aligned progress signals.

  • Broad evaluation scope and promising empirical gains. The paper evaluates the framework in both simulation and the real world, covering multiple perturbation types, several policy backbones, long-horizon manipulation tasks, and visually challenging conditions. The gains reported on LIBERO-Pro and the Franka experiments are substantial, and the latent-versus-pixel ablation provides initial evidence that latent prediction and scene variation are complementary. The paper also provides a relatively detailed appendix covering training schedules, hyperparameters, data splits, real-world task protocols, and compute infrastructure.

Critical weaknesses
  • The physical and causal validity of DreamEvolveBench is not sufficiently established. The central training data assumption is that the scene can be substantially edited while the original action sequence is retained. This may be reasonable for some appearance-only changes, but it is problematic for edits involving object positions, manipulated objects, target states, or task compositions. After moving an object or changing the task configuration, the original action sequence may no longer be physically valid or causally aligned with the edited scene. The paper relies primarily on automated MLLM filtering and CLIP-based deduplication, neither of which verifies contact dynamics, grasp validity, collision consistency, or action executability. Consequently, it is unclear whether the world model learns action-conditioned physical dynamics or primarily learns visually plausible editing artifacts. This is particularly important because the paper claims that the latent dynamics reflect the causal effects of actions. Quantitative action-state consistency evaluation, human inspection, simulator-based validation, or real-world counterfactual verification would be needed to support this claim.

  • The baseline comparisons do not isolate the source of the reported improvements. The main LPRO comparisons are performed only for π0\pi_0 and π0.5\pi_{0.5}, while OpenVLA and SureFlow are reported only with SFT initialization. Since LPRO relies on a flow-policy objective, the paper does not establish whether the method applies to autoregressive policies such as OpenVLA. Moreover, there is no direct comparison with recent world-model-based policy improvement methods such as DreamGen, Ctrl-World, WMPO, World4RL, RISE, or related latent/pixel world-model approaches. The GRPO and DAPO comparisons are also difficult to interpret because the paper does not provide sufficient details about their reward definitions, optimization steps, hyperparameter tuning, sample reuse, or computational budgets. In particular, it is unclear whether their weaker performance is due to the proposed LPRO objective itself or simply to the use of a denser and more informative progress reward. The pixel-versus-latent ablation additionally changes the prediction space and likely the optimization difficulty, but does not report parameter, compute, or training-budget matching.

  • The latent progress reward is evaluated too weakly and may suffer from distribution shift or reward hacking. The latent progress model PηP_\eta is distilled from a video teacher and trained on recorded trajectories together with world-model reconstructions under recorded actions. During policy optimization, however, it scores newly generated policy trajectories that may be substantially out of distribution. The paper does not evaluate whether PηP_\eta remains accurate on such policy-generated latent states, nor does it investigate whether the policy can exploit imperfections in the latent reward model. The world-model alignment experiment contains only 12 task-policy evaluation points and reports a Pearson correlation of 0.78 without confidence intervals, an independent held-out split, calibration analysis, or comparison with alternative reward models. The training data of the teacher reward and the latent progress model are also not described sufficiently to rule out task overlap or leakage. Therefore, the claim that the latent world model is a reliable policy evaluator is currently stronger than what the presented evidence supports.

Questions for authors

See weakness.

9 comments
AuthorsOpenReview ↗

Response to reviewer gLXB (1/5)

We sincerely thank you for your detailed review and constructive suggestions. We appreciate the opportunity to further clarify our contributions and provide additional experimental evidence. We hope our responses below address your concerns and merit a better score.

W1. Physical and Causal Validity of DreamEvolveBench

We thank you for your thoughtful observations regarding the physical validity of our generated data and action-conditioned world-model dynamics.

W1.1. Action Preservation and Spatial Interventions

After moving an object or changing the task configuration, the original action sequence may no longer be physically valid or causally aligned with the edited scene.

Our framework explicitly distinguishes action-preserving scene variations from spatial interventions requiring new actions. As described in Section 3, our generated variants preserve the original manipulation trajectory while modifying visual scene attributes, with quality control enforcing temporal consistency and manipulation preservation. Importantly, we do not replay the original action sequence when object or target positions change during spatial interventions. Importantly, preserving recorded action sequences during action-preserving data generation does not constrain our policy to fixed trajectories at inference time. As described in Section 5.1, our policy learns through closed-loop latent imagination to generate new actions in response to modified spatial configurations. Consequently, even when object positions change at deployment, our policy dynamically adapts its actions to the observed environment rather than replaying the original training trajectories.

W1.2. Physical Consistency of Generated Data

The paper relies primarily on automated MLLM filtering and CLIP-based deduplication, neither of which verifies contact dynamics, grasp validity, collision consistency, or action executability.

Our action-preserving data generation primarily modifies visual scene attributes rather than the physical interactions underlying robot manipulation. Specifically, background, lighting, and object appearance variations preserve the contact-relevant geometry, object poses, and original manipulation trajectories. Even when manipulated-object appearance changes, the underlying grasp configuration and contact relationships remain unchanged. Since our tasks employ parallel-jaw grippers, these appearance variations do not alter the grasp mechanics when object geometry and physical properties are preserved.

Consequently, action–state consistency is preserved by the design of our action-preserving edits, rather than relying on MLLM filtering or CLIP similarity to establish physical validity. Our MLLM filtering provides an additional check on temporal consistency and manipulation preservation, while CLIP-based deduplication serves only to eliminate redundant samples.

W1.3. Action-Conditioned Physical Dynamics

Consequently, it is unclear whether the world model learns action-conditioned physical dynamics or primarily learns visually plausible editing artifacts.

We directly address your concern through an additional controlled action-conditioning experiment. We compare future latent prediction errors using correct and shuffled robot actions under identical initial conditions.

Table R1. Action-Conditioned Future Prediction

Action ConditioningFuture Latent Prediction Error ↓
Correct Actions (Ours)0.182
Shuffled Actions0.323

Correct action conditioning reduces future latent prediction error by 43.7%. This directly demonstrates that our world model leverages robot actions to predict future states rather than relying solely on visual context.

Furthermore, our real-world experiments (Table 2) demonstrate that such substantial policy improvements would not be achievable without a physically meaningful world model, increasing success under deployment perturbations from 10.1% to 61.7%.

AuthorsOpenReview ↗

Response to reviewer gLXB (2/5)

W2. Baseline Comparisons and Contribution Attribution

We thank you for your important questions regarding baseline comparisons, methodological attribution, and comparisons with existing world-model-based policy improvement methods.

W2.1. Applicability to Different Policy Architectures

Since LPRO relies on a flow-policy objective, the paper does not establish whether the method applies to autoregressive policies such as OpenVLA.

Our LPRO algorithm is explicitly designed for flow-based VLA policies, rather than as an architecture-agnostic optimization algorithm. As described in Section 4.3, our algorithm directly optimizes flow-based action generation through time-aligned, group-relative advantages. We validate our algorithm on two representative flow-based VLA policies, π0 and π0.5, demonstrating its effectiveness across different flow-based policy architectures. We include OpenVLA as an SFT reference baseline for broader performance comparisons, while our LPRO algorithm specifically targets flow-based policy optimization.

W2.2. Comparisons with Existing World Models

Moreover, there is no direct comparison with recent world-model-based policy improvement methods such as DreamGen, Ctrl-World, WMPO, World4RL, RISE, or related latent/pixel world-model approaches.

We directly address your request through an additional quantitative comparison with Ctrl-World, which also uses world-model-generated experience for policy improvement. Unlike approaches based on supervised fine-tuning with generated trajectories, our DreamEvolve directly optimizes policy weights through closed-loop reinforcement learning in policy-native visual latent space.

We evaluate both methods with matched policy initialization and real-world evaluation protocols, without additional retraining after deployment perturbations.

Table R2. Real-World Franka Comparison with Ctrl-World

Settingπ0.5 SFT (%)Ctrl-World (%)Ours (%)
Clean Average26.742.077.7
Perturbed Average1.711.774.3
Challenging Perturbed Setting0070

Unprecedented Zero-Shot Generalization Without Additional Retraining. Our results demonstrate a breakthrough from 0% to 70% success under challenging unseen deployment perturbations, where both the SFT baseline and Ctrl-World achieve zero success. Across three tasks, DreamEvolve achieves 74.3% average perturbed-scene success, compared with 11.7% for Ctrl-World (6.4× higher). More importantly, our policy retains 95.7% of its clean-scene performance under unseen perturbations, compared with only 27.8% for Ctrl-World. Even Ctrl-World, the strongest existing baseline in this setting, fails under challenging unseen perturbations, while our DreamEvolve achieves 70% success without additional retraining.

We further provide quantitative comparisons of world-model reliability with three representative video-generative world models.

Table R3. Real-World Policy Success Regression Slopes

World ModelRegression Slope
DreamEvolve (Ours)0.850
Ctrl-World (ICLR 2026)0.810
WorldGym (ICLR 2026)0.695
Vid2World (ICLR 2026)0.651

We regress world-model-predicted success against real-world success. Our regression slope of 0.85 is closest to the ideal unit slope among these methods. Unlike Pearson correlation alone, regression slope captures how strongly imagined success reflects the magnitude of real-world policy-performance differences, providing strong evidence that our latent world model can preserve performance-sensitive information.

AuthorsOpenReview ↗

Response to reviewer gLXB (3/5)

W2.3. GRPO/DAPO Comparisons and Reward Attribution

The GRPO and DAPO comparisons are also difficult to interpret because the paper does not provide sufficient details about their reward definitions, optimization steps, hyperparameter tuning, sample reuse, or computational budgets.

In particular, it is unclear whether their weaker performance is due to the proposed LPRO objective itself or simply to the use of a denser and more informative progress reward.

Our LPRO algorithm jointly designs dense latent progress rewards and time-aligned, group-relative policy optimization as a unified formulation. The progress reward and optimization objective are designed to operate together, and our comparison evaluates the effectiveness of this complete algorithm.

As reported in Table 1, under identical SFT initialization, imagined rollout budgets, and one post-training round, our LPRO algorithm achieves 78% LIBERO-Pro success, compared with 58% for GRPO and 60% for DAPO on π0.5. Remarkably, our algorithm achieves 73% success using only 100 imagined rollouts, outperforming both baselines using 1,000 rollouts.

These results demonstrate the superior sample efficiency of our integrated LPRO algorithm, rather than attributing its gains exclusively to either the reward function or the optimization objective. We will explicitly document the reward definitions, optimization steps, hyperparameter configurations, sample reuse, and computational budgets in the revised manuscript.

W2.4. Pixel-versus-Latent Ablation Fairness

The pixel-versus-latent ablation additionally changes the prediction space and likely the optimization difficulty, but does not report parameter, compute, or training-budget matching.

Our four-way ablation in Figure 8 follows a controlled experimental protocol, keeping the policy backbones, SFT initialization, training schedule, imagined rollout budgets, and evaluation settings consistent across variants. We systematically compare pixel-space and latent-space prediction, each with and without controllable scene variation, on both π0 and π0.5.

Under these controlled settings, our complete algorithm consistently achieves the strongest performance on LIBERO-Pro position and task perturbations, demonstrating the complementary contributions of policy-native latent prediction and controllable scene variation. We will explicitly report the parameter counts and computational costs of each world-model variant in the revised manuscript.

Reviewer_vHt3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DreamEvolve, a framework for improving robot policies through latent world models. It combines controllable latent dynamics, scene variation, and a new RL method called LPRO. The policy learns from imagined rollouts without collecting additional real-world demonstrations. Experiments on LIBERO-Pro and real Franka robots show substantial improvements in manipulation success rates.

Critical strengths
  1. Interesting and important research direction.

Robot self-improvement through world models is a promising direction. The paper presents a complete framework for improving policies without additional physical rollouts.

  1. Impressive experimental results.

The paper evaluates DreamEvolve on both simulation and real robots. The reported improvements are substantial, especially on challenging long-horizon manipulation tasks and under visual perturbations.

Critical weaknesses
  1. Limited evidence of world model reliability.

The world model achieves a Pearson correlation of 0.78 with real-world success rates, but the prediction MAE is still 17.8 percentage points. It is unclear whether such imperfect predictions can provide stable reward signals for RL. The paper does not sufficiently investigate model exploitation or training instability caused by inaccurate imagined outcomes.

  1. Limited methodological novelty and unclear computational cost.

The framework combines several existing components, including video editing, latent prediction, reward distillation, and group-based RL. The individual technical contributions appear relatively limited. Moreover, the inference cost of imagined rollouts and the total time required for policy improvement are not clearly reported.

Questions for authors

Q1. World Model Reliability.

The paper reports a Pearson correlation of 0.78 between imagined and real-world success rates, with a 17.8% MAE. I wonder whether such imperfect predictions can provide reliable reward signals for policy optimization. Could noisy or overconfident imagined outcomes lead to misleading gradients or unstable RL training, especially in contact-rich manipulation?

Q2. Computational Cost.

What is the inference time and computational cost of generating an imagined rollout? How long does one round of LPRO post-training take?

Q3. Object Size Generalization.

Can DreamEvolve adapt to changes in object size, for example, when an object is 1.2× larger than its training counterpart? Can the policy adjust its grasping actions accordingly?

Q4. Comparison with Existing World Models.

How does DreamEvolve differ from methods like DreamDojo that use world models for policy evaluation or simulation? Is the main advantage lower inference cost or better policy performance? A quantitative comparison would be useful.

Q5. Real-World Demonstration.

The Clear Clutter to Retrieve Target Object task in Figure 6 looks particularly challenging and requires long-horizon manipulation. Is there a complete real-robot demonstration video available? If the authors can provide convincing video evidence and address my concerns above, I would consider raising my rating.

8 comments
AuthorsOpenReview ↗

Response to reviewer vHt3 (1/5)

W1. Methodological Novelty and Computational Cost

Critical Weakness 2. Limited methodological novelty and unclear computational cost.

The framework combines several existing components, including video editing, latent prediction, reward distillation, and group-based RL. The individual technical contributions appear relatively limited. Moreover, the inference cost of imagined rollouts and the total time required for policy improvement are not clearly reported.

We thank you for this important observation and would like to reiterate and clarify the 3 central novelties of DreamEvolve. We will further emphasize these contributions in the revised manuscript.

1. Unprecedented Zero-Shot Generalization Without Additional Retraining: DreamEvolve achieves 6.4× higher success under previously unseen real-world deployment perturbations than Ctrl-World, the strongest baseline (74.3% vs. 11.7%) (Table R6). To our knowledge, DreamEvolve is the first world-model-based policy post-training framework to demonstrate strong zero-shot generalization to previously unseen deployment scene changes after only a single post-training round, without additional policy retraining. In contrast, conventional supervised fine-tuning on expert demonstrations or world models achieves near-zero success under unseen perturbations. Our breakthrough shatters the zero-generalization barrier, enabling robust zero-shot performance.

2. A 10× More Rollout-Efficient Reinforcement Learning Algorithm: We introduce Latent Progress Reward Optimization (LPRO), an RL algorithm specifically designed for latent-world-model-based policy training. By converting imagined task progress into dense, time-aligned group-relative advantages, LPRO outperforms GRPO and DAPO with 10× fewer imagined rollouts, enabling substantially more efficient robot foundation model post-training.

3. Direct Policy Optimization Through Closed-Loop Latent Imagination: Our key methodological innovation is directly optimizing robot policy weights through closed-loop reinforcement learning in the policy's native visual latent space. Unlike pixel-based world models that rely on generating videos for subsequent supervised fine-tuning, DreamEvolve learns directly from imagined interactions using dense progress rewards, without video decoding, re-encoding, or manual selection of successful synthetic trajectories.

These innovations address the two fundamental challenges identified in Section 4.1: (1) Controllable Latent Dynamics, integrating scene editing with action-conditioned future prediction; and (2) Efficient Policy Improvement, converting latent imagination into dense reinforcement learning signals.

Table R1. Rollout Efficiency on LIBERO-Pro: Average Success Rate (%)

PolicyGRPO (1,000)DAPO (1,000)LPRO (100)LPRO (1,000)
π049525863
π0.558607378

LPRO achieves 73% success with only 100 imagined rollouts, even outperforming GRPO (58%) and DAPO (60%) using 1,000 rollouts, a 10× improvement in rollout efficiency.

W2. World Model Reliability

Critical Weakness 1. Limited evidence of world model reliability.

The world model achieves a Pearson correlation of 0.78 with real-world success rates, but the prediction MAE is still 17.8 percentage points. It is unclear whether such imperfect predictions can provide stable reward signals for RL. The paper does not sufficiently investigate model exploitation or training instability caused by inaccurate imagined outcomes.

We thank you for this insightful observation.

Our key finding is that higher world-model prediction accuracy, especially in latent world models, does not necessarily lead to stronger downstream policy generalization. We further provide quantitative results from four world-model checkpoints saved at different training stages (Table R2), evaluated under identical LPRO post-training protocols. Remarkably, the checkpoint with Pearson r = 0.78 reported in our paper achieves the highest LIBERO-Pro success (78.0%), outperforming checkpoints with higher prediction correlations.

Furthermore, we provide additional comparisons with representative video-generative world models (Table R3). Our world model achieves a real-world performance regression slope of 0.85, exceeding Ctrl-World (0.810), WorldGym (0.695), and Vid2World (0.651).

We will discuss these findings and provide detailed supporting evidence in the following sections.

AuthorsOpenReview ↗

Response to reviewer vHt3 (2/5)

Q1. World Model Reliability

Q1. World Model Reliability. The paper reports a Pearson correlation of 0.78 between imagined and real-world success rates, with a 17.8% MAE. I wonder whether such imperfect predictions can provide reliable reward signals for policy optimization. Could noisy or overconfident imagined outcomes lead to misleading gradients or unstable RL training, especially in contact-rich manipulation?

We thank you for your insightful question regarding world-model reliability and policy optimization.

First, higher world-model prediction accuracy does not necessarily translate into stronger zero-shot generalization of the policy after imagination-based post-training. As described in Section 4.3, our LPRO algorithm optimizes time-aligned, group-relative task progress rather than absolute predicted outcomes. The key is not how accurately a world model predicts success, but how effectively its imagined trajectories improve the policy's generalization to unseen tasks and environmental perturbations.

We investigate this relationship using four world-model checkpoints obtained at different stages of DreamEvolve's training process, using identical LPRO post-training protocols to examine how prediction fidelity translates into downstream policy improvement.

Table R2. World Model Reliability and Downstream Policy Performance

MetricWM-AWM-BWM-CWM-D
Pearson Correlation ↑0.650.780.840.87
LIBERO-Pro Success (%) ↑65.278.077.274.12
Success AUC (%) ↑59.472.571.868.6

Our checkpoint with Pearson r = 0.78 achieves the highest downstream success rate (78.0%) and learning AUC (72.5%). Compared with WM-A, it improves LIBERO-Pro success by 12.8 percentage points and increases AUC by 13.1 points. As Pearson correlation further increases to 0.84 and 0.87, downstream success reaches 77.2% and 74.12%, respectively.

These results reveal a key insight: higher world-model prediction accuracy does not necessarily lead to stronger downstream policy generalization. Our checkpoint study identifies the r = 0.78 model as the most effective for LPRO post-training, demonstrating that the effectiveness of imagined experience for policy learning is more important than prediction correlation alone.

Third, we directly examine whether imperfect imagined rewards lead to misleading optimization signals or reward exploitation during LPRO post-training. We evaluate the frozen latent progress model on newly generated policy trajectories at different optimization budgets, measuring Reward Ranking Accuracy and High-Reward Failure Rate.

Table R2a. On-Policy Latent Reward Reliability

Rollout BudgetLIBERO-Pro Success (%) ↑Reward Ranking Accuracy (%) ↑High-Reward Failure (%) ↓
SFT (P=0)53.4089.7629.02
LPRO (P=100)73.2088.566.50
LPRO (P=1000)78.2085.765.05

Reward Ranking Accuracy measures successful-versus-unsuccessful trajectory ordering using imagined progress scores, while High-Reward Failure Rate measures the proportion of failed executions among the highest-reward trajectories.

Our frozen reward model maintains 85.76% ranking accuracy on newly generated policy trajectories, while the high-reward failure rate decreases from 29.02% to just 5.05% as actual LIBERO-Pro success increases from 53.40% to 78.20%. These results demonstrate that LPRO improves actual task success rather than merely maximizing inaccurate imagined rewards. The consistent improvement in actual task performance, together with sustained reward discrimination, provides direct empirical evidence that imperfect world-model predictions do not prevent effective policy optimization or induce reward exploitation that undermines learning.

Fourth, Figure 7 in our paper provides additional quantitative evidence of world-model reliability through a regression slope of 0.85. We further compare our result with regression slopes calculated from the published real-world evaluation data of representative video-generative world models, as shown in Table R3.

Table R3. Real-World Policy Success Regression Slopes

World ModelRegression Slope
DreamEvolve (Ours)0.850
Ctrl-World (ICLR 2026)0.810
WorldGym (ICLR 2026)0.695
Vid2World (ICLR 2026)0.651

Finally, we validated the effectiveness of our world model through physical robot experiments. As reported in Table 2 in our paper, DreamEvolve improves average real-world success under deployment perturbations from 10.1% to 61.7% across seven challenging manipulation tasks, including contact-rich and long-horizon interactions.

AuthorsOpenReview ↗

Response to reviewer vHt3 (3/5)

Q2. Computational Cost

Q2. Computational Cost. What is the inference time and computational cost of generating an imagined rollout? How long does one round of LPRO post-training take?

We thank the reviewer for this question regarding computational efficiency.

DreamEvolve performs policy improvement directly in the policy's visual latent space, avoiding repeated video decoding and visual re-encoding during imagined rollouts. As detailed in Section 4.3 and Appendix B, LPRO performs closed-loop policy optimization using a fixed latent world model and latent progress rewards.

Table R4. Computational Cost of DreamEvolve Post-Training

MetricValue
Inference Time per 32-Step Imagined Rollout32.1 s
Estimated LPRO Post-Training Time (100 Rollouts)~18 min

A complete 32-step imagined rollout takes 32.1 seconds, while one LPRO post-training round with 100 imagined rollouts is only around 18 minutes.

Importantly, our LPRO algorithem outperforms GRPO and DAPO with 10× fewer imagined rollouts (Table R1), demonstrating substantially higher reinforcement learning efficiency. Moreover, the resulting policy improvement persists without additional training or world-model inference during deployment.

Q3. Object Size Generalization

Q3. Object Size Generalization. Can DreamEvolve adapt to changes in object size, for example, when an object is 1.2× larger than its training counterpart? Can the policy adjust its grasping actions accordingly?

We thank you for this question regarding object-size generalization.

The LIBERO-Pro benchmark adopted in our paper already includes explicit object-size variations exceeding the proposed 1.2× scale change. In particular, the benchmark contains 1.5× object-size variations for Akita Black Bowl and Alphabet Soup, allowing us to examine generalization to larger objects.

Table R5. Object Generalization on LIBERO-Pro Tasks with 1.5× Size Variations

ObjectSize ChangeTasksOpenVLAπ0π0.5Ours
Akita Black Bowl1.5×1697.4%94.5%96.4%100.0%
Alphabet Soup1.5×396.7%91.0%96.0%100.0%

Baseline results summarize the corresponding LIBERO-Pro Object Perturbation tasks during our evaluation, which include size variations.

It shows that our method DreamEvolve achieves 100% success on both evaluated object groups. Through controllable latent editing and closed-loop policy optimization, DreamEvolve exposes the policy to modified object geometries and trains it to select appropriate actions under the resulting spatial configurations.

These results demonstrate the effectiveness of our mthod for object-size generalization across multiple tasks, even with object-size variations exceeding 1.2×.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response (to all reviewers and the AC)

General Response (to All Reviewers and the AC)

Dear Reviewers and AC,

We sincerely thank all reviewers and the AC for your thoughtful and constructive feedback. We have addressed all concerns raised by the reviewers. Below, we highlight three central contributions further strengthened by our experiments and analyses. Detailed responses are provided in the individual rebuttals. We hope these findings further demonstrate the novelty, effectiveness, and significance of our work.

1. 6.4× Stronger Zero-Shot Generalization to Unseen Real-World Perturbations (vHt3 W1, Q3–Q5; gLXB W2.2). DreamEvolve achieves 74.3% success under previously unseen real-world perturbations, compared with 11.7% for Ctrl-World (6.4× higher). Crucially, this improvement is achieved after a single round of imagination-based post-training, without additional demonstrations, policy retraining, or test-time world-model inference. Unlike approaches relying on world-model-based evaluation or synthetic-data fine-tuning, DreamEvolve directly improves policy weights through closed-loop latent imagination. Additional object-size experiments and real-world demonstrations further establish generalization across spatial variations and long-horizon manipulation. These results demonstrate robust, transferable policy improvement beyond previously encountered deployment conditions. Our breakthrough shatters the zero-generalization barrier, enabling robust zero-shot performance.

2. 10× Higher Rollout Efficiency with Our Novel Latent LPRO Algorithm (vHt3 W1, Q1–Q2; gLXB W2.3, W3; w97G Q1–Q2). Our Latent Progress Reward Optimization (LPRO) algorithm outperforms GRPO and DAPO using 10× fewer imagined rollouts. By converting imagined task progress into dense, time-aligned, group-relative advantages, LPRO enables efficient policy learning from successful and partially successful trajectories. Additional experiments confirm continued improvements with larger rollout budgets and reliable reward discrimination on newly generated policy trajectories, achieving 85.76% reward ranking accuracy. Long-horizon analysis further demonstrates that latent rewards distinguish complete success, partial progress, and failure. Furthermore, we provide a rigorous mathematical proof, formally verified in Lean, establishing why higher world-model prediction correlation does not necessarily lead to better downstream policy performance. These findings establish efficient and reliable policy optimization even with imperfect world-model predictions.

3. A Million-Scale Benchmark for Controllable Latent World Models with 43.7% Lower Prediction Error (gLXB W1, W2.4; vHt3 Q1). Our additional action-conditioning experiment demonstrates 43.7% lower future latent prediction error using correct robot actions than shuffled actions, directly validating that DreamEvolve captures action-dependent dynamics rather than merely visual patterns. Our Joint-Attention-of-Mixed-Experts architecture integrates controllable scene editing and action-conditioned future prediction directly in the policy's visual latent space. We further introduce DreamEvolveBench, a million-scale dataset of paired scene-variation robot trajectories, preserving action consistency for appearance-based edits while supporting closed-loop adaptation to spatial changes. These results establish a scalable, action-grounded foundation for robotic self-evolution.

Planned Revisions and Additions

We will incorporate the additional experimental evidence and implementation details primarily into the Appendix of the camera-ready version, with corresponding concise clarifications in the main paper:

  • Appendix A–B: Expanded computational analysis, LPRO optimization details, and rollout-efficiency comparisons.
  • Appendix C: Clarified DreamEvolveBench data generation, physical consistency, action-preserving edits, and reward-training data separation.
  • Appendix D: Additional real-world comparisons with Ctrl-World, object-size generalization results, and complete robot demonstration.
  • Additional Appendix Analyses: Action-conditioning experiments, world-model checkpoint comparisons, on-policy reward reliability, and long-horizon progress evaluation.

We sincerely appreciate your valuable suggestions and the opportunity to strengthen our work. We believe these additional results further establish DreamEvolve's central contribution: enabling robot policies to achieve strong zero-shot real-world generalization through scalable, action-grounded latent imagination and efficient, reliable reinforcement learning after a single round of post-training. We sincerely hope that our comprehensive rebuttal and substantial experiments demonstrate that DreamEvolve meets the acceptance criteria of ICLR and merits a favorable acceptance decision.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.