- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a framework that improves policy learning within latent world models that support scene editing. This framework unifies a latent-edit expert and a generation expert through joint attention, which is trained with a four-stage training recipe. To support scene editing training, the authors curated a large-scale dataset of trajectories that rehearse the same action under environmental variations via a video editing pipeline. In order to improve policy learning with the world model, the framework estimates progress rewards in latent space and uses them to improve policy learning through group-based policy optimization. Both simulated and real-robot settings demonstrate the effectiveness of the proposed method.
Critical strengths
- Augmenting world models with scene edits allows the trained policy to react more robustly to environmental/task perturbations.
- Estimating progress rewards in latent space facilitates computational efficiency and long-term credit assignment.
- The proposed method showcases effective improvement with both \pi-series VLAs on position and task perturbations, which are quite challenging for the baselines.
Critical weaknesses
- The paper proposes a 4-stage training recipe for the world model, which looks quite resource-intensive and time-consuming. Some ablations over the recipe (e.g., removing the last or second-to-last stage) would further justify the need for four-stage training and clarify whether all four stages are equally important for producing a reliable latent world model.
- See the questions below.
Questions for authors
- If “one round of post-training” means that the policy only gets a single update using a group of rollouts, how would more optimization steps affect the performance?
- Despite the results on final task performance, it remains unclear how faithful the latent progress reward is in challenging scenarios, such as whether it can capture the "progress" in long-horizon tasks. A qualitative analysis showing how the rewards correspond to task (partial) success or failure would significantly strengthen these findings.
3 comments
Response to reviewer w97G(2/2)
Q2. Faithfulness of Latent Progress Rewards in Long-Horizon Manipulation
Q2. Despite the results on final task performance, it remains unclear how faithful the latent progress reward is in challenging scenarios, such as whether it can capture the "progress" in long-horizon tasks. A qualitative analysis showing how the rewards correspond to task (partial) success or failure would significantly strengthen these findings.
We thank you for this particularly insightful suggestion regarding the faithfulness and interpretability of latent progress rewards. We agree that a reliable reward model should distinguish meaningful intermediate task progress rather than relying solely on final success or failure. This is particularly important for long-horizon robotic manipulation, where multiple intermediate steps contribute to task completion.
First, our LPRO formulation explicitly models intermediate task progress. As described in Section 4.3, our latent progress model estimates instruction-conditioned progress directly from the policy's visual latent representations. LPRO then converts temporal progress increments into time-aligned, group-relative advantages, providing dense learning signals throughout task execution.
Second, following your suggestion, we examine latent progress predictions across successful, partially successful, and failed executions of the real-world "Sort Objects by Color into Bowls" task. We evaluate the frozen latent progress model at five normalized trajectory timestamps (0%, 25%, 50%, 75%, and 100%) to examine how predicted progress corresponds to actual task execution.
Table R2. Long-Horizon Latent Progress Reward Faithfulness
| Trajectory | 0% | 25% | 50% | 75% | 100% | Actual Outcome |
|---|---|---|---|---|---|---|
| Success | 0.05 | 0.31 | 0.57 | 0.77 | 0.99 | All objects correctly sorted by color |
| Partial Success | 0.05 | 0.33 | 0.50 | 0.56 | 0.58 | Some objects sorted; remaining objects unsorted |
| Failure | 0.06 | 0.08 | 0.04 | 0.04 | 0.12 | No objects correctly sorted |
Progress scores represent the latent progress model's outputs at normalized timestamps along each execution trajectory.
These results demonstrate three important properties:
-
Successful execution: Predicted progress increases from 0.05 to 0.99 as the robot completes successive object-sorting steps, reflecting meaningful task advancement.
-
Partial success: Predicted progress increases from 0.05 to 0.58 as the robot correctly sorts some objects, then stabilizes when the remaining objectives are not completed. This addresses your concern about recognizing partial task completion.
-
Failed execution: Predicted progress remains low throughout the trajectory (0.04–0.12), consistent with the absence of successfully completed sorting steps.
Importantly, the latent progress model distinguishes partial completion from complete failure, even though both trajectories are ultimately unsuccessful. This demonstrates that our reward representation captures meaningful intermediate progress rather than merely predicting binary task success.
Together, these results provide complementary evidence: the trajectory-level analysis illustrates how latent rewards reflect intermediate task progress, while the on-policy evaluation confirms that these rewards remain informative for downstream policy optimization.
We particularly appreciate your suggestion because interpretable intermediate progress is essential for reliable long-horizon robot learning. We will incorporate this trajectory-level analysis into the revised manuscript.
Dear Reviewer w97G,
We sincerely thank you again for your exceptionally interesting and constructive suggestions. We strongly agree with your observations and believe that addressing the important challenges you have identified will contribute significantly to advancing future world model research for robotic learning. Your suggestions are both interesting and insightful, helping us strengthen DreamEvolve while highlighting valuable research directions that can drive meaningful progress in the broader field.
We are truly grateful for your thoughtful review and encouragement. We hope our responses and additional evidence address your questions and further strengthen your positive evaluation of our work. Please feel free to let us know if you have any further questions or suggestions.
Best regards,
Authors
Response to reviewer w97G(1/2)
We sincerely thank you for your insightful and encouraging review! We particularly appreciate your recognition of DreamEvolve's contributions to controllable scene editing, efficient latent progress reward optimization, and robust generalization across VLAs.
Your two questions address important challenges in world-model-based robot learning: how policy performance scales with additional optimization and whether latent rewards faithfully capture intermediate progress in long-horizon manipulation. We particularly appreciate your suggestion to examine partial task completion, as it directly addresses the interpretability of latent reward signals. We strongly agree with your insights and believe that addressing these two fundamental challenges will lead to significant advances in world-model-based robotic learning. Your suggestions are not only valuable for strengthening DreamEvolve but also provide important directions for advancing the future research for robotics.
In response, we clarify our existing optimization-budget results and provide an additional analysis of latent progress rewards across successful, partially successful, and failed task executions.
Our detailed responses follow.
Q1. Effect of Additional Policy Optimization
We thank you for this insightful question. Your distinction between a single post-training round and the number of optimization updates is important for understanding the scalability of world-model-based policy learning.
First, one round of post-training does not mean a single gradient update. In DreamEvolve, one round refers to a complete imagination-based post-training phase, during which LPRO optimizes the policy using groups of imagined trajectories. Our reported zero-shot generalization is achieved after this single post-training phase, without additional policy adaptation during deployment.
Second, our experiments in table R1 demonstrate that increasing the imagined rollout budget consistently improves downstream policy performance. We compare LPRO with 100 and 1,000 imagined rollouts against GRPO and DAPO using 1,000 rollouts.
Table R1. Rollout Efficiency on LIBERO-Pro: Average Success Rate (%)
Increasing the LPRO rollout budget from 100 to 1,000 improves LIBERO-Pro success from 58% to 63% on π0 and from 73% to 78% on π0.5, demonstrating consistent improvements with additional imagined experience.
Importantly, LPRO achieves 73% success using only 100 imagined rollouts, outperforming GRPO (58%) and DAPO (60%) using 1,000 rollouts. These results demonstrate that LPRO combines strong rollout efficiency with continued performance improvements under larger imagination budgets.
We will clarify the distinction between post-training rounds, gradient updates, and imagined rollout budgets in the revised manuscript.