- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PROSE derives intermediate supervision for sparse-reward embodied tasks from completed trajectories. It segments observations into periods, groups recurring situations using visual and semantic descriptors, and scores transitions by their association with terminal success. A degradation penalty and learned visual/action residuals refine these scores, which are transferred to a reward predictor for world-model rollouts. A spatial-memory controller redirects exploration when progress stalls. Across five MineDojo tasks and two reproduced world-model baselines, the combined method improves all ten success-rate point estimates, with a reported mean gain of 9.1 percentage points. Incremental ablations on Cut a tree and Obtain water support contributions from several components.
Critical strengths
-
The method connects sparse terminal outcomes to reusable transition-level supervision without manually specifying intermediate goals. The separation between a shared transition prior and execution-specific residuals is technically meaningful: visually similar situations can require different credit depending on how the agent acts (Section 2.2; Appendix A.2).
-
The controlled comparisons show consistent improvements across two different world-model agents and five tasks, rather than relying on comparisons with published numbers alone. The additional Obtain water ablation in Appendix B.3 supports the Cut a tree trend, and the failure analysis explicitly recognises limitations in perception and fine-grained control. These results make a credible case for the system's practical usefulness.
Critical weaknesses
-
The training-experience accounting is insufficiently explicit for the central efficiency claim. Appendix C reports sizeable archived graph-fitting pools, while Appendix B describes online graph and residual updates. The provenance and timing of those pools, their inclusion in the interaction budget, and the corresponding baseline budgets are not clear. Similarly, Tables 1–2 report three evaluation seeds, but do not establish the number of independent training runs; the treatment of failed episodes in average steps to first success is unspecified. These details determine whether the gains reflect comparable learning resources and what the efficiency numbers mean.
-
The evidence for the progress signal itself is limited relative to its role in optimisation. The residual is trained from trajectory outcomes and the deployed reward mix eventually gives the progress predictor weight 0.95 (Appendices A.2–A.3, B.1). The incremental ablations show aggregate benefit, but do not establish whether success-associated transitions remain useful under policy shift or whether repeated visits can receive misleading credit. This matters because the actor is trained against this learned proxy after warm-up, although the paper appropriately describes the edge statistics as associations rather than causal effects.
Questions for authors
-
Clarification: were all graph-fitting trajectories collected within each reported training run and included in the same environment-interaction budget as its baseline? Please specify independent training versus evaluation seeds, evaluation episode counts, and how unsuccessful episodes enter Table 2.
-
Targeted validation of the existing mechanism: do archived trajectories show repeated rewarded transitions without eventual completion, or degradation in held-out reward prediction as the policy changes? A small diagnostic using existing traces, alongside the rule for unsupported/unmapped transitions, would help assess the proxy concern; a new benchmark suite is not requested.
3 comments
Thank you for the additional analyses. The equal-replay comparisons across all five tasks, independent training runs on two tasks, and clarification of unsuccessful episodes substantially address my evaluation concerns. The cycle and policy-shift diagnostics also strengthen the evidence for the progress signal, although I don't feel the trajectory-level discrimination establishes accurate local credit. I remain at weak acceptance.
Thank you and we are glad that the additional analyses substantially address your evaluation concerns and strengthen the evidence for the progress signal. We agree with your suggestions and comments. The evidence for local credit is indirect, coming from its effect on policy learning. We will state this explicitly in the revised paper.
[Weakness 1 / Question 1] Training-experience accounting and evaluation details
PROSE reuses baseline training trajectories to initialize its graph and replay buffer. These trajectories are collected by the unmodified baseline before PROSE online training. Online graph and residual updates subsequently incorporate trajectories collected during the PROSE online training run.
PROSE still improves over the baseline when both use the same archived replay data. The baselines reported in Table 1–2 (main paper) follow their original training setup and start with an empty replay buffer. To address the concern that PROSE may benefit from additional offline training experience unavailable to the baseline, we additionally evaluate LS-Imagine initialized with the same archived offline replay trajectories used by PROSE. PROSE achieves a much higher success rate of 92.33%, compared with 83.67% for the baseline with offline replay on "Cut a tree" task (Table A). Even with only 700 trajectories, including 100 successful ones, PROSE achieves 86.67%, exceeding the baseline (LS-Imagine) initialized with all 2975 archived trajectories.
The reported mean and standard deviation reflect variation across evaluation seeds. For each method–task setting trained in our original study, we used one independent training run.
For unsuccessful episodes, we assign the maximum episode length (1,000 steps for Cut a tree, Obtain water, Obtain sand, and Obtain wool, and 2,000 steps for Mine iron) when computing the average steps to first success, following CPP and LS-Imagine. For example, an unsuccessful episode on "Cut a tree" contributes 1,000 steps to this metric.
Table A: Effects of offline replay and initial data availability on Cut a tree. Successful and Total denote the numbers of successful and total trajectories in the initial offline pool.
[Weakness 2 / Question 2] Progress rewards under repeated visits and policy shift
Archived trajectories do contain repeated rewarded transitions without eventual task completion but the execution-specific refinement design helps to mitigate the misleading credit. 11 of 160 failed trajectories receive positive net progress rewards from graph cycles, and this number drops to 1 after incorporating the execution-specific refinement.
Progress rewards continue to distinguish successful trajectories from failed ones under policy shift. We apply the same graph to 400 baseline trajectories and newly collected 400 trajectories from PROSE training runs (excluded in graph construction). Each trajectory is scored by summing its progress rewards. AUROC is used to measure how often successful trajectories receive higher scores than failed trajectories, with 0.5 indicating random ranking. With 250 successful and 150 failed trajectories in each group, AUROC is 0.954 for the baseline and 0.910 for PROSE.
Beyond evaluating progress-reward discrimination under policy change, we further examine cross-backbone transfer. Specifically, we construct the offline graph using trajectories collected by LS-Imagine and use it to provide progress supervision during DreamerV3+PROSE training. Unlike the previous experiment, which evaluates reward discrimination under a policy change within the same backbone, this experiment transfers the graph across different world-model agents. On Cut a tree, the resulting policy achieves 89.33 ± 1.53% success and 370.45 ± 6.97 steps to first success. For reference, the original DreamerV3+PROSE results in the main paper are 83.00 ± 4.69% and 405.01 ± 39.69 steps. This result provides evidence that progress supervision learned from LS-Imagine trajectories can transfer to a different world-model policy.
For unsupported/unmapped transitions in collected trajectories, we set the progress reward to zero when node mapping fails. For edges supported by fewer than five distinct trajectories, we set the frequency reward to zero while retaining the execution-specific visual-action residual.