- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the failure of latent world model (LeWM/DINO-WM) planning on distant image goals, attributing it to two factors: under a fixed candidate budget, randomly initialized action proposals provide increasingly poor coverage of an exponentially growing action space; and a distant final goal offers only weak guidance for evaluating local action candidates. SAGE freezes the world model and learns two components: (1) a multi-duration subgoal generator that, given the current history, the final goal, the remaining goal horizon , and a requested option duration , predicts the latent subgoal steps ahead; (2) a subgoal-conditioned action prior — a Transformer with a trajectory-level GMM (8 modes) — that proposes -step action candidates conditioned on the predicted subgoal to initialize the online search (CEM refinement on PushT/Cube). Experiments are conducted on PushT, OGBench Cube, and LIBERO.
Critical strengths
- The problem decomposition is accurate and the core design is coherent. The paper separates the long-horizon planning bottleneck into two independent dimensions — proposal quality and evaluation target — and lets the same temporally matched subgoal constrain both.
Critical weaknesses
- Limited novelty, and the most relevant baselines are missing, making it difficult to determine whether the gains come from "world-model planning" or from "hierarchical imitation on expert data".
- A subgoal generator plus a goal-conditioned action prior is essentially hierarchical goal-conditioned imitation (Play-LMP, HIRO-style subgoals, Guider, FF-JEPA — all discussed in the related work). Both components are trained with supervision on expert or oracle trajectories.
- The paper's own data show that world-model search sometimes helps little, and can even hurt: on DINO-WM Cube, Generator+PriorTop reaches 141/150 at , above SAGE's 132/150 (Table 15).
- The paper does not compare against goal-conditioned BC or hierarchical imitation policies trained on the same data (e.g., GCBC, HIQL, Diffusion Policy), nor against FF-JEPA, the closest concurrent work.
- PRISM, the only learned baseline, uses fixed 25-step chunks whose execution cadence differs from SAGE's, so it is not a matched comparison.
- Consequently, the central claim that subgoal-conditioned action generation substantially improves world-model planning lacks sufficient controlled support.
- The evaluation protocol may favor SAGE, and some baselines are weak, which discounts the strength of the conclusions.
- The subgoal generator takes the true remaining goal horizon as input, and goals are future frames of held-out trajectories from the same distribution — in effect, waypoint tracking along demonstration trajectories. Validity on out-of-distribution goals and multimodal paths is untested (the generator is a deterministic regressor, with the attendant risk of predicting an "averaged" latent corresponding to no valid state).
- The LIBERO planning configuration is weak overall: all methods use only 64 candidates with a single ranking pass (vs. 300 candidates × 30 CEM rounds on PushT/Cube), and Base LeWM's proposal is a residual random walk with a fixed scale of 0.05, which limits the strength of the "0% → 48.7%/58.0%" comparison.
Questions for authors
- Can you add the following comparisons, all trained on the same data? (a) Goal-conditioned BC or hierarchical imitation policies (e.g., GCBC + subgoals, HIQL, Diffusion Policy); (b) FF-JEPA (subgoals used only as optimization targets). If such methods — which do not use the world model — achieve success rates close to SAGE, the paper's contribution would need to be repositioned.
- What would Base LeWM achieve on LIBERO with the same 300-candidate × 30-round CEM configuration as on PushT/Cube (or under equal compute)? And what would PRISM achieve under the same execution schedule as SAGE?
- Can you provide a failure attribution: do failures stem mainly from unreachable or inaccurate subgoals, or from world-model prediction/scoring errors?
- Does SAGE remain effective when trained on non-expert or random play data, or when the goal lies outside the training distribution (i.e., not on any recorded trajectory)?
- Could a duration selector be learned to adaptively choose from the current state, removing the dependence on preset per- schedules?
- How were the per- execution schedules in the main experiments determined? Were the evaluation manifests, or data overlapping with them, used in this process?
- In Table 1, PriorOnly on Cube drops sharply from 75.3% at to 1.3% at . What explains this?
11 comments
Response to Reviewer tShe (2/6)
Q2: What would Base LeWM achieve on LIBERO with the same 300-candidate × 30-round CEM configuration as on PushT/Cube (or under equal compute)? And what would PRISM achieve under the same execution schedule as SAGE?
Thank you for these suggestions. We conduct two additional experiments to address these questions.
First, we evaluate Base LeWM on LIBERO with both SAGE's original search budget (64 candidates and one ranking pass) and the CEM budget as on PushT(300 candidates and 30 rounds).
Table R3: Success rates (%) of Base LeWM and SAGE on full-episode LIBERO tasks under different search budgets.
| Method | Search budget | Scene2 | Caddy |
|---|---|---|---|
| Base LeWM | 64 candidates, 1 ranking pass | 0.0 | 0.0 |
| Base LeWM | 300 candidates, 30 CEM rounds | 0.0 | 0.0 |
| SAGE | 64 candidates, 1 ranking pass | 48.7 | 58.0 |
Under both configurations, Base achieves 0.0% success on Scene2 and Caddy, compared with 48.7% and 58.0% for SAGE on the same queries. The results have been added to Appendix A.7 of the revised version.
Second, we re-evaluate PRISM's released action prior using CEM under SAGE's exact queries, search budget, and execution schedule, ensuring a controlled comparison.
Table R4: Success rates (%) of SAGE and PRISM-CEM under matched search budgets and execution schedules.
| H | PushT (SAGE / PRISM) | Cube (SAGE / PRISM) |
|---|---|---|
| 50 | 81.3 / 52.0 | 76.0 / 62.0 |
| 100 | 72.7 / 11.3 | 85.3 / 71.3 |
| 150 | 64.7 / 14.0 | 67.3 / 43.3 |
As shown in the table, SAGE consistently outperforms PRISM across all evaluated horizons, with particularly large improvements at longer horizons. The results for all six horizons have been added to Appendix A.25. These results further suggest that SAGE's improvements cannot be achieved simply by increasing the search budget or changing the execution schedule.
Q3: Can you provide a failure attribution: do failures stem mainly from unreachable or inaccurate subgoals, or from world-model prediction/scoring errors?
Thank you for this suggestion. To disentangle the contributions of subgoal errors and world-model prediction errors, we conduct a controlled intervention study on the first 15-step decision. Specifically, we investigate two factors: (1) Subgoal accuracy: replacing SAGE's predicted first subgoal with the corresponding ground-truth subgoal from the expert trajectory; and (2) Prediction accuracy: keeping the subgoal, candidate actions, and latent distance unchanged, while replacing the world model's predicted endpoint of each candidate with the endpoint actually reached by executing that candidate in the simulator, encoded by the same frozen encoder. This corresponds to scoring candidates with a perfect predictor. By combining these two interventions, we examine the individual and joint effects of the two error sources.
We collect a run on LeWM Cube at H=150, achieving 104/150 successes, and verify that replay reproduces all 150 outcomes, including every CEM round of the first decision. For each of the 46 failures, we restart from the saved state and rerun the full first decision (300 candidates, 30 CEM rounds), modifying only the subgoal and/or the candidate endpoints used in every CEM round. We then continue execution using SAGE.
Table R5: Number of failures rescued by correcting the subgoal and/or the candidate endpoints at the first decision on LeWM Cube at H=150.
| Subgoal | Candidate endpoints in CEM | Rescued failures |
|---|---|---|
| Generated subgoal | World-model prediction | 0/46 |
| Expert subgoal | World-model prediction | 22/46 |
| Generated subgoal | Simulator endpoint | 23/46 |
| Expert subgoal | Simulator endpoint | 19/46 |
The results show that correcting either the subgoal or the world-model predictions rescues a similar number of failures (22/46 and 23/46, respectively), suggesting that both error sources contribute comparably. The two single corrections rescue 15 failures in common, indicating that the two error sources often co-occur. Overall, these results indicate that failures arise from both inaccurate subgoal predictions and world-model prediction errors, rather than being dominated by either factor alone. The results have been added to Appendix A.34 of the revised version.
Response to Reviewer tShe (3/6)
Q4: Does SAGE remain effective when trained on non-expert or random play data, or when the goal lies outside the training distribution (i.e., not on any recorded trajectory)?
Thank you for this question. We conduct two additional experiments to evaluate SAGE's effectiveness beyond expert demonstrations and its generalization to out-of-distribution (OOD) goals.
First, we evaluate SAGE on two datasets collected without expert policies. Specifically, the training process on PointMaze-Large uses goal-agnostic random exploration, where uniformly sampled actions are executed without access to goals or the map. TwoRoom uses the released LeWM dataset collected by a noisy heuristic controller.
Table R6: Success rates (%) of Base LeWM and SAGE using datasets collected without expert policies.
| Benchmark | Data collection | Base | SAGE |
|---|---|---|---|
| PointMaze-Large (H=150) | Goal-agnostic random exploration | 0.0 | 56.0 |
| TwoRoom (H=75) | Noisy heuristic controller | 30.0 | 90.0 |
SAGE achieves 56.0% success on PointMaze-Large and 90.0% on TwoRoom, substantially outperforming Base LeWM on both benchmarks. The results have been added to Appendices A.29 and A.30.
Second, to evaluate whether SAGE can reach goals that are not represented in the training trajectories, we conduct an additional experiment on PushT with OOD goals. Specifically, the joint pusher–block configurations of these goals lie 1.25–2.0 normalized units away from the training data support. We compare SAGE against Base LeWM, GCBC, and Hierarchical BC.
Table R7: Success rates (%) on original and moderate OOD PushT goals.
| Method | Original goals | Moderate OOD goals |
|---|---|---|
| Base | 4.7 | 5.3 |
| GCBC | 6.7 | 0.7 |
| Hierarchical BC | 14.0 | 4.0 |
| SAGE | 62.0 | 40.0 |
SAGE succeeds on 93/150 original goals (62.0%) and 60/150 moderate OOD goals (40.0%) in two runs on the same 150 recorded queries. SAGE's success rate drops by 22.0 percentage points and remains above the evaluated baselines. Appendix A.28 reports the evaluation protocol and results.
These results indicate that SAGE can remain effective when trained on non-expert data and generalize beyond the recorded trajectories.
Q5: Could a duration selector be learned to adaptively choose from the current state, removing the dependence on preset per-schedules?
Thank you for this suggestion. To investigate whether action durations can be selected adaptively rather than relying on preset schedules, we implement a lightweight learned duration selector. It consists of two linear classifiers, one for 20-step and one for 25-step execution, which take the current latent state, the goal difference, and the remaining planning horizon as inputs. Each classifier predicts whether switching from 15 steps to that duration at the current decision improves, preserves, or worsens final task success; the selector chooses the option with the largest positive expected gain and defaults to 15 steps otherwise. Training labels come from paired interventions on 64 PushT H=150 pilot episodes disjoint from the test set, and the trained selector is frozen for all horizons.
Table R8: Success rates (%) of fixed 15-step execution and the learned duration selector on PushT. The fixed baseline differs from the mixed-duration schedule used in the main results.
| Horizon | Fixed 15-step | Learned selector | Gain |
|---|---|---|---|
| H=100 | 73.3 | 75.3 | +2.0 % |
| H=150 | 64.0 | 66.0 | +2.0 % |
As shown in the table, the learned selector slightly improves success rates by 2.0 percentage points across different horizons. These results demonstrate the feasibility of learning adaptive action durations from the current state, reducing reliance on manually preset duration schedules. Further details have been added to Appendix A.27.
Response to Reviewer tShe (1/6)
We thank the reviewer for the detailed and constructive review. We will first address each question in the following, and then discuss each weakness.
Q1(a): Can you add goal-conditioned BC or hierarchical imitation baselines?
Thank you for this suggestion. We have added goal-conditioned behavior cloning (GCBC) and hierarchical behavior cloning baselines, all trained on the same data as SAGE. These policies are executed without world-model search, allowing us to evaluate how much can be achieved through direct imitation alone. We first consider two GCBC variants: (1) GCBC (frozen encoder) uses LeWM's pretrained visual encoder with frozen parameters to extract visual features, followed by a goal-conditioned action policy; (2) GCBC (end-to-end visual) takes visual observations as input and directly predicts goal-conditioned action distributions. We further include Hierarchical BC, which models actions through a subgoal generator and a subgoal-conditioned GCBC, evaluating both sampling from its learned action mixture and deterministic selection of the most likely mode.
Table R1: Success rates (%) over 150 queries for behavior cloning baselines and SAGE. PushT and Cube are evaluated at H=150, while Scene2 and Caddy report full-episode LIBERO success rates.
As shown in the table, SAGE achieves the highest success rate across all four tasks, outperforming the best BC baseline by 48.7, 38.7, 13.3, and 5.3 percentage points on PushT, Cube, Scene2, and Caddy. These results indicate that direct imitation of training trajectories alone is insufficient to achieve SAGE's performance, highlighting the importance of combining subgoal-conditioned planning with a latent world model. The results have been added in Appendix A.24 of the revised version.
Q1(b): How does SAGE compare with the concurrent work FF-JEPA?
Thank you for your suggestion. We reimplement FF-JEPA to enable a direct comparison. On PushT and Cube, both methods use fixed 25-step action chunks at H=150; on Scene2 and Caddy, we train FF-JEPA's diffusion variant on the same 8,000 demonstrations and use the same full-episode queries, 15-step commitment, and recovery budget as SAGE (Appendix A.32).
Table R2: Success rates (%) of FF-JEPA and SAGE under matched execution settings. PushT and Cube use H=150 with fixed 25-step planning and execution; Scene2 and Caddy report full-episode LIBERO success.
As shown in the table, SAGE significantly outperforms FF-JEPA on these benchmarks. Besides, we also consider FF-JEPA's closest goal-conditioned analogue in our setup, i.e., Subgoal Scoring, which achieves 51.3% success on Cube at H=150. These results further highlight the effectiveness of our method.