- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper argues that hierarchical JEPA planners can produce latent sub-goals that do not correspond to any physically realisable state, and introduces a decoder-free realisability residual to measure this. It proposes METRO-WM, a training-free macro planner that builds a graph over encoded frames from recorded episodes, connects consecutive frames within an episode and visually similar frames across episodes, and selects sub-goals by shortest-path search so that every sub-goal is a recorded state. Experiments on PushT, OGBench Cube, TwoRoom, and Reacher with LeWM and PLDM as micro planners report higher long-horizon success and lower online latency than flat and hierarchical baselines.
Critical strengths
-
At the longest goal distance, METRO-WM improves clearly over both flat planning and the hierarchical baselines evaluated, on all four environments and with both micro planners, while running faster online. The long-horizon gain is therefore real and not specific to one environment or one world model.
-
The macro planner requires no training and only two hyperparameters, so it is cheap to apply on top of any frozen latent world model.
Critical weaknesses
-
The paper does not discuss the assumptions the method depends on or the conditions under which it stops working. METRO-WM can only issue sub-goals that exist in the recorded data and can only stitch episodes where latent-similar frames are physically interchangeable. For example, the method may not work when the goal lies in a region of the state space that no demonstration has visited, since the graph then contains no usable sub-goal and the method falls back to flat planning, whose behaviour is not evaluated. It is also exposed to perceptual aliasing, where physically different states have similar latents and a bridge edge connects them with a cost the micro planner cannot meet; this is a known problem in retrieval-based planning that Emmons et al. (2020), cited by the paper, explicitly address, and the paper does not discuss it. The discussion of limitations in the paper falls far short of what the method requires. A more detailed discussion of the assumptions and applicability of the approach, together with experiments that show where it breaks, is required.
-
The ablations do not test the factor the paper attributes its gains to. The central claim is that retrieved sub-goals succeed because they are physically realisable states, but METRO-WM differs from the hierarchical baselines in several ways at once, such as planning over a precomputed shortest-path cost for every recorded frame, replacing the learned macro predictor and its CEM search with a non-parametric lookup, and stitching routes across demonstrations, and no experiment varies realisability alone. The paper therefore shows that the retrieval planner works, but not that realisability is the reason it works.
Questions for authors
-
Can the authors provide the additional analysis requested above: an explicit statement of the assumptions under which METRO-WM applies, and experiments that show how it behaves when these assumptions are violated?
-
The related work in the main text is limited to two paragraphs, and the extended discussion in Appendix F covers hierarchical latent planning only within the JEPA family. Latent sub-goal generation in learned world models has a longer history outside JEPA, including work that addresses the same realisability concern. Can the authors provide a more thorough discussion of how METRO-WM relates to this broader line of work?
4 comments
Follow-up on our response
Dear Reviewer, thank you again for your review. As the discussion period closes on Nov 18, we wanted to check whether our response has addressed your concerns. We would be happy to clarify anything further or provide additional details. If our responses have resolved your concerns, we would be grateful if you would consider updating your assessment.
Follow-up Questions After Rebuttal
Thank you for the detailed response and the new appendix on assumptions. The fallback ablation, the experiment in R3-b, and the expanded related work address several of the points I raised. However, the result of R3-b changes how I read the contribution, and the new assumptions raise further questions about the design. My remaining concerns are about what the paper contributes and whether the method is internally consistent, rather than about missing experiments. The questions below ask for explanation rather than new experiments, and whether I adjust my rating will depend on the answers. I am keeping my score at 2 for now.
-
R3-b shows that making Hi-LeWM-C's sub-goals realisable gives only a small, non-significant gain, and you conclude that realisability alone is not sufficient and attribute the rest to shortest-path search over recorded experience. Shortest-path retrieval over recorded states with cross-episode connections and re-planning is the structure of SoRB, SPTM, and SGM. Setting realisability aside, what is the technical contribution of Metro-WM relative to this line of work? The title, abstract, and the end of Section 4 still present the method as a fix for unrealisable sub-goals; please clarify how the framing will be revised to match the finding.
-
A2 acknowledges that bridges can connect physically different states with similar latents, and offers three mitigations: the ε cap, frame-based pricing, and re-entry. All three are functions of latent distance. Aliased pairs have small latent distance by definition, so they pass the ε cap, receive a low price, and re-entry attaches an aliased current state to the wrong node. How do these mitigations address errors caused by aliasing, as opposed to errors where the bridge is correct but the micro planner fell short?
-
A4 assumes that latent distance grows with the number of frames at short gaps, yet Eq. 5 requires a running maximum to make the empirical quantile Λ(d) monotone, which means the assumption does not hold on the paper's own data even at small d. How should this be reconciled, and how is Eq. 5 defined where Λ(d+1) = Λ(d)?
-
When the graph is disconnected, Section 5.3 routes to the reachable vertex closest to the goal in latent space. Appendix D shows that latent distance saturates at long range, so this choice is uninformative in exactly the case it is meant to handle. Could the authors explain this design, and whether it accounts for the no-bridge variant in R1-e falling below flat planning on PushT, TwoRoom, and Reacher?
Response to Follow-up Questions
We are grateful for your detailed follow-up and constructive feedback. We answer each question below:
-
SoRB, SPTM and SGM were developed for RL and navigation. Metro brings graph retrieval to planning with frozen JEPA world models and differs from them in four ways:
- Metro is training-free. The works you mention learn edge distances with a value function (SoRB) or a reachability network (SPTM), on which SGM builds, and execute with a trained policy. Metro instead prices edges in demonstrator frames from the corpus's own latent statistics and executes with the world model's zero-shot MPC. It thereby inherits the world model's limitations, such as aliasing (A2). Training a distance as in SPTM or SGM on top of Metro could filter aliased bridges and further improve performance, at the cost of training; we leave this to future work.
- Unlike SoRB and SGM, Metro needs no rewards or online interaction: SoRB learns its distances and policy with goal-conditioned RL, and SGM removes faulty edges by executing them at test time. Metro's graph is built from offline, reward-free expert or random-action data.
- Metro's sub-goals are recorded latent states that replace the goal in the world model's planning objective (Eq. 1), rather than goals for a goal-conditioned policy.
- We evaluate on manipulation (PushT, Cube, Reacher) as well as navigation tasks.
To our knowledge, Metro is the first to show that, for JEPA world models, a training-free retrieval macro planner matches or outperforms trained hierarchies (Hi-LeWM-C, H-JEPA, HWM).
On framing, we agree that realisability is necessary but not sufficient. The abstract and the end of Section 4 now say that Metro's sub-goals are realisable and selected by shortest-path search over recorded experience. We keep the title, as realisable sub-goals are what our analysis shows existing hierarchical JEPA approaches lack, rather than the cause of Metro's gains.
-
We apologise for the confusing wording in A2. We wrote the sentence about unreachable bridges in general, not specifically about aliasing: the ε cap and frame pricing mitigate long bridges, and neither applies to aliased pairs. We have reworded A2 to make this clear. As A2 states, aliasing is a property of the world model's representation, which the micro planner's objective shares, and Metro cannot detect it. Re-entry does not detect it either; it only recovers when a bridge leads to a sub-goal the micro planner cannot reach, by re-planning from wherever the planner landed. If the current state itself is aliased, re-entry can attach it to the wrong vertex; if this happens repeatedly, A2 is violated and Metro fails.
-
Empirically, A4 holds at 99.6% of the points of the pricing curves across all benchmarks in both our setup and the H-JEPA protocol; the only exception is Λ(2) on TwoRoom (7.83 < Λ(1) = 8.22). The running maximum is a safeguard for such edge cases, so that a larger latent gap is never charged fewer frames.
You are right that Eq. 5 is undefined when Λ(d+1) = Λ(d). To make Eq. 5 well defined, we change the interval from [Λ(d), Λ(d+1)] to [Λ(d), Λ(d+1)). A tied interval is then empty, so a gap equal to the tied value falls into the next interval, whose denominator is non-zero, or is clipped to H if the tie is at the end of the curve. In practice, no bridge falls in a tied interval, as every TwoRoom bridge is shorter than Λ(1), so this does not change any of our results.
-
You are right on both points. Routing by latent distance is our only fallback, as latent distance is the only signal left when the graph has no route from the current state to the goal, i.e. when A1 (coverage) is violated. Metro then steps to the entry candidate closest to the goal (we corrected the imprecise wording in Section 5.3) if it is closer than the current state, and otherwise hands the goal to the micro planner. We agree that this choice is uninformative when the goal is far, as the latent distance saturates; the micro planner has the same problem, as its objective uses the same distance. With the full corpus, this fallback is used in at most 0.2% of macro decisions.
Yes, this accounts for the no-bridge variant falling below flat planning. As noted in R1-e, few single demonstrations connect start and goal, so without bridges Metro mostly falls back, which is uninformative at these distances. We revised the paper to state this in Appendix I (A1) and Appendix E.3.
Thank you again for your rigorous review; your feedback has significantly improved our paper. We hope these answers address your remaining concerns.
Response to Reviewer 6CTP
R3-a. Assumptions, limitations, where it breaks. Thank you for the suggestion. We added a new appendix section (App. I), detailing both assumptions and limitations. In summary, Metro assumes:
R3-b. Is realisability the reason it works? We do not claim that realisability alone explains the gain, and intuitively it cannot. A realisable sub-goal that leads away from the goal, or lies beyond the micro planner's reach, does not help. A good sub-goal must be both realisable and useful, "both within π_lo's reach and on a route to meet the goal" (Sec. 5). Retrieval provides the first; the shortest-path search over recorded experience provides the second.
To vary realisability alone, as requested, we kept Hi-LeWM-C's macro and micro planners unchanged and replaced each of its sub-goals with the nearest realisable state, found by the same r(z) search as in Sec. 4. Success (%), d = 75, at Hi-LeWM-C's budget T = 150
Making Hi-LeWM-C's sub-goals realisable gives only a small gain that is not significant, far below Metro-LeWM’s performance: realizability alone is not sufficient.
R3-c. Related work beyond JEPA. Thank you for the suggestion. To clarify, App. F already covers two lines of work outside JEPA:
We agree that latent sub-goal generation outside of JEPA deserved its own discussion. We added a paragraph in App. F, and extended the pointer to it in the main text (Sec. 2), on methods that generate sub-goals, in a learned latent space or in state space, and constrain them towards realisability (the sub-goal encodes a real state), reachability (the lower level can attain it), or both. They do so through:
These mechanisms make generated sub-goals more likely to be realisable or reachable, and most rely on additional trained components, but none guarantees realisability.