- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the challenge of extending short-horizon JEPA world models to long-horizon task execution. The authors propose LeCompass, a framework consisting of three main components: (1) State-indexed Proposer, which dynamically generates intermediate latent targets based on observation history and the final goal; (2) CompassFlow, which combines a Flow Matching-based trajectory prior, frozen world-model scoring, and Source-CEM search to select actions under a limited computational budget; and (3) Observation-grounded Execution, which employs spatial readouts and execution memory to support phase transitions, closed-loop control, and failure recovery.
The method is evaluated in three simulated environments: PushT, TwoRoom, and Cube. Predictive ranking improves success rates by approximately 40.2 and 43.9 percentage points on the first two tasks, respectively, while the recovery mechanism substantially improves task completion following contact loss.
Critical strengths
- Important research problem and a systematic framework design.
The paper addresses the limited task-level execution capabilities of short-horizon world models by integrating dynamic target generation, budget-constrained trajectory search, and observation-grounded control. State-indexed targets significantly improve success rates in cross-episode composition experiments, demonstrating the value of replanning based on observed execution progress.
- Comprehensive experiments with convincing analyses of component contributions.
The authors conduct experiments across three environments and use matched comparisons to evaluate the effectiveness of predictive ranking, target generation, and execution recovery. Predictive ranking yields substantial improvements on PushT and TwoRoom, while the recovery mechanism increases semantic completion on Cube from 5.8% to 80.4%. The paper also transparently reports performance differences across tasks, strengthening the credibility of its findings.
Critical weaknesses
Limited framework generalizability and incomplete attribution of some experimental improvements. The complete observation-grounded execution interface is primarily evaluated on Cube and relies on task-specific spatial readouts and control rules. Its transferability to other tasks has not been explicitly validated. In addition, the comparison between Source-CEM and Action-space CEM on PushT changes multiple factors simultaneously, including the candidate prior and target refresh frequency, making it difficult to isolate the independent contribution of source-space search.
Questions for authors
-
What is the key distinction between the State-indexed Proposer and existing goal-conditioned latent subgoal planning methods? Could the authors further clarify its unique underlying mechanism?
-
Could the authors provide more strictly controlled experiments to disentangle the respective contributions of the Source-CEM search space, trajectory prior, and target refresh strategy?
-
How much task-specific design is required to transfer the complete execution interface to other robotic tasks? Could the authors also provide an analysis of end-to-end inference latency and computational overhead?
Response to Reviewer CmnZ
Thank you for the constructive questions. We added controlled comparisons and runtime measurements.
1. Execution settings, prior/sampling, and search coordinates
The budget-192 PushT comparisons match target refresh, warm-start, observation history, and execution settings: five-action prefixes, target refresh every plan, and no warm-start. At 192 predictive evaluations per plan, we distinguish three comparisons:
Our primary aim is to improve candidate efficiency through a learned trajectory prior. Demonstrations help concentrate proposals around useful behavior. Proposer references are not guaranteed to correspond to reachable physical states or stable demonstration timestamps and lack paired expert action trajectories. We therefore separate target-independent generation from target-dependent scoring. Source refinement searches through this generator; the shared-initialization comparison measures its additional benefit over action updates.
2. State-indexed proposer versus existing subgoal planning
Thank you especially for asking about the mechanism and the idea behind LeCompass. Inspired by hierarchical JEPA, our intuition views encoded observations as nodes and demonstration transitions as local paths: the proposer learns a goal-directed local reference, a “compass” for successive short-horizon plans.
We build on goal-conditioned latent subgoal prediction; the distinction we emphasize is its interface to a single frozen short-horizon JEPA. The proposer learns from demonstration transitions, conditioned on history, final goal, and span. Its prediction is a local scoring reference, recomputed after execution. This reference guides predictive candidate evaluation, while the trajectory prior remains target-independent; the world model and its prediction horizon remain unchanged.
Unlike Fixed-k's elapsed-time waypoint schedule, task progress is re-indexed from observations. Unlike SuSIE, we use latent scoring references rather than generated image subgoals; unlike HWM, we retain a single frozen short-horizon predictor rather than a multi-timescale predictor hierarchy. Reference updates do not require reaching an exact retrieved waypoint.
This motivates testing new context–goal combinations: cross-episode composition improves from 2.9% to 63.7% on PushT and 39.1% to 60.5% on TwoRoom (Table 1; Sec. 4.1).
3. Inference latency and overhead
At 192 evaluations per plan, decision-time medians for action-CEM / ranked IID / Source-CEM are:
Source-CEM adds about 10 ms of Flow decoding across three dependent rounds; IID generates and scores in one batch. Source-CEM is not faster per decision at this matched budget (Appendix I.3).
For system timing, Direct and Fixed-25 retain LeWM's original 300 candidates × 30 iterations = 9,000 predictive evaluations per plan and their original execution configurations; full LeCompass uses 192 evaluations. On 50 tasks/domain, mean seconds/task for Direct9000 / Fixed-25 9000 / LeCompass192 are:
Timing includes observation, planning, execution, simulation, and termination, excluding model loading/setup. PushT uses five-environment batch-amortized costs. Speedups over Direct are 1.16× / 7.94× / 2.67× (Sec. 6.4; Appendix I.3).
4. Execution-interface adaptation
Transfer requires task-specific readout targets/supervision, action assignments, phase/contact conditions, and termination/recovery rules. Readouts are learned from frozen JEPA patch tokens; memory and execution logic are explicit designs. Development rollouts exposed grasp-timing and mid-air-drop failures, while CLS probes showed poor decoding of fine relative geometry (Appendix D), motivating these complementary components.
The complete interface is validated on Cube. Bare-planner ranking improves native reach from 6.6% to 25.4% (Table 4(a)); with the full interface, execution rules modify action channels after scoring and ranking's additional benefit is small. Integrating spatial and contact constraints into candidate generation and predictive evaluation is a future direction toward aligning scored and executed actions.