acceptodds
Under review as a conference paper at ICLR 2027

Beyond Action Prediction: Learning Search-Free Planning in Latent Space

Abstract

Latent world models learn action-conditioned state transitions in a compact representation space, providing a predictive model for visual planning and control. However, planning with these models typically requires costly test-time optimization over action sequences. We propose to replace this online search with a goal-conditioned inverse model that directly predicts actions from current and goal observations. A key challenge is that action prediction alone neither ensures that goal representations encode action-relevant information nor that predicted actions are consistent with the learned latent prediction model. We introduce Hindsight Inverse JEPA Planning (HIJP), a search-free framework that addresses both aspects during training. HIJP relabels future observations along offline trajectories as goals, providing action supervision across multiple temporal horizons. By retaining gradients through the goal representations, this supervision directly shapes the visual representation. HIJP further feeds predicted actions into the latent predictor model and aligns the resulting next-state representations with encoded successor observations, enforcing action-transition consistency. At inference time, the inverse dynamics model and latent predictor models are applied alternately to generate multi-step action sequences without online search. Across four visual control tasks, HIJP achieves 86.8% overall success and a 1.9–16.5× planning speedup over search-based planning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Latent World Models, Goal-Conditioned Visual Control, Inverse Dynamics, Search-Free Planning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oBJeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HIJP jointly trains a visual encoder, forward dynamics, and a goal-conditioned inverse head. The inverse head learns the immediate action toward sampled future observations, while a cycle objective checks that this action predicts the observed next state. At inference, alternating inverse and forward predictions produces an action sequence without iterative search. Four benchmarks evaluate success across increasing goal distances, with component ablations and timing comparisons.

Critical strengths
  1. The coupling is coherent and reasonably well isolated. Table 3 tests future-goal sampling, gradients through the goal representation, the cycle loss, and replacing direct inference with CEM on the same representation. These controls provide useful evidence beyond comparing separately trained systems.

  2. The manipulation results are promising: Table 1 reports overall success of 86.8%, versus 81.9% for INTACT, with particularly strong difficult-goal results on PushT and OGCube. Table 2 also demonstrates a substantial runtime advantage over the evaluated CEM planner under matched replanning schedules.

Critical weaknesses
  1. The reliability and generalization of the success gains need clearer evidence. Each task and difficulty cell uses 50 evaluation instances. Appendix A.1 splits overlapping training windows into training and validation sets, while A.2 samples evaluation starts from offline trajectories. This does not establish leakage, but the relationship between training trajectories and evaluation episodes is insufficiently explicit. The 4.9-point aggregate advantage should be supported by paired uncertainty estimates and a clear episode-level holdout protocol.

  2. The efficiency evidence is narrower than the comparison to existing planning methods suggests. Timing uses batches of 50 episodes and a CEM configuration with 300 candidates and 30 iterations; the closest search-free baseline, INTACT, is not timed. Success also does not improve uniformly, notably on Reacher. A success-versus-runtime comparison across CEM budgets and against INTACT would clarify the practical advantage.

Questions for authors
  1. Are evaluation trajectories or initial/goal observations disjoint from training at the episode level? Please specify the episode-level split and which observations are available during training.

  2. Can you compare total planning latency and success against INTACT and several CEM budgets, at batch size one as well as 50, under the same replanning schedule?

5 comments
AuthorsOpenReview ↗

Response to Reviewer oBJe (Part 1)

Thank you for your careful review and constructive comments. We are glad that you recognize how HIJP combines the inverse and forward dynamics models and how the component ablations support the individual design choices. We also appreciate your positive assessment of HIJP’s performance on Push-T and OGCube and its efficiency advantage over the CEM planner we evaluated at the same replanning frequency. To address your questions about evaluation independence, the reliability of the results, and the efficiency comparison, we conducted additional experiments and respond to each point below.

Q1: Are evaluation trajectories or initial/goal observations disjoint from training at the episode level? Please specify the episode-level split and which observations are available during training.

A1: Thank you for pointing out that the original manuscript did not clearly explain the relationship between training trajectories and evaluation episodes. The window-level split described in Appendix A.1 was used for training and validation. After reconstructing the evaluation tasks under the original protocol, we found that some evaluation instances in Table 1 came from offline trajectories that contained training windows.

To assess performance on unseen trajectories, we split the data in each environment 90/10 by complete trajectory and retrained both methods on the same 90% of trajectories, following INTACT’s released training and inference configurations for the baseline. All observations and actions used to train both methods, including future observations used to construct training goals, came from these trajectories. The remaining 10% of trajectories were used exclusively for evaluation, and all evaluation start states and goals were sampled from these held-out trajectories.

For each evaluation seed and difficulty level, both methods were evaluated on the same 50 start–goal pairs. We report the mean and sample standard deviation across the three evaluation seeds.

TaskMethodEasyDefaultHardExtremeAvg.
TwoRoomHIJP100.0 ± 0.0100.0 ± 0.0100.0 ± 0.098.7 ± 1.299.7 ± 0.3
TwoRoomINTACT68.7 ± 4.249.3 ± 7.643.3 ± 6.130.0 ± 2.047.8 ± 3.2
ReacherHIJP50.7 ± 6.488.0 ± 6.092.7 ± 3.194.0 ± 2.081.3 ± 2.9
ReacherINTACT52.7 ± 8.192.0 ± 8.098.0 ± 2.0100.0 ± 0.085.7 ± 3.6
Push-THIJP92.7 ± 4.683.3 ± 6.156.0 ± 3.540.7 ± 1.268.2 ± 0.3
Push-TINTACT94.7 ± 4.272.0 ± 5.345.3 ± 9.011.3 ± 5.055.8 ± 4.2
OGCubeHIJP96.7 ± 1.294.7 ± 2.393.3 ± 4.296.7 ± 3.195.3 ± 2.4
OGCubeINTACT100.0 ± 0.0100.0 ± 0.065.3 ± 3.178.7 ± 6.486.0 ± 2.0

Under the trajectory-level holdout protocol, HIJP and INTACT achieved average success rates of 86.1% and 68.8%, respectively, with equal weight given to each of the 16 task–difficulty combinations. The difference was 17.3 percentage points. HIJP led INTACT by 51.9 percentage points on Two-Room, 12.4 on Push-T, and 9.3 on OGCube, and trailed INTACT by 4.4 percentage points on Reacher. In the revised manuscript, we will explicitly describe the trajectory overlap in the original evaluation and report the split protocol and results of the holdout experiment.

AuthorsOpenReview ↗

Response to Reviewer oBJe (Part 2)

Q2: What is the paired uncertainty around HIJP’s reported 4.9-percentage-point aggregate advantage over INTACT, given 50 evaluation episodes per task and difficulty level?

A2: To quantify paired uncertainty, we used Push-T, where both methods were trained with three training seeds under the trajectory-level holdout protocol. Within each training run, HIJP and INTACT were evaluated on identical start–goal pairs (3 evaluation seeds × 50 pairs per level). We report HIJP − INTACT success-rate differences in percentage points, with 95% paired bootstrap confidence intervals obtained by resampling start–goal pairs within each evaluation seed (20,000 resamples).

Training seedEasyDefaultHardExtremeAvg.
0−3.3 [−8.0, +0.7]−2.7 [−10.0, +4.0]+9.3 [−0.7, +19.3]+17.3 [+8.7, +26.0]+5.2 [+1.2, +9.0]
42−6.0 [−10.7, −2.0]−0.7 [−8.0, +6.7]+5.3 [−4.7, +14.7]+13.3 [+4.0, +22.0]+3.0 [−1.0, +6.8]
3072−2.0 [−7.3, +3.3]+11.3 [+2.7, +20.0]+10.7 [+0.7, +20.7]+29.3 [+20.7, +38.0]+12.3 [+8.2, +16.5]

The paired difference in the four-level average was positive in all three training runs (+5.2, +3.0, and +12.3 percentage points), and its confidence interval excluded zero in two of the three runs. The advantage was concentrated at Extreme, where the confidence interval excluded zero in every run (exact McNemar test, p ≤ 0.008). At Easy, INTACT achieved higher success in all three runs, with the confidence interval excluding zero in one run. These results indicate that HIJP’s advantage on Push-T comes mainly from longer-horizon tasks rather than a uniform improvement across levels.

AuthorsOpenReview ↗

Response to Reviewer oBJe (Part 3)

Q3. Can you compare total planning latency and success against INTACT and several CEM budgets, at batch size one as well as 50, under the same replanning schedule?

A3: Thank you for the suggestion. We compared HIJP, INTACT Direct, and LeWM + CEM with four search budgets on the same start–goal pairs and with the same replanning schedule: all methods replanned every 25 environment steps. LeWM + CEM plans with LeWM’s own encoder and forward model, and its budget is the number of candidate action sequences evaluated per planning call. We report success rates, planning time per call, and total planning time per episode at batch sizes of 1 and 50, averaged across the four environments and four difficulty levels.

Both HIJP and INTACT Direct predict latent states step by step as they generate an action sequence. The original HIJP implementation then performs two additional rollouts of the generated action sequence to compute the initial and final planning costs. In the search-free setting (M=0M=0), neither computation affects action selection. INTACT Direct instead computes its diagnostic metrics from the final latent state predicted during action generation. We therefore report timings for both the original HIJP implementation and a version that skips the two additional rollouts while keeping the original action generation procedure.

MethodB1 success (%)B50 success (%)B1 planning per call (ms)B50 planning per call, 50 envs (ms)B1 total planning per episode (s)B50 total planning per episode (s)
HIJP (original implementation)86.686.949.376.40.1400.004
HIJP (without extra rollouts)86.586.820.835.80.0580.002
INTACT Direct81.881.925.446.30.0720.003
LeWM + CEM (60)51.852.1128.81,539.90.3730.086
LeWM + CEM (250)62.162.1271.36,785.90.7400.372
LeWM + CEM (1,000)65.266.51,035.227,278.32.8381.494
LeWM + CEM (2,250)67.069.01,798.359,784.14.9313.274

With the extra rollouts skipped, HIJP took 20.8 ms per planning call at a batch size of 1, compared with 49.3 ms for the original implementation and 25.4 ms for INTACT Direct. At a batch size of 50, the corresponding times per batch were 35.8, 76.4, and 46.3 ms. Skipping the rollouts does not change the selected actions: the mean success rates (86.5% at a batch size of 1 and 86.8% at 50) were within 0.1 percentage points of the original implementation and remained above those of INTACT Direct (81.8% and 81.9%). The version without the extra rollouts was timed in a separate session from the other rows, so its comparison with INTACT Direct is approximate; the two methods perform a similar amount of computation per call (two encodings, five action predictions, and five forward-model steps). At both batch sizes, HIJP achieved a higher mean success rate and a lower total planning time per episode than LeWM + CEM at every tested search budget.

The original HIJP implementation spent more planning time than INTACT Direct; after skipping the two rollouts used only to log diagnostic costs, the two were comparable (0.058 vs. 0.072 s per episode at a batch size of 1, and 0.002 vs. 0.003 s at 50).

Reviewer_shJzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hindsight Inverse JEPA Planning (HIJP), a search-free visual planning framework based on latent world models. The method uses future observations from offline trajectories as hindsight goals to train a goal-conditioned inverse dynamics model. It further introduces an action-transition consistency objective that feeds predicted actions into a latent predictor and aligns the resulting latent states with observed successors. During inference, the inverse dynamics model and latent predictor are applied alternately to generate action sequences without online search. Experiments across four visual control environments report improvements in task success, planning efficiency, and action smoothness.

Critical strengths
  1. Reducing test-time planning costs in latent world models is a meaningful research direction with potential benefits for efficient visual control.
  2. The evaluation covers several control environments and includes component ablations and efficiency analyses, offering useful initial insights into search-free planning.
Critical weaknesses
  1. Appendix A.1 randomly divides overlapping windows from offline trajectories into training and validation sets, while Appendix A.2 constructs evaluation goals from future states along offline trajectories. The paper does not clearly establish whether evaluation trajectories are disjoint from those used during training. Consequently, evaluated state–goal relationships may substantially overlap with the demonstrated trajectories used to train the inverse model. Although all methods follow the same evaluation protocol, this does not establish that HIJP generalizes to independently sampled trajectories and novel goals rather than exploiting the behavioral structure of the recorded demonstrations.
  2. GC-IDM and INTACT have already investigated replacing online planning with goal-conditioned action prediction, with INTACT additionally supporting end-to-end representation learning. Although HIJP introduces hindsight goal sampling and action-transition consistency, its overall improvement over INTACT is 4.9 percentage points, and it performs worse on Reacher (81.5% versus 85.0%). Furthermore, GC-IDM is not included in the main experimental comparison, and the comparability of model capacity and training budgets with the closest approaches is insufficiently documented. It therefore remains unclear whether the proposed additions provide a sufficiently distinct and reliable methodological improvement.
  3. In Eqs. (6)–(7), the predicted action is passed through the learned latent predictor, and the resulting representation is aligned with the demonstrated successor state. However, when the predicted action differs from the demonstrated action, minimizing this objective does not guarantee that executing the predicted action would produce the corresponding transition. Since the inverse model and latent predictor are jointly optimized, they may adapt to each other without necessarily improving forward-dynamics accuracy under generated actions. Although Table 3 shows that the consistency loss improves task success, it does not directly establish improvements in actual transition fidelity or multi-step prediction reliability. This limits the evidence for the proposed mechanism.
Questions for authors
  1. Are the trajectories used to construct evaluation goals completely disjoint from the training trajectories?
  2. Could the authors provide evaluation results with trajectory-level data separation and independently sampled start–goal pairs?
5 comments
AuthorsOpenReview ↗

Response to Reviewer shJz (Part 1)

Thank you for your careful review. We are glad that you recognize the value of reducing test-time planning costs in latent world models and the potential of this research direction for efficient visual control. We also appreciate your observation that the evaluations across multiple control environments, component ablations, and planning efficiency analysis provide preliminary evidence for understanding the performance of search-free planning. To address your questions about generalization, how our method differs from and compares with existing methods, and how the consistency loss works, we conducted additional experiments and analyses. We respond to each point below.

Q1: Are the trajectories used to construct evaluation goals completely disjoint from the training trajectories? Could the authors provide evaluation results with trajectory-level data separation and independently sampled start–goal pairs?

A1: Thank you for pointing this out. In the original manuscript, we followed LeWM (Maes et al., 2026) in using a window-level split for training and validation and sampling start states and goals from the same offline trajectory. After checking the split, we confirmed that some of the evaluation tasks reported in Table 1 of the original manuscript were constructed from trajectories that contained training windows.

For the trajectory-level holdout experiment, we split the data 90/10 by complete trajectory, retrained both methods on the new training split, and evaluated them on the same held-out start–goal pairs, using INTACT’s released training and inference configurations for the baseline.

TaskMethodEasyDefaultHardExtremeAvg.
TwoRoomHIJP100.0 ± 0.0100.0 ± 0.0100.0 ± 0.098.7 ± 1.299.7 ± 0.3
TwoRoomINTACT68.7 ± 4.249.3 ± 7.643.3 ± 6.130.0 ± 2.047.8 ± 3.2
ReacherHIJP50.7 ± 6.488.0 ± 6.092.7 ± 3.194.0 ± 2.081.3 ± 2.9
ReacherINTACT52.7 ± 8.192.0 ± 8.098.0 ± 2.0100.0 ± 0.085.7 ± 3.6
Push-THIJP92.7 ± 4.683.3 ± 6.156.0 ± 3.540.7 ± 1.268.2 ± 0.3
Push-TINTACT94.7 ± 4.272.0 ± 5.345.3 ± 9.011.3 ± 5.055.8 ± 4.2
OGCubeHIJP96.7 ± 1.294.7 ± 2.393.3 ± 4.296.7 ± 3.195.3 ± 2.4
OGCubeINTACT100.0 ± 0.0100.0 ± 0.065.3 ± 3.178.7 ± 6.486.0 ± 2.0

Under the trajectory-level holdout protocol, HIJP’s average success rate across the four difficulty levels exceeded INTACT’s by 51.9 percentage points on TwoRoom, 12.4 on Push-T, and 9.3 on OGCube. On Reacher, INTACT outperformed HIJP by 4.4 percentage points. In the revised manuscript, we will explicitly describe the overlap between training and evaluation trajectories in the original evaluation and separately report the setup and results of the trajectory-level holdout experiment.

As in LeWM, each goal was a later observation from the same trajectory as the start state, so every start–goal pair was connected by a demonstration segment.

To evaluate sampled pairs independently, we sampled a start frame and a goal frame independently from the held-out Push-T trajectories. HIJP and INTACT used the checkpoints from the trajectory-level holdout experiment above and were evaluated on the same 150 tasks (3 evaluation seeds × 50), with up to 100 environment steps per task and the original success criterion (pusher position, T-block position, and T-block orientation all within tolerance). We did not filter the pairs for feasibility, so some tasks may not be solvable within 100 steps.

MethodSuccess rate (%)
HIJP8.0 ± 4.0
INTACT6.0 ± 2.0

On Push-T, HIJP achieved success rates of 56.0% in the Hard setting of the trajectory-level holdout evaluation and 52.0% in the additional fixed 50-frame evaluation. In both settings, the goal was a later observation from the same held-out trajectory. With independently sampled start–goal pairs, success rates were low for both methods: 8.0% for HIJP and 6.0% for INTACT.

AuthorsOpenReview ↗

Response to Reviewer shJz (Part 2)

Q2:What is HIJP’s methodological advance over GC-IDM and INTACT, given that both already use goal-conditioned action prediction in place of online planning? Could the authors provide a GC-IDM baseline and more details on model capacity and training budgets?

A2:Thank you for highlighting the relationship between HIJP, GC-IDM[2], and INTACT. All three methods replace online search with goal-conditioned action prediction. GC-IDM trains an inverse dynamics head on a frozen, pretrained LeWM and conditions it on the remaining horizon. HIJP instead trains the encoder, forward model, and inverse dynamics head jointly, does not take the horizon as input, and adds the action–transition consistency loss. INTACT is also trained end to end, but HIJP differs from it in three ways: (i) goals are sampled from any later frame in the training window, rather than only the window endpoint ; (ii) the action loss updates the goal representation, whereas INTACT stops gradients through the goal embedding; and (iii)the predicted action is passed through the forward model, and the result is supervised with the observed successor state.

To isolate the effect of joint training, we compared end-to-end HIJP with a two-stage variant. In the first stage, we used pretrained LeWM checkpoints (encoder and forward model trained with LeWM’s own objective for 10 epochs). In the second stage, we froze both and trained only an inverse dynamics head with the same architecture, hindsight goal sampling, inverse dynamics loss, and consistency loss as end-to-end HIJP, using the same number of updates and batch size. Because the forward model is frozen, the consistency loss updates only the inverse dynamics head. Here, we report a two-stage training ablation and a direct comparison with GC-IDM.

TaskMethodEasyDefaultHardExtremeAvg.
Push-TEnd-to-end HIJP95.3 ± 4.682.7 ± 6.462.0 ± 8.739.3 ± 4.269.8 ± 0.3
Push-TTwo-stage training87.3 ± 7.674.7 ± 2.350.0 ± 4.029.3 ± 3.160.3 ± 2.8
Push-TGC-IDM93.3 ± 1.279.3 ± 5.071.3 ± 8.346.0 ± 2.072.5 ± 2.2
OGCubeEnd-to-end HIJP95.3 ± 3.194.7 ± 1.292.7 ± 7.694.0 ± 3.594.2 ± 2.8
OGCubeTwo-stage training100.0 ± 0.088.7 ± 3.173.3 ± 2.380.0 ± 8.785.5 ± 1.8
OGCubeGC-IDM99.3 ± 1.282.0 ± 4.071.3 ± 6.169.3 ± 8.380.5 ± 4.3
ReacherEnd-to-end HIJP62.7 ± 7.692.0 ± 5.394.0 ± 2.094.0 ± 4.085.7 ± 0.8
ReacherTwo-stage training50.7 ± 6.490.7 ± 3.196.0 ± 2.094.0 ± 5.382.8 ± 2.9
ReacherGC-IDM39.3 ± 4.679.3 ± 5.890.7 ± 1.285.3 ± 9.273.7 ± 2.1
Two-RoomEnd-to-end HIJP100.0 ± 0.0100.0 ± 0.0100.0 ± 0.099.3 ± 1.299.8 ± 0.3
Two-RoomTwo-stage training100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0
Two-RoomGC-IDM100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0

End-to-end training improved over the two-stage variant on Push-T (69.8% vs. 60.3%) and OGCube (94.2% vs. 85.5%). On OGCube, the two-stage head’s training loss had plateaued, so the gap is unlikely to reflect under-training; the gains were largest at Hard (+19.3 points) and Extreme (+14.0). On Push-T, the two-stage head’s loss was still decreasing at the end of training, so OGCube provides the cleaner comparison. On Reacher, the two variants were close (85.7% vs. 82.8%), and on Two-Room all methods were near 100%.

Under the same replanning schedule, HIJP was more successful than GC-IDM on OGCube (+13.7 points) and Reacher (+12.0), whereas GC-IDM was more successful on Push-T (72.5% vs. 69.8%), mainly at Hard and Extreme.

Regarding model capacity, end-to-end HIJP and the two-stage variant use the same architecture and have the same total number of parameters. The table below lists the total and trainable parameter counts for each method in the Push-T experiments. The GC-IDM inverse dynamics head has 2.1 times as many parameters as HIJP’s. The official INTACT checkpoint contains 2.2 million more parameters than HIJP.

MethodTotal parametersTrainable parameters
End-to-end HIJP18,762,04018,762,040
Two-stage training18,762,040727,562
GC-IDM19,586,3601,551,882
INTACT (official checkpoint)20,955,97220,955,972

For the two-stage variant and GC-IDM, the trainable parameter counts include only the inverse dynamics head trained with LeWM frozen.

AuthorsOpenReview ↗

Response to Reviewer shJz (Part 3)

Q3:Does the consistency loss improve transition fidelity or multi-step prediction under predicted actions?

A3: Thank you for raising this issue. We agree that minimizing Eqs. (6)–(7) only trains the predictor to map the predicted action to the successor state in the demonstration. It does not guarantee that executing the action will produce the same transition. Joint optimization may also allow the inverse model and the predictor to adapt to each other. Therefore, the improvement in success rates in Table 3 alone does not establish that transition predictions are more accurate.

To test this directly, we compared the full model with the variant trained without the consistency loss on Push-T, Reacher, and OGCube. We evaluated 600 tasks per environment (4 difficulty levels × 3 evaluation seeds × 50 tasks), using the same start states and goals for both models. For each task, each model used its own inverse model to generate an action sequence spanning five model steps and its own predictor to predict the corresponding latent states in open loop. We then executed the action sequence in the simulator, encoded the resulting observations with the same model’s encoder, and computed the mean squared error between the predicted and encoded latent states. Because the two models have different latent spaces, we normalized each model’s error by the variance of its own latent states to obtain the normalized mean squared error (NMSE). The table below reports NMSE as HIJP / HIJP without LcycL_{cyc}.

EnvironmentModel step 1 (5 environment steps)Model step 3 (15 environment steps)Model step 5 (25 environment steps)
Push-T0.038 / 0.066 (−43%)0.097 / 0.182 (−47%)0.175 / 0.322 (−46%)
Reacher0.230 / 0.213 (+8%)0.692 / 0.729 (−5%)0.789 / 0.947 (−17%)
OGCube0.033 / 0.017 (+98%)0.075 / 0.094 (−21%)0.104 / 0.202 (−48%)

At the fifth model step (25 environment steps), the full model had 46%, 17%, and 48% lower NMSE than the variant without the consistency loss on Push-T, Reacher, and OGCube, respectively. At the first model step, the results varied by environment: the full model had lower error on Push-T, slightly higher error on Reacher, and higher error on OGCube, although both one-step errors on OGCube were small (0.033 and 0.017). From the third model step onward, the full model had lower error in all three environments. Because each plan is executed open-loop for five model steps before replanning, errors accumulated over this horizon affect the states the agent reaches.

This analysis measures forward-dynamics accuracy under generated actions, which the success rates in Table 3 do not capture: predictions are compared with the states actually reached in the simulator. In all three environments, the full model predicted the outcomes of its own actions more accurately over multiple steps than the variant without the consistency loss. We will include the transition fidelity analysis in the appendix and point readers to it in §4.4.

[2]Nguyen, Hoang, Xiaohao Xu, and Xiaonan Huang. "Latent Geometry Beyond Search: Amortizing Planning in World Models." arXiv preprint arXiv:2605.08732 (2026).

Reviewer_aLm4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors present a goal-conditioned visual controller, namely Hindsight Inverse JEPA Planning (HIJP), which jointly learns a visual encoder, an inverse action model, and a latent forward model. Its main additions are future-goal sampling within trajectory windows, gradients through the goal representation, and a consistency loss that aligns transitions induced by predicted actions with observed successors. At inference, the inverse and forward models alternate to construct action sequences without online search.

Critical strengths

(1). The action-transition consistency objective has a clear motivation. The inverse model and forward model are composed during deployment, so directly supervising their composition is a reasonable addition to separate action and dynamics prediction losses. The formulation is simple and its role is easy to understand.

(2). The manipulation results are promising, and the ablations provide useful evidence. HIJP achieves 42% success on PushT Extreme compared with 28% for the strongest baseline, and 96% on OGCube Hard compared with 70%. Removing the consistency loss reduces the three-task ablation average from 82.5% to 76.0%. The comparison with CEM using the same encoder and forward model is also a useful control. The gains vary across tasks: HIJP remains below INTACT on average in Reacher, and consistency has little effect on OGCube.

Critical weaknesses

(1). The stability of the reported improvements is difficult to assess. Tables 1 and 3 do not report variation across training runs, and each task–tier combination is evaluated on 50 episodes. Some differences therefore amount to only a few successful episodes. The larger gains on OGCube are encouraging, but the available results leave uncertainty about the reproducibility of the aggregate advantage over INTACT and some component-level improvements. The confidence intervals in Figure 2 describe latent-distance curves and do not address this uncertainty in success rates or training stability.

(2). The hindsight-goal ablation does not isolate the explanation offered for its benefit. Under Equation (4), uniformly sampling an anchor and then a future goal gives greater probability to shorter temporal offsets. The endpoint-goal baseline has a different offset distribution. Thus, the comparison changes both goal selection and the balance of short- and long-horizon supervision. Table 3 supports the effectiveness of the sampling recipe as a whole, while leaving the specific contribution of goal diversity unresolved.

Questions for authors

(1) Do the trajectories used to construct the evaluation episodes overlap with the training trajectories? Appendix A.1 describes a window-level split for training and monitoring, while Appendix A.2 samples evaluation starts and goals from offline trajectories without specifying their relationship to the training subset. The window-level split alone does not establish overlap in the final evaluation. However, the current description does not distinguish generalization to held-out trajectories from control starting at states represented in the training data. This distinction matters when interpreting the scope of the results.

(2) Are the results in Tables 1 and 3 from individual training runs or aggregated across multiple runs? This would help interpret the uncertainty in the comparisons discussed above.

Additional Comments

The following points concern presentation and interpretation and are not decisive for my current assessment.

(1) The discussion could distinguish more clearly between the one-step alignment directly supervised by the consistency loss and the behavior of recursive rollouts.

(2) The evaluation tiers change both goal offset and execution budget, so their ordering need not correspond to monotonically increasing difficulty. This is particularly visible in Reacher and warrants more cautious interpretation of the tier names.

7 comments
AuthorsOpenReview ↗

Response to Reviewer aLm4 (Part 1)

Thank you for your careful review and constructive comments. We are glad that you recognize the motivation for the design of the action–transition consistency loss. The inverse action model and the forward dynamics model are used in combination at inference time, so it makes sense to directly supervise their composition during training. We also thank you for recognizing HIJP’s performance on PushT Extreme and OGCube Hard and for pointing out that the ablation experiment on the consistency loss and the comparison with CEM, which uses the same visual encoder and forward dynamics model as HIJP, provide valuable evidence. To address your questions about the stability of the results, the evaluation protocol, and the interpretation of the goal-sampling ablation, we added experiments with multiple training seeds and trajectory-level holdout experiments. Below, we respond to each point and clarify the relevant conclusions.

Q1: Are the results in Tables 1 and 3 from individual training runs or aggregated across multiple runs? How robust are the reported gains over INTACT and the component-level improvements, given that each task–tier pair is evaluated on 50 episodes?

A1: Thank you for raising this question. In Tables 1 and 3 of the original manuscript, each method or ablation configuration was run independently once per task. Each task–difficulty combination included 50 evaluation episodes. The reported averages were computed across difficulty levels or across task–difficulty combinations.

We agree that the confidence intervals in Figure 2 capture variability in latent distances and do not indicate whether success rates vary across training runs.

To address this, we ran additional HIJP experiments on Two-Room, Push-T, Reacher, and OGCube using three training seeds. We report success rates (%) as the mean ± sample standard deviation across training seeds. For the average across the four difficulty levels, we first computed the average separately for each seed and then calculated the mean and sample standard deviation of those averages.

TaskEasyDefaultHardExtremeAvg.Paper Avg.
Two-Room100.0 ± 0.0100.0 ± 0.099.3 ± 1.298.7 ± 1.299.5 ± 0.599.5
Push-T93.8 ± 1.783.6 ± 1.060.4 ± 1.739.1 ± 3.769.2 ± 1.269.5
Reacher51.3 ± 8.394.0 ± 2.096.0 ± 3.594.7 ± 2.384.0 ± 2.281.5
OGCube94.7 ± 4.295.3 ± 1.295.3 ± 1.296.0 ± 0.095.3 ± 1.096.5

As shown in the table, for each of the four tasks, HIJP’s success rate averaged across the four difficulty levels is within 2.5 percentage points of the value reported in the paper. The sample standard deviations of these averages across training runs range from 0.5 to 2.2 percentage points. At the individual difficulty levels, Reacher Easy shows the greatest variability, with a mean success rate of 51.3% and a sample standard deviation of 8.3 percentage points. For the other three Reacher difficulty levels, mean success rates range from 94.0% to 96.0%, with sample standard deviations of 2.0–3.5 percentage points.

For the component ablations, we repeated two comparisons on Push-T under the original evaluation protocol. For each training run, we used three evaluation seeds and evaluated 50 episodes per seed at each difficulty level. We first aggregated the evaluation results within each training run, then reported the mean ± sample standard deviation across runs. We conducted three runs each for the full HIJP model and the variant without LcycL_{{cyc}}, and two runs for the Endpoint goal variant.

MethodEasyDefaultHardExtremeAvg.
HIJP (full model)94.0 ± 1.884.2 ± 1.464.4 ± 5.442.7 ± 3.371.3 ± 2.5
w/o LcycL_{cyc}89.8 ± 2.177.1 ± 2.057.8 ± 2.730.7 ± 2.963.8 ± 0.4
Endpoint goal85.3 ± 2.871.7 ± 2.451.7 ± 2.439.0 ± 3.361.9 ± 0.1

To check whether the comparison with INTACT[3] holds across training runs, we also trained HIJP and INTACT on Push-T with three training seeds under the trajectory-level holdout protocol described in A3. In each run, both methods were trained on the same 90% of trajectories and evaluated on the same start–goal pairs drawn from the held-out 10% (three evaluation seeds × 50 pairs per difficulty level).

MethodEasyDefaultHardExtremeAvg.
HIJP93.1 ± 0.883.6 ± 0.457.8 ± 2.539.1 ± 3.968.4 ± 1.7
INTACT96.9 ± 2.080.9 ± 7.849.3 ± 3.519.1 ± 6.961.6 ± 5.0

Across the three training runs, HIJP’s average success rate across the four difficulty levels was 68.4%, 6.8 percentage points higher than INTACT’s 61.6%. The sample standard deviations of these averages across runs were 1.7 and 5.0 percentage points, respectively. HIJP’s gains were concentrated at the Hard and Extreme levels, with the largest gap of 20.0 percentage points at Extreme. INTACT had a higher mean success rate at Easy. This pattern is consistent with the results in Table 1 of the original paper.

AuthorsOpenReview ↗

Response to Reviewer aLm4 (Part 2)

Q2: How much of the gain in Table 3 comes from goal diversity, given that the hindsight-goal ablation changes both goal diversity and the distribution of temporal offsets?

A2:Thank you for pointing this out. We agree that, under Eq. (4), sampling the anchor uniformly and then sampling the goal from the later frames of the window places more probability on shorter gaps. In the Endpoint goal variant in Table 3, anchors are also sampled uniformly, but the goal is fixed to the last frame of the window, so all gaps within the window are sampled equally often. The two variants therefore differ in their gap distributions, so the comparison between the full model and the Endpoint goal variant in Table 3 reflects the effect of the sampling recipe as a whole.

Under the same evaluation settings, the sampling strategy in Eq. (4) increased the mean success rate across the three tasks in Table 3 from 71.7% to 82.5%. The multi-seed results on Push-T show the same trend: the full model reached 71.3 ± 2.5%, compared with 61.9 ± 0.1% for the Endpoint goal variant. We will revise §4.4 to describe the temporal-offset distribution induced by Eq. (4) and attribute the observed gains to the sampling strategy as a whole.

Q3: Do the trajectories used to construct evaluation episodes overlap with the training trajectories?

A3: Thank you for pointing this out. To compare all methods in the main experiments under the same evaluation protocol, we followed LeWM’s[1] approach to constructing evaluation tasks from offline trajectories. We sampled a start state from a trajectory and used a state several steps later in the same trajectory as the goal. The window-level split described in Appendix A.1 of the original manuscript was used for training and validation. In the original evaluation reported in Table 1, the training and evaluation trajectories did overlap.

To assess performance on unseen trajectories, we ran an additional experiment with a 90/10 split at the level of complete trajectories. We retrained both methods on the new training split, following INTACT’s released training and inference configurations for the baseline, and evaluated them on the same start–goal pairs drawn from the held-out trajectories. We fixed the training seed at 3072 and used three evaluation seeds.

TaskMethodEasyDefaultHardExtremeAvg.
TwoRoomHIJP100.0 ± 0.0100.0 ± 0.0100.0 ± 0.098.7 ± 1.299.7 ± 0.3
TwoRoomINTACT68.7 ± 4.249.3 ± 7.643.3 ± 6.130.0 ± 2.047.8 ± 3.2
ReacherHIJP50.7 ± 6.488.0 ± 6.092.7 ± 3.194.0 ± 2.081.3 ± 2.9
ReacherINTACT52.7 ± 8.192.0 ± 8.098.0 ± 2.0100.0 ± 0.085.7 ± 3.6
Push-THIJP92.7 ± 4.683.3 ± 6.156.0 ± 3.540.7 ± 1.268.2 ± 0.3
Push-TINTACT94.7 ± 4.272.0 ± 5.345.3 ± 9.011.3 ± 5.055.8 ± 4.2
OGCubeHIJP96.7 ± 1.294.7 ± 2.393.3 ± 4.296.7 ± 3.195.3 ± 2.4
OGCubeINTACT100.0 ± 0.0100.0 ± 0.065.3 ± 3.178.7 ± 6.486.0 ± 2.0

Under trajectory-level holdout evaluation, HIJP achieved a higher average success rate across the four difficulty levels on three of the four tasks. HIJP led INTACT by 51.9 percentage points on TwoRoom, 12.4 on Push-T, and 9.3 on OGCube. On Reacher, INTACT led by 4.4 percentage points. In the revised manuscript, we will clarify the trajectory overlap in the original evaluation and report the setup and results of the trajectory-level holdout experiment separately.

Q4: Can the discussion distinguish more clearly between the one-step alignment directly supervised by the consistency loss and the behavior of recursive rollouts?

A4: Thank you for the suggestion. We will clarify the distinction between one-step supervision and recursive planning after Eq. (9) in §3.5. During training, Eq. (7) starts from the observed anchor state and aligns the one-step latent state induced by the predicted action with the observed successor state. During planning, Eq. (9) uses the predicted latent states to generate subsequent actions, so prediction errors may accumulate over the five-step rollout. At each replanning step, the model starts from the latest environment observation.

AuthorsOpenReview ↗

Response to Reviewer aLm4 (Part 3)

Q5: How should the evaluation tier names be interpreted, given that the tiers change both goal offset and execution budget and their ordering need not correspond to monotonically increasing difficulty, particularly on Reacher?

A5: Thank you for pointing this out. We agree. The four settings change both the goal offset and the execution budget (10/25, 25/50, 50/100, and 75/100 environment steps), so they are not ordered by difficulty. Reacher shows this most clearly: success is lowest at the shortest offset (51.3% across training runs, compared with 94.0–96.0% in the other settings). In that setting, the 25-step budget equals one replanning interval, so each episode runs open-loop with no chance to correct errors, and Reacher appears more sensitive to this than the other tasks.

In the revised manuscript, we will label the evaluation settings by goal offset (K@10, K@25, K@50, and K@75) and list the corresponding execution budgets in Appendix A.2.

[1]Maes, Lucas, et al. "Leworldmodel: Stable end-to-end joint-embedding predictive architecture from pixels." arXiv preprint arXiv:2603.19312 (2026).

[3]Sun, Junhan, Hao Zhao, and Guofeng Zhang. "INTACT: Isomorphic intent-to-action learning for search-free world models." arXiv preprint arXiv:2607.26056 (2026).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Experiments and Clarifications

Thank you for your time and constructive feedback throughout the review process. As the discussion period concludes, we briefly summarize the additional experiments and clarifications provided in our responses:

Trajectory-level holdout: (Reviewers aLm4, shJz, oBJe) Under a strict 90/10 trajectory-level split, retrained HIJP achieves an 86.1% average success rate across all four tasks, outperforming INTACT (68.8%) by 17.3 percentage points.

Multi-seed stability: (Reviewers aLm4, oBJe) Across three training seeds, HIJP task averages remain within 2.5 points of the original values (SD 0.5–2.2%), with consistent multi-seed ablation gains on Push-T (71.3% full vs. 63.8% w/o consistency and 61.9% endpoint goal). On held-out Push-T Extreme, HIJP outperformed INTACT for all three training seeds (exact McNemar test, p≤0.008p \le 0.008 for each seed).

Additional baselines: (Reviewer shJz) End-to-end HIJP outperforms a frozen-LeWM two-stage variant (+9.5% on Push-T, +8.7% on OGCube) and surpasses GC-IDM on OGCube (+13.7%) and Reacher (+12.0%) while using fewer parameters.

Transition fidelity: (Reviewer shJz) Executing generated actions in the simulator shows that the consistency loss reduces five-step (25 environment steps) latent NMSE by 46% on Push-T, 17% on Reacher, and 48% on OGCube.

Planning efficiency: (Reviewer oBJe) Skipping diagnostic logging rollouts reduces HIJP latency to 20.8 ms per call at batch size 1 and 35.8 ms at batch size 50 (versus 25.4 ms and 46.3 ms for INTACT Direct), outperforming LeWM + CEM in both speed and success across all budgets (60–2,250).

Protocol clarifications: (Reviewer aLm4) We clarified the temporal-offset distribution of Eq. (4), distinguished one-step supervision from recursive rollouts, and relabeled evaluation tiers by goal offset (K@10K@10–K@75K@75) with explicit execution budgets.

We incorporate all new experimental results, baseline comparisons, and protocol clarifications into the revised manuscript. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.