acceptodds
Under review as a conference paper at ICLR 2027

Latent-Predictive Universal Horizon Models for Long Horizon Zero-Shot RL

Abstract

While behavior foundation models (BFMs) have achieved impressive zero-shot performance on various human locomotion tasks, these methods struggle on tasks that require longer horizon reasoning. Recent work has explored using geometric horizon models (GHMs) for planning on top of pre-trained BFMs to achieve longer horizon tasks; however, such methods require significant test-time compute, operate in raw observation space, and require a complex two-stage training process. In this work, we instead explore training horizon models in latent space and as an auxiliary loss during BFM training. In particular, we propose a novel latent-predictive Universal Horizon Model, which we learn jointly with a latent-predictive BFM with both models shaping the state- and task-encoding spaces. We demonstrate that this auxiliary representation learning objective is theoretically compatible with the primary TD-JEPA objective, with respect to the learned state and task encodings, and we demonstrate empirically that it can be beneficial for downstream zero-shot performance. Finally, we show that this UHM model can be used to perform model-based updates in the zero-shot RL setting, allowing us to stably train a latent-predictive BFM at discounts as high as .

Primary area:
reinforcement learning
Keywords:
Offline reinforcement learning, zero-shot reinforcement learning, successor measure, goal-conditioned reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2zeFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the difficulty of learning long-horizon behavior in BFMs, particularly at large discount factors of horizon prediction parameter γ\gamma. Whereas prior approaches train a separate generative horizon model on top of pretrained BFMs, the authors propose to jointly train a BFM with a policy-conditioned, latent-predictive modification of Universal Horizon Model (UHM). The UHM provides an auxiliary loss that shapes shared representations and its predictions support model-based updates during BFM training. The paper provides theoretical analysis under simplifying assumptions and empirical evidence that this approach improves long-horizon zero-shot performance.

Critical strengths
  1. Empirical results are very strong. At large horizon length γ=0.999\gamma=0.999, model-based TD-JEPA achieves much stronger results compared to baselines. This supports paper central motivation.

  2. The method is intuitive extension and combination of BFMs and UHM, making it easier in practice to develop models capable of long-horizon zero-shot policy extraction.

  3. The analysis (Theorem 1) provides conditions under which UHM and successor-feature objectives admit compatible representation subspaces (even though under simplistic assumptions). There is also a characterization of the fixed point of latent bootstrapping as repeated projected transitions.

Critical weaknesses
  1. The evaluation covers only a narrow class of environments (AntMaze navigation). There is no validation on manipulation, pixel-based, or substantially stochastic environments. Consequently, the claimed advantages of operating in latent space, particularly scalability to complex observations, remain largely untested.

  2. The theoretical claims of the latent fixed points are made under simplifying assumptions. Formally, it is not clear what is the bound of error introduced when an approximate UHM prediction is passed through the nonlinear policy and successor-feature predictor. Since, in general, F(E[h]≠E[F(h)])F(\mathbb{E}[h] \neq \mathbb{E}[F(h)]). I think this is crucial because latent predictions could average over multiple possible futures. The empirical tests made on deterministic environments makes the claims correct, but it is not clear whether they hold and remain robust in other settings.

  3. The number of baselines reported is very limited (only TD-JEPA). I understand that this is the most performing recent baseline, but anyway, reporting results from vanilla FB [1], HILP [2], One-step FB [3] and other zero-shot RL approach would benefit empirical claims.

[1] Ahmed Touati and Yann Ollivier: "Learning one representation to optimize all rewards."

[2] S. Park et al: "Foundation policies with hilbert representations."

[3] C. Zheng et al.: "Can We Really Learn One Representation to Optimize All Rewards?"

Questions for authors

Questions:

  1. Could authors perform additional empirical evaluations on larger domain of environments? Together with computing other baselines?

  2. The fixed-point analysis uses exact transition expectations, whereas the implementation bootstraps through predicted latents and nonlinear policy and successor-feature networks. Can you quantify prediction error across horizons and explain how it affects target bias or training stability?

  3. The one-step auxiliary loss performs similarly to the UHM auxiliary loss in Table 2. Would model-based updates using repeated application of a learned one-step latent dynamics model provide similar benefits?

  4. When several future states are possible, a squared-error latent predictor can return their conditional mean. Why should this mean be a suitable input to the policy and successor-feature predictor? Under what conditions would you expect this approximation to fail?

Minor issues:

  1. Caption under Figure 1 specifies "blue dot", however, I see green dot on maze images.
6 comments

We thank the reviewer for their thoughtful review of our paper and insightful comments. We are glad that the reviewer found the method intuitive and the experimental results strong. We reiterate the core contributions of our paper are as follows: (1) introducing a latent-predictive UHM which jointly trains and shapes representations alongside latent-predictive BFM training (2) theoretically motivating this method through an analysis of the shared low-rank decomposition of each method and (3) introducing ν\nu-Bellman updates for TD-JEPA that bootstrap from latent-predictive UHM predictions, which we demonstrate enables training TD-JEPA at high discount factors. We address each of the concerns below:

W1/Q1a: The evaluation covers only a narrow class of environments (AntMaze navigation). There is no validation on manipulation, pixel-based, or substantially stochastic environments. Could authors perform additional empirical evaluations on larger domain of environments?

We agree that extending the method to manipulation and pixel-based environments is of particular interest. There is, however, a critical difference between the navigation environments and the pixel-based and manipulation environments for TD-JEPA which makes this non-trivial. Behavior regularization in the pixel-based and manipulation environments requires an actor conditioned on the raw state space (rather than the latent ϕ\phi), in order for the behavior cloning regularization to be effective [5]. Since our method predicts future states in latent space only, and requires an action at these imagined latent states in order to compute the bootstrap, the method must be adapted to this scenario. For this reason, we felt that these environments represent a material extension of the current method and would be most appropriate as future work.

However, we do believe that our method can be extended to environments that require a state-conditioned actor. As a demonstration, we train an extension of our method on cube-single (horizon ~200) at γ = 0.99. MB-TD-JEPA, with a raw-state actor trains stably and achieves average of ~56% success vs the baseline of 53% (over two seeds, best of sweep). Because this task is short-horizon, we don’t expect model-based targets to improve performance here, however, the experiment shows that our approach can extend to raw-state actors.

We use the following method to extend our method to this scenario: We train an action-free network Vϕ(ϕ(s),z)∈RdψV_\phi(\phi(s),z)\in\mathbb{R}^{d_\psi} by expectile regression, as in IQL [1] with expectile weights computed from the projection onto z [2] :

LV(Vϕ,ϕ,Tϕ)=E(s,a)∼D,z∼Z[∣τ−1{u⊤z<0}∣ ∥u∥2],u=Tϕ−(ϕ(s),a,z)−Vϕ(ϕ(s),z).\mathcal{L} _ V(V_\phi, \phi, T_\phi) = \mathbb{E} _ {(s,a)\sim\mathcal{D},z\sim\mathcal{Z}} \left[\left|\tau - \mathbf{1}\lbrace u^\top z < 0\rbrace\right| \thinspace \Vert u \Vert^2\right], \quad u = T _ \phi^-(\phi(s),a,z) - V _ \phi(\phi(s),z).

which avoids querying the actor entirely. Alternatively, we can simply train an auxiliary action-free network Vϕ(ϕ(s),z)∈RdψV_\phi(\phi(s),z)\in\mathbb{R}^{d_\psi} with the loss:

LV(Vϕ,ϕ,Tϕ)=Es∼D,z∼Z,a∼πz(⋅∣s)[∣∣V(ϕ(s),z)−T(s,a,z)∣∣2]\mathcal{L}_ V(V_\phi, \phi, T_\phi) =\mathbb{E}_{s \sim D, z\sim\mathcal{Z}, a\sim\pi_z(\cdot|s)}\left[||V(\phi(s),z) - T(s,a,z)||^2\right]

where we train the action-free successor features predictor from the action-conditioned successor features predictor over true dataset states, at which we are able to query the raw-state policy. Using either method, we can then bootstrap off the imagined state during MB-TD-JEPA updates using this auxiliary network, without needing to compute the action at the imagined state.

We leave a comprehensive evaluation of this extension on long-horizon manipulation tasks to future work.

Regarding substantially stochastic environments, one of the motivations for our method is simplifying jumpy predictions in approximately deterministic environments, where full generative modeling may be unnecessarily expensive, and our method is primarily intended for environments that are approximately deterministic, such as the Mujoco environments tested here. This is a large class of important environments and is generally true for robotics environments for which this method is primarily intended. However, our method could potentially be extended to incorporate stochasticity in the UHM such as through noise conditioning and an energy-based loss as in [3]

W2/Q4: The theoretical claims of the latent fixed points are made under simplifying assumptions...latent predictions could average over multiple possible futures. When several future states are possible, a squared-error latent predictor can return their conditional mean. Why should this mean be a suitable input to the policy and successor-feature predictor?

We appreciate the reviewers' concern. Our method assumes approximately deterministic dynamics and policies - though stated in the text of the paper we appreciate that this is not made sufficiently clear throughout and have updated the text in several places to make this more apparent to the reader. Under deterministic dynamics and policy, the average is exact, as the reviewer notes, however, we agree that the point prediction assumption does not hold under high stochasticity and this method would not be appropriate in such settings unless extended to incorporate stochasticity in the UHM such as through noise conditioning and an energy-based loss as in [3].

Q2: The fixed-point analysis uses exact transition expectations, whereas the implementation bootstraps through predicted latents and nonlinear policy and successor-feature networks. Can you quantify prediction error across horizons and explain how it affects target bias or training stability?

We have added a section in the Appendix (C.2) to report the accuracy of our learned UHM predictions. Specifically, we provide plots of the average cosine similarity between the UHM predictions and the average of encoded states reached in policy rollouts from the same starting state and task conditioning. We compare against the cosine similarity between a single policy rollout and the same average. Although the environment is deterministic, we add a small amount of noise to the actor’s actions during rollouts, so the rollouts are somewhat stochastic. However, despite this we can see that the cosine similarity between the UHM predictions and the rollout average is similar to that of a single true policy rollout and improves over a single rollout at longer horizons in 6 out of 7 environments (Figure 6).

W3/Q1b: The number of baselines reported is very limited (only TD-JEPA)

We address the reviewer’s concern regarding limited baselines, particularly in the 0.999 discount setting by running FB and HILP at γ=0.999\gamma=0.999 under (1) matched settings (actor conditioned on encoding, normalized SM, deterministic policy, slower target updates) and (2) original settings. For each, we sweep across 8 hyperparameter configurations on 4 seeds. For the normalized setting, we scale the orthonormality coefficient down from their original sweep by 100, as we do in TD-JEPA (matched). We add FB and HILP columns to Table 1 in the paper, reporting the best across all 16 configs. We replicate these columns here alongside our method. We note that these results are currently for 4 seeds and will be extended to 10 seeds for the final revision and that a few configurations in the matched setting and in antmaze-colossal are still running due to cluster failures; the table will be updated once they complete. We will also replicate results in the γ=0.995\gamma=0.995 setting for completeness in the final revision. Both FB and HILP significantly underperform our method at high discount, across all tested configurations.

EnvironmentMB-TD-JEPAFBHILP
antmaze-medium-stitch41.33 ± 3.342.83 ± 0.323.50 ± 0.17
antmaze-medium-explore22.27 ± 2.940.50 ± 0.320.00 ± 0.00
antmaze-medium-navigate91.47 ± 0.3854.17 ± 6.0616.67 ± 1.28
antmaze-large-stitch9.20 ± 1.750.50 ± 0.320.67 ± 0.27
antmaze-large-navigate67.47 ± 4.1121.33 ± 5.368.00 ± 0.72
antmaze-giant-navigate39.67 ± 1.770.33 ± 0.190.50 ± 0.17
Mean45.2413.284.89

Q3: The one-step auxiliary loss performs similarly to the UHM auxiliary loss in Table 2. Would model-based updates using repeated application of a learned one-step latent dynamics model provide similar benefits?

We perform an ablation of different model-based update strategies in the high discount setting (1) MB-TD-JEPA (ours) (2) Monte Carlo samples (MC) which samples states from the dataset trajectories (up to kmaxk_\text{max} or trajectory end), equivalent to TD(λ\lambda) and (3) iterative one-step rollouts (1-Step) with policy queries at each imagined state, up to kmaxk_\text{max}. We provide the results from these experiments in Table 3 in the updated paper. We find that the proposed combination of a world-model auxiliary loss and model-based updates is necessary for achieving the best performance at long horizons, however, the method for performing model-based updates is flexible in these locomotion environments, with MC and 1-Step performing slightly better than the UHM overall. In the case of MC, however, we note that this is highly dependent on data quality, with the stitch and navigate datasets - which contain meaningful behavior in the dataset trajectories - displaying strong performance, while in the explore dataset, MC updates provide no benefit and model-based rollouts are necessary. While one-step rollouts are performant in navigation environments, consistent with prior work[4], our method provides an advantage over one-step rollouts by parallelizing compute, while one-step rollouts must be sampled iteratively. Further, unlike in the navigation environments which can use a policy conditioned on ϕ\phi, raw-state conditioning is necessary in manipulation and pixel environments. As discussed in our response to W1, our method can be extended to handle such cases, while one-step rollouts would require training a distilled policy on ϕ\phi.

Caption under Figure 1 specifies "blue dot", however, I see green dot on maze images.

Thank you for noting this, we have updated the coloring and text.

[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit Q-learning." International Conference on Learning Representations (2022).

[2] Ghosh, Dibya, Chethan Anand Bhateja, and Sergey Levine. "Reinforcement learning from passive data via latent intentions." International Conference on Machine Learning. PMLR, 2023.

[3] Shen, Xinwei, and Nicolai Meinshausen. "Engression: extrapolation through the lens of distributional regression." Journal of the Royal Statistical Society Series B: Statistical Methodology 87.3 (2025): 653-677.

[4]Chung, Hojun, Junseo Lee, and Songhwai Oh. "Offline reinforcement learning with universal horizon models." arXiv preprint arXiv:2605.15603 (2026).

[5]Bagatella, Marco, et al. "Td-jepa: Latent-predictive representations for zero-shot reinforcement learning." International Conference on Learning Representations. Vol. 2026. 2026.

Reviewer_twf1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that standard geometric horizon model (GHM) learning methods like TD-Flows and Jumpy World Models are slow because of heavy test-time compute, a two-stage pipeline of learning base policies first and then learning on a GHM on top of them, and the use of raw observation space. It proposes a concurrent training for the policies and GHM using TD-JEPA base policies with Universal Horizon Models (UHM), which are basically a generalization of GHMs, as an auxiliary loss - referred to as MB-TD-JEPA. MB-TD-JEPA achieves stable training over longer horizons and outperforms TD-JEPA on the navigation tasks of OGBench.

Critical strengths
  1. The idea of concurrent training and UHM affecting the representation of BFM policies is neat.
  2. The paper involves detailed proofs and shows connections that the horizon-consistency objective of JWM (Farebrother et al., 2026) is a special case of the proposed ν\nu-Bellman equation for successor measure.
  3. Figure 3 provides a nice description of the UHM training procedure.
  4. The paper also clearly points out its limitations and describes the future work adequately.
Critical weaknesses
  1. The theoretical proofs rely on the assumption of unit covariance, but will the MB-TD-JEPA loss actually reach the unit covariance? The method only encourages identity covariance with a soft orthonormality regularizer with small coefficients. Also, theorem 1 covers the Monte Carlo case, but the method itself uses bootstrapped losses.
  2. The paper is motivated for long-horizon tasks, but the performance of MB-TD-JEPA drops from moving γ=0.995\gamma=0.995 to γ=0.999\gamma=0.999. So, the central claim is unsupported by this experiment. The current set of experiments only shows that training will be stable at higher γ\gamma's.
  3. Baseline reported is not directly comparable. TD-JEPA (original) reported numbers are with γ=0.99\gamma=0.99 while the comparison in Table 2 is with γ=0.995\gamma=0.995. Also, performance on the stitch dataset drops a lot, and the effect is unclear. Is it due to using TD3+BC instead of FQL, the symmetric variant, or because of something else? Although mentioned in the future work section, the paper's claims would be much stronger if it included the pixel-based variants as well. The current set of comparisons is a bit restricted to antmaze settings only. Also, there are no comparisons with the GHM/TD-Flow planning methods, other BFMs (FB, Laplacian USFA, HILP), or Azad et al. (2026). The paper's bolding rule is also lenient.
  4. The one-step auxiliary loss almost matches the UHM auxiliary loss (Table 2), but this weakens the claim that UHM is helping in representation learning.
Questions for authors
  1. Why does the TD-JEPA (and its +UHM variant) loss explode with longer training? What are the possible reasons for this divergence?
  2. What does "multiple samples from the UHM" mean for a deterministic predictor?
  3. Experiment section mentions that it learns normalized measure (1−γ)M(1-\gamma)M, but Algorithm 1 and Eq. 17 use the unnormalized target, so which one is actually used?
  4. Why does the method assume that the top-dd eigenvalues are "distinct"?
  5. What is the final empirical feature covariance that MB-TD-JEPA achieves?

Minor Suggestions:

  1. Some citations are missing in the paper:

    a. TD Models: Modeling the World at a Mixture of Time Scales. Sutton (1995), which originally discusses the notion of a mixture of time scales.

    b. Universal Successor Features Approximators. Borsa et al. (2019), which laid the foundations for both FB and TD-JEPA methods.

    c. Laplacian Keyboard: Beyond the Linear Span. Chandrasekar and Machado (2026) discuss the USFA-based architectures for zero-shot RL.

  2. Some references are not formatted properly:

    a. Jesse Farebrother, Matteo Pirotta, Andrea Tirinzoni, Remi Munos, Alessandro Lazaric, and Ahmed Touati. Temporal difference flows. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 16102–16143. PMLR, 13–19 Jul 2025.

    b. Shantanu Thakoor, Mark Rowland, Diana Borsa, Will Dabney, Remi Munos, and Andre Barreto. Generalized policy improvement with geometric policy composition. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162.

  3. Eqs. 10 and 16 sample a′∼πz(⋅∣s)a' \sim \pi_z(\cdot|s); it should be s′s'.

  4. Lines 82-85 are a bit repetitive.

  5. There are some typos in the paper: "pluggin", "minimzers", "the the".

8 comments

We thank the reviewer for the thoughtful review and helpful suggestions. We are glad that the reviewer finds the main contribution of our paper of jointly training a UHM and BFM with shared representations, to be interesting. We reiterate the core contributions of our paper are as follows: (1) introducing a latent-predictive UHM which jointly trains and shapes representations alongside latent-predictive BFM training (2) theoretically motivating this method through an analysis of the shared low-rank decomposition of each method and (3) introducing ν\nu-Bellman updates for TD-JEPA that bootstrap from latent-predictive UHM predictions, which we demonstrate enables training TD-JEPA at high discount factors. We address each of the concerns in turn below:

W1/Q5: The theoretical proofs rely on the assumption of unit covariance, but will the MB-TD-JEPA loss actually reach the unit covariance? What is the final empirical feature covariance that MB-TD-JEPA achieves?

The assumption of unit covariance is standard in prior theoretical work on BYOL objectives[1,2,3]. The same assumption and orthonormality regularizer are used in standard TD-JEPA. To confirm that the orthonormality regularizer is achieving orthonormal features in ϕ\phi by the end of training, we report the normalized Frobenius distance between the feature second-moment matrix and the identity, ∣∣Es∼D[ϕ(s)ϕ(s)⊤]−I∣∣F/d||\mathbb{E}_{s\sim\mathcal{D}}[\phi(s)\phi(s)^\top] - I||_F/\sqrt{d}, estimated from the diagonal and off-diagonal terms of the orthonormality loss on training batches. All values being close to 0 indicate that there is high similarity to the identity matrix and thus orthonormality is approximately maintained.

EnvironmentUHM + MB Updates
antmaze-medium-stitch0.075 ± 0.002
antmaze-medium-explore0.044 ± 0.002
antmaze-medium-navigate0.044 ± 0.001
antmaze-large-stitch0.068 ± 0.002
antmaze-large-navigate0.041 ± 0.001
antmaze-giant-navigate0.042 ± 0.001
antmaze-colossal-navigate0.036 ± 0.002

W2: The paper is motivated for long-horizon tasks, but the performance of MB-TD-JEPA drops...the current set of experiments only shows that training will be stable at higher γ\gamma's.

We have performed additional experiments on a longer horizon environment, the antmaze-colossal environment, proposed in [4], which directly addresses this concern. This environment requires as many as 4000 environment steps to complete, compared to the longest maximum horizon in our existing results of 1000 environment steps. We have included these results as an additional row in Table 1, and also reproduced this row below (we note that these results are over 4 seeds currently but will be extended to full 10 seeds for the final revision of the paper). Here we can see that the higher discount value of 0.999 is strictly necessary. Without model-based updates, at either γ=0.995\gamma=0.995 or γ=0.999\gamma=0.999, the best performance reaches only 17% success. Combining high γ\gamma and model-based updates, we are able to achieve success of ~45%. This clearly demonstrates the utility of training at higher γ\gamma in long horizon environments and the necessity of the model-based updates in this setting.

EnvironmentTD-JEPA (Matched)+ UHMMB-TD-JEPA
0.995antmaze-colossal-navigate15.50 ± 2.0417.33 ± 2.496.50 ± 2.50
0.999antmaze-colossal-navigate10.50 ± 1.6215.17 ± 2.4645.33 ± 0.67

W3a: Baseline reported is not directly comparable. Also, performance on the stitch dataset drops a lot, and the effect is unclear

We agree that TD-JEPA (original) is not directly comparable. For this reason, in Table 2 alongside original, we supply results for TD-JEPA (matched), which exactly matches the settings for our experiments (γ=0.995,0.999\gamma=0.995, 0.999, full encoder as actor input, symmetric variant and deterministic actor). We also gray out the TD-JEPA original column. While the stitch dataset performance drops under the matched settings, we note that navigation performance increases, and overall, across the antmaze datasets, the performance is roughly equivalent.

W3b: Although mentioned in the future work section...the current set of comparisons is a bit restricted to antmaze settings only.

We agree that extending the method to manipulation and pixel-based environments is of particular interest. There is, however, a critical difference between the navigation environments and the pixel-based and manipulation environments for TD-JEPA which makes this non-trivial. Behavior regularization in the pixel-based and manipulation environments requires an actor conditioned on the raw state space (rather than the latent ϕ\phi), in order for the behavior cloning regularization to be effective [5]. Since our method predicts future states in latent space only, and requires an action at these imagined latent states in order to compute the bootstrap, the method must be adapted to this scenario. For this reason, we felt that these environments represent a material extension of the current method and would be most appropriate as future work.

However, we do believe that our method can be extended to environments that require a state-conditioned actor. As a demonstration, we train an extension of our method on cube-single (horizon ~200) at γ = 0.99. MB-TD-JEPA, with a raw-state actor trains stably and achieves average of ~56% success vs the baseline of 53% (over two seeds, best of sweep). Because this task is short-horizon, we don’t expect model-based targets to improve performance here, however, the experiment shows that our approach can extend to raw-state actors.

We use the following method to extend our method to this scenario: We train an action-free network Vϕ(ϕ(s),z)∈RdψV_\phi(\phi(s),z)\in\mathbb{R}^{d_\psi} by expectile regression, as in IQL [6] with expectile weights computed from the projection onto z [7] :

LV(Vϕ,ϕ,Tϕ)=E(s,a)∼D,z∼Z[∣τ−1{u⊤z<0}∣ ∥u∥2],u=Tϕ−(ϕ(s),a,z)−Vϕ(ϕ(s),z).\mathcal{L} _ V(V_\phi, \phi, T_\phi) = \mathbb{E} _ {(s,a)\sim\mathcal{D},z\sim\mathcal{Z}} \left[\left|\tau - \mathbf{1}\lbrace u^\top z < 0\rbrace\right| \thinspace \Vert u \Vert^2\right], \quad u = T _ \phi^-(\phi(s),a,z) - V _ \phi(\phi(s),z).

which avoids query the actor entirely. Alternatively, we can simply train an auxiliary action-free network Vϕ(ϕ(s),z)∈RdψV_\phi(\phi(s),z)\in\mathbb{R}^{d_\psi} with the loss:

LV(Vϕ,ϕ,Tϕ)=Es∼D,z∼Z,a∼πz(⋅∣s)[∣∣V(ϕ(s),z)−T(s,a,z)∣∣2]\mathcal{L}_ V(V_\phi, \phi, T_\phi) =\mathbb{E}_{s \sim D, z\sim\mathcal{Z}, a\sim\pi_z(\cdot|s)}\left[||V(\phi(s),z) - T(s,a,z)||^2\right]

where we train the action-free successor features predictor from the action-conditioned successor features predictor over true dataset states, at which we are able to query the raw-state policy. Using either method, we can then bootstrap off the imagined state during MB-TD-JEPA updates using this auxiliary network, without needing to compute the action at the imagined state.

We leave a comprehensive evaluation of this extension on long-horizon manipulation tasks to future work.

W3c: Also, there are no comparisons with the GHM/TD-Flow planning methods, other BFMs

We address the reviewer’s concern regarding limited baselines, particularly in the 0.999 discount setting by running FB and HILP at γ=0.999\gamma=0.999 under (1) matched settings (actor conditioned on encoding, normalized SM, deterministic policy, slower target updates) and (2) original settings. For each, we sweep across 8 hyperparameter configurations on 4 seeds. For the normalized setting, we scale the orthonormality coefficient down from their original sweep by 100, as we do in TDJEPA (matched). We add FB and HILP columns to Table 1 in the paper, reporting the best across all 16 configs. We replicate these columns here alongside our method. We note that these results are currently for 4 seeds and will be extended to 10 seeds for the final revision and that a few configurations in the matched setting and antmaze-colossal are still running due to cluster failures; the table will be updated once they complete. We will also replicate results in the γ=0.995\gamma=0.995 setting for completeness in the final revision. Both FB and HILP significantly underperform our method at high discount, across all tested configurations.

EnvironmentMB-TD-JEPAFBHILP
antmaze-medium-stitch41.33 ± 3.342.83 ± 0.323.50 ± 0.17
antmaze-medium-explore22.27 ± 2.940.50 ± 0.320.00 ± 0.00
antmaze-medium-navigate91.47 ± 0.3854.17 ± 6.0616.67 ± 1.28
antmaze-large-stitch9.20 ± 1.750.50 ± 0.320.67 ± 0.27
antmaze-large-navigate67.47 ± 4.1121.33 ± 5.368.00 ± 0.72
antmaze-giant-navigate39.67 ± 1.770.33 ± 0.190.50 ± 0.17
Mean45.2413.284.89

Regarding GHM/TD-Flow methods, these operate in a different setting of test-time planning, while our method is fully amortized and requires minimal test-time computation. GHMs have not been trained alongside BFMs and incorporating GHM training into BFM training is an interesting, but separate, research question.

W4: The one-step auxiliary loss almost matches the UHM auxiliary loss...weakens the claim that UHM is helping

We explicitly acknowledge this and claim only that world model losses in general improve representation learning (“Experimentally, we demonstrate that world model auxiliary losses are modestly beneficial for zero-shot policy learning,“).

Q1: Why does the TD-JEPA loss explode with longer training?

The divergence occurs due to standard instability of TD learning at high discount factors [8]. At high discount factors, the contraction margin is small and so noise during training can outweigh it, leading to divergence. Further, there is a spectral argument for why high discount factors are challenging specifically when learning a low-rank factorization of the successor measure. Increasing the discount factor sharpens the spectra of the successor measure matrix, accentuating differences between large-magnitude singular values, but blurring low magnitude singular values which may be necessary for high frequency behavior [9]. Our auxiliary world model loss aims to combat this effect by also training on lower horizon predictions, as we discuss in Section 5.1.

Q2:What does "multiple samples from the UHM" mean for a deterministic predictor?

The full sampling process from a UHM involves first sampling t∼Geom(1−β)t\sim \text{Geom}(1-\beta) and then querying the deterministic model T(s,a,z,t)T(s,a,z,t) at the sampled tt. Thus, while the predictor TT is deterministic, the full sampling process is not. We have added a note on this in the paper.

Q3: Experiment section mentions that it learns normalized measure , but Algorithm 1 and Eq. 17 use the unnormalized target

For consistency with the zero-shot RL literature and TD-JEPA, we state Alg 1 and Eq 17 in terms of the unnormalized successor measure, but in practice, we estimate the normalized successor measure. The difference amounts to a simple rescaling by (1−γ)(1-\gamma). We believe the clarity of the paper is best served by continuing to report Eq 17 as unnormalized for easier comparison to standard TD-JEPA updates, however we updated Alg 1 to reflect the normalized updates actually used.

Reviewer_QNdDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper jointly trains a policy-conditioned latent horizon model with TD-JEPA and uses its predictions for model-based successor-feature updates. It studies representation compatibility in a restricted linear setting and evaluates the method on six OGBench antmaze datasets. The clearest result is that model-based updates prevent the late-training collapse observed at discount 0.999.

Critical strengths
  • The high-discount stability result is substantial. At 0.999, mean success rises from 17.78% for matched TD-JEPA and 22.73% with the UHM auxiliary loss to 45.34% with model-based updates. Learning curves support the interpretation that these updates prevent late-training collapse, and the main tables use ten seeds.
  • The matched baseline controls for changes to policy inputs, behavior-cloning regularization, and successor-measure normalization. The one-step auxiliary-loss comparison also makes clear that representation gains are not exclusive to horizon modeling.
Critical weaknesses
  • The practical benefit of enabling a larger discount remains less clear than the stability benefit. In Table 1, MB-TD-JEPA at 0.999 averages 45.34%, below its 54.47% at 0.995; even on giant-navigate, 39.67% is below the auxiliary-only model’s 43.33% at 0.995. This does not undermine the stated stability contribution, but limits the evidence that higher-discount training currently yields better long-horizon task performance.
  • The key model-based approximation is not directly validated. Theorem 4 uses true future-state samples, whereas Eq. 17 substitutes latent point predictions and feeds them into a nonlinear policy and successor-feature predictor. In general, applying these functions to a predicted mean is not equivalent to averaging over future states. Appendix B.3 describes ensembling, multiple samples, and target blending, but does not quantify the remaining target discrepancy. The restricted Monte Carlo compatibility result in Theorem 1, whose limitations the authors acknowledge, does not resolve this approximation.
Questions for authors
  1. Is there an evaluated task or goal-distance regime where the 0.999 variant improves over the best 0.995 variant? If not, please clarify the practical scope of the contribution as enabling stable high-discount training rather than demonstrating superior long-horizon task performance.
  2. For matched starting states, policy conditions, and sampled horizons, how do model-based update targets compare with reference expectations estimated from repeated true policy rollouts? Reporting the discrepancy by horizon, under the ensemble and blending settings actually used, would help assess the approximation in Eq. 17.
2 comments

We thank the reviewer for their positive assessment of our paper and are glad that the reviewer appreciated the strong stability results at high discounts. We reiterate the core contributions of our paper are as follows: (1) introducing a latent-predictive UHM which jointly trains and shapes representations alongside latent-predictive BFM training (2) theoretically motivating this method through an analysis of the shared low-rank decomposition of each method and (3) introducing ν\nu-Bellman updates for TD-JEPA that bootstrap from latent-predictive UHM predictions, which we demonstrate enables training TD-JEPA at high discount factors. We address each of the specific concerns raised below.

W1/Q1 The practical benefit of enabling a larger discount remains less clear than the stability benefit. In Table 1, MB-TD-JEPA at 0.999 averages 45.34%, below its 54.47% at 0.995.

Is there an evaluated task or goal-distance regime where the 0.999 variant improves over the best 0.995 variant?.

We have performed additional experiments on a longer horizon environment, the antmaze-colossal environment, proposed in [1], which directly address this concern. This environment requires as many as 4000 environment steps to complete, compared to the longest maximum horizon in our existing results of 1000 environment steps. We have included these results as an additional row in Table 1, and also reproduced this row below (we note that these results are over 4 seeds currently but will be extended to full 10 seeds for the final revision of the paper). Here we can see that the higher discount value of 0.999 is strictly necessary. Without model-based updates, at either γ=0.995\gamma=0.995 or γ=0.999\gamma=0.999, the best performance reaches only 17% success. Combining high γ\gamma and model-based updates, we are able to achieve success of ~45%. This clearly demonstrates the utility of training at higher γ\gamma in long horizon environments and the necessity of the model-based updates in this setting.

EnvironmentTD-JEPA (Matched)+ UHMMB-TD-JEPA
0.995antmaze-colossal-navigate15.50 ± 2.0417.33 ± 2.496.50 ± 2.50
0.999antmaze-colossal-navigate10.50 ± 1.6215.17 ± 2.4645.33 ± 0.67

W2/Q2 The key model-based approximation is not directly validated. Theorem 4 uses true future-state samples, whereas Eq. 17 substitutes latent point predictions and feeds them into a nonlinear policy and successor-feature predictor. In general, applying these functions to a predicted mean is not equivalent to averaging over future states

For matched starting states, policy conditions, and sampled horizons, how do model-based update targets compare with reference expectations estimated from repeated true policy rollouts?

The primary setting we are interested in this work is approximately deterministic dynamics and policies, where generative modeling of the successor measure may be excessively expensive. In this setting, the expectation is exactly a single sample. We have adjusted several places in the text to make this more explicit. We note, however, that this method could be extended to incorporate stochasticity in the UHM such as through noise conditioning and an energy-based loss, as in [2].

We agree with the reviewer’s suggestion that we report the accuracy of our learned UHM model predictions. We have added a section in the Appendix (C.2). In this section, we provide plots of the average cosine similarity between the UHM predictions and average encoded state reached in policy rollouts from the same starting state and task conditioning. We compare against the cosine similarity between a single policy rollout and the same average. Although the environment is deterministic, we add a small amount of noise to the actor’s actions during rollouts, so the rollouts are somewhat stochastic. However, we can see that the cosine similarity between the UHM predictions and the rollout average is similar to that of a single true policy rollout and improves over a single rollout at longer horizons in 6 out of 7 environments (Figure 6).

[1] Zheng, Bill Chunyuan, et al. "Multistep quasimetric learning for scalable goal-conditioned reinforcement learning." arXiv preprint arXiv:2511.07730 (2025).

[2] Shen, Xinwei, and Nicolai Meinshausen. "Engression: extrapolation through the lens of distributional regression." Journal of the Royal Statistical Society Series B: Statistical Methodology 87.3 (2025): 653-677.

Dear Reviewer, We thank you again for your thoughtful feedback which has helped us to significantly improve the quality of our paper. As the discussion period is coming to an end, we would appreciate it if you could let us know if we have adressed your concerns and if you have any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Preliminary pixel results

We would like to thank all the reviewers again for their thoughtful reviews of our paper, which have helped us to significantly improve the paper. We would like to provide all reviewers with the following updates:

Pixel Environments

Many reviewers were interested in image-based results from our method. We have produced some preliminary results on the AntMaze RGB environments that indicate that multi-step methods perform much better than standard TD-JEPA in the high γ\gamma setting (all results presented here are for γ=0.999\gamma=0.999):

EnvironmentTD-JEPA (Matched)MB-TD-JEPAMC (TD(λ\lambda)) +UHM
antmaze-medium-stitch11.33 ± 4.8240.89 ± 6.1910.44 ± 4.26
antmaze-medium-explore0.00 ± 0.000.22 ± 0.220.00 ± 0.00
antmaze-medium-navigate17.78 ± 1.5690.22 ± 3.0989.33 ± 0.38
antmaze-large-stitch0.00 ± 0.000.44 ± 0.221.11 ± 0.59
antmaze-large-navigate6.44 ± 1.5644.67 ± 19.2344.00 ± 19.69
antmaze-giant-navigate11.78 ± 3.8613.56 ± 2.1214.89 ± 3.86
Mean7.8931.6726.63

We note the following caveats: due to time and compute constraints, these results are over 3 seeds; MB-TD-JEPA and MC are swept over only 2 hyperparameter configurations, and the baseline is swept over 4, of which the best-performing ones at ~1–1.5M steps were run to the full 2M steps. Due to the limited seeds and hyperparameter sweep, these results should be read as evidence that our method enables high-discount training in pixel environments, rather than a rigorous comparison across methods. However, for the camera-ready version, we will train additional seeds and perform a more extensive hyperparameter sweep.

To produce these pixel results, we made the following updates to our algorithm:

(1) We use actors conditioned on raw RGB encoder output, and train an auxiliary network Vϕ(ϕ(s),z)∈RdψV_\phi(\phi(s),z)\in\mathbb{R}^{d_\psi} with the loss:

LV(Vϕ,ϕ,Tϕ)=Es∼D,z∼Z,a∼πz(⋅∣s)[V(ϕ(s),z)−T(ϕ(s),a,z)∣∣]\mathcal{L}_ V(V_\phi, \phi, T_\phi) =\mathbb{E}_{s \sim D, z\sim\mathcal{Z}, a\sim\pi_z(\cdot|s)}\left[V(\phi(s),z) - T(\phi(s),a,z)||\right]

This auxiliary network allows MB-TD-JEPA bootstrap updates without querying the actor on imagined states (which are unavailable in raw RGB encoding space)

(2) We noise-condition the UHM to account for stochasticity induced by the partial observability of pixel observations. We use FiLM [2] conditioning on the output of a noise- and tt-conditioned MLP to incorporate the noise into the predictions. We train with an energy loss as in [1], which minimizes the Euclidean distance between model samples and the target and maximizes spread between samples. We use a noise dimension of 16 and 4 model samples to compute the loss.

(3) We increase the ratio of task dropout described in Section B2 to 30% and infer zz from a future state along the same trajectory, rather than the batch mean.

(4) Versus the reference implementation, we add an EMA target for the RGB encoder and reduce predictor target update rate to 0.001

For TD-JEPA (matched) we make all relevant changes to the baseline method as well (in particular, adding RGB encoder EMA and decreasing the target update rate), this includes the changes made in proprioceptive environments (normalization, symmetric), except actor conditioning which is kept as raw RGB encoder output, as discussed above.

For MC, we match all UHM training parameters, but replace model samples in the TD-JEPA updates with future states from the dataset trajectories. We note that this performs well in navigate environments with high-quality long trajectories but degrades in the stitch environment.

Updated FB/HILP Baselines

The completed FB/HILP baseline numbers at 0.999 (4 seeds), including AntMaze Colossal are provided below:

EnvironmentMB-TD-JEPAFBHILP
antmaze-medium-stitch41.33 ± 3.342.83 ± 0.323.50 ± 0.17
antmaze-medium-explore22.27 ± 2.941.67 ± 0.330.33 ± 0.19
antmaze-medium-navigate91.47 ± 0.3854.17 ± 6.0616.67 ± 1.28
antmaze-large-stitch9.20 ± 1.750.67 ± 0.670.67 ± 0.27
antmaze-large-navigate67.47 ± 4.1121.33 ± 5.368.00 ± 0.72
antmaze-giant-navigate39.67 ± 1.770.33 ± 0.190.50 ± 0.17
antmaze-colossal-navigate*45.33 ± 0.671.33 ± 0.384.67 ± 1.31
Mean45.2511.764.90

[1] Shen, Xinwei, and Nicolai Meinshausen. "Engression: extrapolation through the lens of distributional regression." Journal of the Royal Statistical Society Series B: Statistical Methodology 87.3 (2025): 653-677.

[2] Perez, Ethan, et al. "Film: Visual reasoning with a general conditioning layer." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.