- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the difficulty of learning long-horizon behavior in BFMs, particularly at large discount factors of horizon prediction parameter . Whereas prior approaches train a separate generative horizon model on top of pretrained BFMs, the authors propose to jointly train a BFM with a policy-conditioned, latent-predictive modification of Universal Horizon Model (UHM). The UHM provides an auxiliary loss that shapes shared representations and its predictions support model-based updates during BFM training. The paper provides theoretical analysis under simplifying assumptions and empirical evidence that this approach improves long-horizon zero-shot performance.
Critical strengths
-
Empirical results are very strong. At large horizon length , model-based TD-JEPA achieves much stronger results compared to baselines. This supports paper central motivation.
-
The method is intuitive extension and combination of BFMs and UHM, making it easier in practice to develop models capable of long-horizon zero-shot policy extraction.
-
The analysis (Theorem 1) provides conditions under which UHM and successor-feature objectives admit compatible representation subspaces (even though under simplistic assumptions). There is also a characterization of the fixed point of latent bootstrapping as repeated projected transitions.
Critical weaknesses
-
The evaluation covers only a narrow class of environments (AntMaze navigation). There is no validation on manipulation, pixel-based, or substantially stochastic environments. Consequently, the claimed advantages of operating in latent space, particularly scalability to complex observations, remain largely untested.
-
The theoretical claims of the latent fixed points are made under simplifying assumptions. Formally, it is not clear what is the bound of error introduced when an approximate UHM prediction is passed through the nonlinear policy and successor-feature predictor. Since, in general, . I think this is crucial because latent predictions could average over multiple possible futures. The empirical tests made on deterministic environments makes the claims correct, but it is not clear whether they hold and remain robust in other settings.
-
The number of baselines reported is very limited (only TD-JEPA). I understand that this is the most performing recent baseline, but anyway, reporting results from vanilla FB [1], HILP [2], One-step FB [3] and other zero-shot RL approach would benefit empirical claims.
[1] Ahmed Touati and Yann Ollivier: "Learning one representation to optimize all rewards."
[2] S. Park et al: "Foundation policies with hilbert representations."
[3] C. Zheng et al.: "Can We Really Learn One Representation to Optimize All Rewards?"
Questions for authors
Questions:
-
Could authors perform additional empirical evaluations on larger domain of environments? Together with computing other baselines?
-
The fixed-point analysis uses exact transition expectations, whereas the implementation bootstraps through predicted latents and nonlinear policy and successor-feature networks. Can you quantify prediction error across horizons and explain how it affects target bias or training stability?
-
The one-step auxiliary loss performs similarly to the UHM auxiliary loss in Table 2. Would model-based updates using repeated application of a learned one-step latent dynamics model provide similar benefits?
-
When several future states are possible, a squared-error latent predictor can return their conditional mean. Why should this mean be a suitable input to the policy and successor-feature predictor? Under what conditions would you expect this approximation to fail?
Minor issues:
- Caption under Figure 1 specifies "blue dot", however, I see green dot on maze images.
6 comments
W2/Q4: The theoretical claims of the latent fixed points are made under simplifying assumptions...latent predictions could average over multiple possible futures. When several future states are possible, a squared-error latent predictor can return their conditional mean. Why should this mean be a suitable input to the policy and successor-feature predictor?
We appreciate the reviewers' concern. Our method assumes approximately deterministic dynamics and policies - though stated in the text of the paper we appreciate that this is not made sufficiently clear throughout and have updated the text in several places to make this more apparent to the reader. Under deterministic dynamics and policy, the average is exact, as the reviewer notes, however, we agree that the point prediction assumption does not hold under high stochasticity and this method would not be appropriate in such settings unless extended to incorporate stochasticity in the UHM such as through noise conditioning and an energy-based loss as in [3].
Q2: The fixed-point analysis uses exact transition expectations, whereas the implementation bootstraps through predicted latents and nonlinear policy and successor-feature networks. Can you quantify prediction error across horizons and explain how it affects target bias or training stability?
We have added a section in the Appendix (C.2) to report the accuracy of our learned UHM predictions. Specifically, we provide plots of the average cosine similarity between the UHM predictions and the average of encoded states reached in policy rollouts from the same starting state and task conditioning. We compare against the cosine similarity between a single policy rollout and the same average. Although the environment is deterministic, we add a small amount of noise to the actor’s actions during rollouts, so the rollouts are somewhat stochastic. However, despite this we can see that the cosine similarity between the UHM predictions and the rollout average is similar to that of a single true policy rollout and improves over a single rollout at longer horizons in 6 out of 7 environments (Figure 6).
W3/Q1b: The number of baselines reported is very limited (only TD-JEPA)
We address the reviewer’s concern regarding limited baselines, particularly in the 0.999 discount setting by running FB and HILP at under (1) matched settings (actor conditioned on encoding, normalized SM, deterministic policy, slower target updates) and (2) original settings. For each, we sweep across 8 hyperparameter configurations on 4 seeds. For the normalized setting, we scale the orthonormality coefficient down from their original sweep by 100, as we do in TD-JEPA (matched). We add FB and HILP columns to Table 1 in the paper, reporting the best across all 16 configs. We replicate these columns here alongside our method. We note that these results are currently for 4 seeds and will be extended to 10 seeds for the final revision and that a few configurations in the matched setting and in antmaze-colossal are still running due to cluster failures; the table will be updated once they complete. We will also replicate results in the setting for completeness in the final revision. Both FB and HILP significantly underperform our method at high discount, across all tested configurations.
| Environment | MB-TD-JEPA | FB | HILP |
|---|---|---|---|
| antmaze-medium-stitch | 41.33 ± 3.34 | 2.83 ± 0.32 | 3.50 ± 0.17 |
| antmaze-medium-explore | 22.27 ± 2.94 | 0.50 ± 0.32 | 0.00 ± 0.00 |
| antmaze-medium-navigate | 91.47 ± 0.38 | 54.17 ± 6.06 | 16.67 ± 1.28 |
| antmaze-large-stitch | 9.20 ± 1.75 | 0.50 ± 0.32 | 0.67 ± 0.27 |
| antmaze-large-navigate | 67.47 ± 4.11 | 21.33 ± 5.36 | 8.00 ± 0.72 |
| antmaze-giant-navigate | 39.67 ± 1.77 | 0.33 ± 0.19 | 0.50 ± 0.17 |
| Mean | 45.24 | 13.28 | 4.89 |
Q3: The one-step auxiliary loss performs similarly to the UHM auxiliary loss in Table 2. Would model-based updates using repeated application of a learned one-step latent dynamics model provide similar benefits?
We perform an ablation of different model-based update strategies in the high discount setting (1) MB-TD-JEPA (ours) (2) Monte Carlo samples (MC) which samples states from the dataset trajectories (up to or trajectory end), equivalent to TD() and (3) iterative one-step rollouts (1-Step) with policy queries at each imagined state, up to . We provide the results from these experiments in Table 3 in the updated paper. We find that the proposed combination of a world-model auxiliary loss and model-based updates is necessary for achieving the best performance at long horizons, however, the method for performing model-based updates is flexible in these locomotion environments, with MC and 1-Step performing slightly better than the UHM overall. In the case of MC, however, we note that this is highly dependent on data quality, with the stitch and navigate datasets - which contain meaningful behavior in the dataset trajectories - displaying strong performance, while in the explore dataset, MC updates provide no benefit and model-based rollouts are necessary. While one-step rollouts are performant in navigation environments, consistent with prior work[4], our method provides an advantage over one-step rollouts by parallelizing compute, while one-step rollouts must be sampled iteratively. Further, unlike in the navigation environments which can use a policy conditioned on , raw-state conditioning is necessary in manipulation and pixel environments. As discussed in our response to W1, our method can be extended to handle such cases, while one-step rollouts would require training a distilled policy on .
Caption under Figure 1 specifies "blue dot", however, I see green dot on maze images.
Thank you for noting this, we have updated the coloring and text.
[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit Q-learning." International Conference on Learning Representations (2022).
[2] Ghosh, Dibya, Chethan Anand Bhateja, and Sergey Levine. "Reinforcement learning from passive data via latent intentions." International Conference on Machine Learning. PMLR, 2023.
[3] Shen, Xinwei, and Nicolai Meinshausen. "Engression: extrapolation through the lens of distributional regression." Journal of the Royal Statistical Society Series B: Statistical Methodology 87.3 (2025): 653-677.
[4]Chung, Hojun, Junseo Lee, and Songhwai Oh. "Offline reinforcement learning with universal horizon models." arXiv preprint arXiv:2605.15603 (2026).
[5]Bagatella, Marco, et al. "Td-jepa: Latent-predictive representations for zero-shot reinforcement learning." International Conference on Learning Representations. Vol. 2026. 2026.
We thank the reviewer for their thoughtful review of our paper and insightful comments. We are glad that the reviewer found the method intuitive and the experimental results strong. We reiterate the core contributions of our paper are as follows: (1) introducing a latent-predictive UHM which jointly trains and shapes representations alongside latent-predictive BFM training (2) theoretically motivating this method through an analysis of the shared low-rank decomposition of each method and (3) introducing ν-Bellman updates for TD-JEPA that bootstrap from latent-predictive UHM predictions, which we demonstrate enables training TD-JEPA at high discount factors. We address each of the concerns below:
W1/Q1a: The evaluation covers only a narrow class of environments (AntMaze navigation). There is no validation on manipulation, pixel-based, or substantially stochastic environments. Could authors perform additional empirical evaluations on larger domain of environments?
We agree that extending the method to manipulation and pixel-based environments is of particular interest. There is, however, a critical difference between the navigation environments and the pixel-based and manipulation environments for TD-JEPA which makes this non-trivial. Behavior regularization in the pixel-based and manipulation environments requires an actor conditioned on the raw state space (rather than the latent ϕ), in order for the behavior cloning regularization to be effective [5]. Since our method predicts future states in latent space only, and requires an action at these imagined latent states in order to compute the bootstrap, the method must be adapted to this scenario. For this reason, we felt that these environments represent a material extension of the current method and would be most appropriate as future work.
However, we do believe that our method can be extended to environments that require a state-conditioned actor. As a demonstration, we train an extension of our method on cube-single (horizon ~200) at γ = 0.99. MB-TD-JEPA, with a raw-state actor trains stably and achieves average of ~56% success vs the baseline of 53% (over two seeds, best of sweep). Because this task is short-horizon, we don’t expect model-based targets to improve performance here, however, the experiment shows that our approach can extend to raw-state actors.
We use the following method to extend our method to this scenario: We train an action-free network Vϕ(ϕ(s),z)∈Rdψ by expectile regression, as in IQL [1] with expectile weights computed from the projection onto z [2] :
LV(Vϕ,ϕ,Tϕ)=E(s,a)∼D,z∼Z[τ−1{u⊤z<0}∥u∥2],u=Tϕ−(ϕ(s),a,z)−Vϕ(ϕ(s),z).
which avoids querying the actor entirely. Alternatively, we can simply train an auxiliary action-free network Vϕ(ϕ(s),z)∈Rdψ with the loss:
LV(Vϕ,ϕ,Tϕ)=Es∼D,z∼Z,a∼πz(⋅∣s)[∣∣V(ϕ(s),z)−T(s,a,z)∣∣2]
where we train the action-free successor features predictor from the action-conditioned successor features predictor over true dataset states, at which we are able to query the raw-state policy. Using either method, we can then bootstrap off the imagined state during MB-TD-JEPA updates using this auxiliary network, without needing to compute the action at the imagined state.
We leave a comprehensive evaluation of this extension on long-horizon manipulation tasks to future work.
Regarding substantially stochastic environments, one of the motivations for our method is simplifying jumpy predictions in approximately deterministic environments, where full generative modeling may be unnecessarily expensive, and our method is primarily intended for environments that are approximately deterministic, such as the Mujoco environments tested here. This is a large class of important environments and is generally true for robotics environments for which this method is primarily intended. However, our method could potentially be extended to incorporate stochasticity in the UHM such as through noise conditioning and an energy-based loss as in [3]