- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the reliability of world-model predictions at state-action pairs not covered by the training data. The authors derive a distance-dependent prediction error bound under Lipschitz continuity and encoder faithfulness assumptions, and analyze how learned representations affect data coverage. Experiments on nine MuJoCo datasets examine off-support prediction errors, model disagreement, action diversity, and representation quality. The paper also explores distance-based penalties for model-based planning.
Critical strengths
-
Clear theoretical formulation. The authors explicitly characterize the assumptions behind off-support prediction guarantees and provide both an upper bound and a worst-case tightness result. The analysis also considers how encoder faithfulness and state aliasing affect these guarantees.
-
Systematic experimental analysis. The experiments cover nine MuJoCo datasets with different behavior policies. The evaluations examine prediction error, model disagreement, action coverage, and representation quality, providing empirical support for several theoretical observations.
-
Useful connection to planning. The paper goes beyond theoretical analysis by investigating distance-based penalties for action selection, providing preliminary evidence that distance to the training data can serve as a useful signal for planning.
Critical weaknesses
-
Limited novelty beyond closely related theoretical and empirical results. The central distance-dependent error bound largely follows established Lipschitz-based uncertainty analysis [1]. In particular, Suh et al. [2] already derive a closely related bound connecting model bias to distance from training data. Prior work has also studied distance-aware uncertainty estimation in learned representations [3] and its application to model-based offline RL [4]. Although the encoder-dependent formulation and worst-case analysis provide some additional insights, the paper does not sufficiently establish how these extensions constitute a substantial new contribution beyond existing results.
-
Limited practical applicability of the guarantees. The proposed bound requires knowledge of the true dynamics' Lipschitz constant and encoder faithfulness error, which are generally unavailable for learned world models. Most experiments assume an exact state encoder, while the empirical estimation of the bound relies on simulator access. Consequently, the paper does not convincingly demonstrate how these guarantees could be established for practical learned world models.
-
Empirical findings are largely expected from the theoretical assumptions. The main experiments show that prediction errors increase with distance, broader action coverage reduces off-support errors, and retaining nuisance information affects representation quality. While these results are consistent with the analysis, they largely confirm established intuitions. More challenging experiments or less expected findings would strengthen the scientific contribution.
[1] M. Milanese and C. Novara. “Set Membership Identification of Nonlinear Systems.” Automatica, 40(6):957–975, 2004.
[2] H. J. T. Suh, G. Chou, H. Dai, L. Yang, A. Gupta, and R. Tedrake. “Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching.” Proceedings of the 7th Conference on Robot Learning (CoRL), PMLR 229:2878–2904, 2023.
[3] G. Tennenholtz and S. Mannor. “Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learning.” Advances in Neural Information Processing Systems (NeurIPS), 35, 2022.
[4] Z. Qiao, J. Lyu, K. Jiao, Q. Liu, and X. Li. “SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning.” Proceedings of the AAAI Conference on Artificial Intelligence, 39(19):20033–20041, 2025.
Questions for authors
-
Theoretical contribution. Proposition 2 establishes tightness only for the exact-encoder setting ((\alpha=\varepsilon=0)). Is the encoder-dependent error term in Proposition 1 also tight? What additional theoretical insight does this term provide beyond applying classical Lipschitz interpolation with representation error?
-
Practical applicability. Most experiments use an exact state encoder, and the Lipschitz constant is estimated using the simulator's true dynamics (Appendix C.4). How can the proposed error bound be used for a learned world model when neither the true dynamics nor the encoder error is accessible? Can the authors demonstrate a non-vacuous bound in a learned-representation setting without relying on simulator-derived quantities?
3 comments
Thanks the authors for their detailed response and additional experiments. While the revision addresses some of my concerns, I remain unconvinced about the novelty and significance of the theoretical contribution. I will maintain my original rating.
We are glad the revision addresses some of your concerns. Thank you for your two questions, which led to a tighter band whose encoder terms are all attained in the worst case, a constant from the data, and a world model that learns its encoder from pixels.
Thank you for the careful review, and for finding the theoretical formulation clear, the experiments systematic and the connection to planning useful. Your two questions led to two additions, and the answer to both is yes. Every encoder term of Proposition 1 is attained once (ii)'s is sharpened (Q1). And a world model that learns its encoder from pixels now gets a non-vacuous bound with no simulator-derived input (W2).
W1.
Thank you for [2]–[4], now cited in §1 and §6. We agree that the inequality is standard, and we claim neither it nor distance as a new penalty. As §1 says, for states measured directly the band is a form of [1]'s bound. Suh et al. [2] take the same step for one trained model (their Proposition 2), as do Knuth et al. (2021), whom [2] cites. [3] and [4] penalize a distance, [3] in an encoder-decoder model's pullback metric and [4] by a nearest-neighbor search. Both build on MOPO, whose guarantee assumes the penalty bounds the model's error.
What the paper adds is a guarantee through a learned state, where distance can mislead. In Figure 4, an encoder trained by prediction keeps queries close to the data while their error grows, since it loses part of the relevant state. Proposition 1 states when a learned distance does bound the error, with every encoder term attained (Q1). [2] suggests confidence bounds on its constant from slopes within the dataset, and Knuth et al. estimate theirs from extra samples near the data, valid asymptotically. Our constant from the data (W2) imitates queries with edges of the data, and holds with a finite-sample probability under one stated assumption.
W2.
Thank you for this point and your Q2. You pointed to the missing piece, and answering it made the paper much stronger. As you note, the submitted band tests took the constant and the encoder error from the simulator and an exact encoder. The revision now computes both without the simulator. It estimates the constant from the data, with a formal guarantee. A query lies off the data, so we imitate queries with edges of the data. We then raise the data's largest secant, a lower bound on the world's rate, by the most it grows when an edge is added back. With probability at least 0.99, this constant covers the world's rate at 90% of the queries or more, if all but a few queries are no steeper than the edges. This assumption holds on every dataset, so the guarantee applies there. Those few are at most 3.9% of the queries, within the 5% it allows. See §4 (p.6, l.301) and App G (p.28).
The revision reruns the band experiment with this constant (§5), using the simulator only to search for the queries of largest error, where the band is most likely to fail, and to score them. On 54 new models the band still holds at all but 9 of nearly 700,000 queries. Those 9 lie where the world turns steeper than the constant, as the guarantee allows. At its closest, the band is 0.9–4.1 times the worst error, against 1.7–4.2 with the simulator's constant, and below 1 only at those 9.
For the encoder error, we augment recorded observations with the nuisance and take the largest distance of the encodings from their mean. We test both inputs on a world model that learns its encoder from pixel frames of halfcheetah under varied lighting. No input comes from the simulator, which only renders frames and scores errors. The band holds at all 26,976 drawn queries off the data, under new lighting, and at its closest it is 2.3–3.4 times the worst error (§5, p.8, l.429, App G.3).
W3.
We agree that several findings confirm intuitions, such as error growing with distance and varied actions helping. The new tests in W2 are the more challenging experiments you ask for.
Q1. Yes. Thank you for this question, which led to a tighter band. Proposition 1(i)'s encoder term is attained. In (ii), a direct chain gives (1+2λ_S)ε in place of (1+4λ_S)ε, also attained, and Corollary 1(ii) becomes (1+λ_S)ε + δ. The revised Proposition 2 attains each for every α, ε and δ.
The form of the term is classical, as in [1]'s error-in-variables. But classical interpolation takes the input space as given, while a world model learns it. The band asks of that space only faithfulness, which is less than recovering the state (Remark 3). The encoder then sets the distance, so dropping nuisance brings queries closer to the data (Corollary 1). Its distortion scales the world's rate in the slope, and its error sets a floor that neither more data nor a smoother predictor lowers (Remark 5). Merging two states hides the lost state from distance (Figure 4, Remark 2), yet raises that floor to at least the world's change between them. Both terms are attained, so these limits are sharp.
Q2. Yes, see W2.
We would be glad to answer any further questions.