acceptodds
Under review as a conference paper at ICLR 2027

Off the Record: What a World Model Guarantees at Unrecorded Actions

Abstract

World models let an agent plan by asking what an action would do instead of taking it. They are trained and validated on transitions recorded under a behavior policy, yet planning asks about actions that policy never took. We ask what a world model can guarantee at such actions. It is elementary that the data alone guarantees nothing there, since a model can fit it perfectly and still be arbitrarily wrong. Our two results say what a guarantee rests on. First, a Lipschitz constant declared for the model's predictor turns distance from the data into a bound. The bound takes two assumptions: the true dynamics respect that constant, and the model's encoder retains the relevant state. Under them, the next-state error of every model that fits the data and respects the constant is bounded from above by a band that widens with that distance. The band is attained in the worst case, and a constant declared too low can make it silently false. Half the disagreement between two models bounds the larger of their errors from below, with no constant. Second, the encoder sets that distance, and discarding nuisance shrinks it. On MuJoCo datasets, models that score alike on held-out data diverge at unrecorded actions, and their error and disagreement grow with that distance. An adversarial search finds an error within a small factor of the band. Replacing part of a dataset with widely varied actions lowers error far from the data. We turn these results into guiding principles. Distance from the data, not a held-out score, should govern how far a world model's prediction is trusted.

Primary area:
reinforcement learning
Keywords:
world models, model-based reinforcement learning, offline reinforcement learning, Lipschitz continuity, representation learning, underspecification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RKu6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the reliability of world-model predictions at state-action pairs not covered by the training data. The authors derive a distance-dependent prediction error bound under Lipschitz continuity and encoder faithfulness assumptions, and analyze how learned representations affect data coverage. Experiments on nine MuJoCo datasets examine off-support prediction errors, model disagreement, action diversity, and representation quality. The paper also explores distance-based penalties for model-based planning.

Critical strengths
  1. Clear theoretical formulation. The authors explicitly characterize the assumptions behind off-support prediction guarantees and provide both an upper bound and a worst-case tightness result. The analysis also considers how encoder faithfulness and state aliasing affect these guarantees.

  2. Systematic experimental analysis. The experiments cover nine MuJoCo datasets with different behavior policies. The evaluations examine prediction error, model disagreement, action coverage, and representation quality, providing empirical support for several theoretical observations.

  3. Useful connection to planning. The paper goes beyond theoretical analysis by investigating distance-based penalties for action selection, providing preliminary evidence that distance to the training data can serve as a useful signal for planning.

Critical weaknesses
  1. Limited novelty beyond closely related theoretical and empirical results. The central distance-dependent error bound largely follows established Lipschitz-based uncertainty analysis [1]. In particular, Suh et al. [2] already derive a closely related bound connecting model bias to distance from training data. Prior work has also studied distance-aware uncertainty estimation in learned representations [3] and its application to model-based offline RL [4]. Although the encoder-dependent formulation and worst-case analysis provide some additional insights, the paper does not sufficiently establish how these extensions constitute a substantial new contribution beyond existing results.

  2. Limited practical applicability of the guarantees. The proposed bound requires knowledge of the true dynamics' Lipschitz constant and encoder faithfulness error, which are generally unavailable for learned world models. Most experiments assume an exact state encoder, while the empirical estimation of the bound relies on simulator access. Consequently, the paper does not convincingly demonstrate how these guarantees could be established for practical learned world models.

  3. Empirical findings are largely expected from the theoretical assumptions. The main experiments show that prediction errors increase with distance, broader action coverage reduces off-support errors, and retaining nuisance information affects representation quality. While these results are consistent with the analysis, they largely confirm established intuitions. More challenging experiments or less expected findings would strengthen the scientific contribution.

[1] M. Milanese and C. Novara. “Set Membership Identification of Nonlinear Systems.” Automatica, 40(6):957–975, 2004.

[2] H. J. T. Suh, G. Chou, H. Dai, L. Yang, A. Gupta, and R. Tedrake. “Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching.” Proceedings of the 7th Conference on Robot Learning (CoRL), PMLR 229:2878–2904, 2023.

[3] G. Tennenholtz and S. Mannor. “Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learning.” Advances in Neural Information Processing Systems (NeurIPS), 35, 2022.

[4] Z. Qiao, J. Lyu, K. Jiao, Q. Liu, and X. Li. “SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning.” Proceedings of the AAAI Conference on Artificial Intelligence, 39(19):20033–20041, 2025.

Questions for authors
  1. Theoretical contribution. Proposition 2 establishes tightness only for the exact-encoder setting ((\alpha=\varepsilon=0)). Is the encoder-dependent error term in Proposition 1 also tight? What additional theoretical insight does this term provide beyond applying classical Lipschitz interpolation with representation error?

  2. Practical applicability. Most experiments use an exact state encoder, and the Lipschitz constant is estimated using the simulator's true dynamics (Appendix C.4). How can the proposed error bound be used for a learned world model when neither the true dynamics nor the encoder error is accessible? Can the authors demonstrate a non-vacuous bound in a learned-representation setting without relying on simulator-derived quantities?

3 comments

Thank you for the careful review, and for finding the theoretical formulation clear, the experiments systematic and the connection to planning useful. Your two questions led to two additions, and the answer to both is yes. Every encoder term of Proposition 1 is attained once (ii)'s is sharpened (Q1). And a world model that learns its encoder from pixels now gets a non-vacuous bound with no simulator-derived input (W2).

W1.

Limited novelty beyond closely related theoretical and empirical results.

Thank you for [2]–[4], now cited in §1 and §6. We agree that the inequality is standard, and we claim neither it nor distance as a new penalty. As §1 says, for states measured directly the band is a form of [1]'s bound. Suh et al. [2] take the same step for one trained model (their Proposition 2), as do Knuth et al. (2021), whom [2] cites. [3] and [4] penalize a distance, [3] in an encoder-decoder model's pullback metric and [4] by a nearest-neighbor search. Both build on MOPO, whose guarantee assumes the penalty bounds the model's error.

What the paper adds is a guarantee through a learned state, where distance can mislead. In Figure 4, an encoder trained by prediction keeps queries close to the data while their error grows, since it loses part of the relevant state. Proposition 1 states when a learned distance does bound the error, with every encoder term attained (Q1). [2] suggests confidence bounds on its constant from slopes within the dataset, and Knuth et al. estimate theirs from extra samples near the data, valid asymptotically. Our constant from the data (W2) imitates queries with edges of the data, and holds with a finite-sample probability under one stated assumption.

W2.

Limited practical applicability of the guarantees.

Thank you for this point and your Q2. You pointed to the missing piece, and answering it made the paper much stronger. As you note, the submitted band tests took the constant and the encoder error from the simulator and an exact encoder. The revision now computes both without the simulator. It estimates the constant from the data, with a formal guarantee. A query lies off the data, so we imitate queries with edges of the data. We then raise the data's largest secant, a lower bound on the world's rate, by the most it grows when an edge is added back. With probability at least 0.99, this constant covers the world's rate at 90% of the queries or more, if all but a few queries are no steeper than the edges. This assumption holds on every dataset, so the guarantee applies there. Those few are at most 3.9% of the queries, within the 5% it allows. See §4 (p.6, l.301) and App G (p.28).

The revision reruns the band experiment with this constant (§5), using the simulator only to search for the queries of largest error, where the band is most likely to fail, and to score them. On 54 new models the band still holds at all but 9 of nearly 700,000 queries. Those 9 lie where the world turns steeper than the constant, as the guarantee allows. At its closest, the band is 0.9–4.1 times the worst error, against 1.7–4.2 with the simulator's constant, and below 1 only at those 9.

For the encoder error, we augment recorded observations with the nuisance and take the largest distance of the encodings from their mean. We test both inputs on a world model that learns its encoder from pixel frames of halfcheetah under varied lighting. No input comes from the simulator, which only renders frames and scores errors. The band holds at all 26,976 drawn queries off the data, under new lighting, and at its closest it is 2.3–3.4 times the worst error (§5, p.8, l.429, App G.3).

W3.

Empirical findings are largely expected from the theoretical assumptions.

We agree that several findings confirm intuitions, such as error growing with distance and varied actions helping. The new tests in W2 are the more challenging experiments you ask for.

Q1. Yes. Thank you for this question, which led to a tighter band. Proposition 1(i)'s encoder term is attained. In (ii), a direct chain gives (1+2λ_S)ε in place of (1+4λ_S)ε, also attained, and Corollary 1(ii) becomes (1+λ_S)ε + δ. The revised Proposition 2 attains each for every α, ε and δ.

The form of the term is classical, as in [1]'s error-in-variables. But classical interpolation takes the input space as given, while a world model learns it. The band asks of that space only faithfulness, which is less than recovering the state (Remark 3). The encoder then sets the distance, so dropping nuisance brings queries closer to the data (Corollary 1). Its distortion scales the world's rate in the slope, and its error sets a floor that neither more data nor a smoother predictor lowers (Remark 5). Merging two states hides the lost state from distance (Figure 4, Remark 2), yet raises that floor to at least the world's change between them. Both terms are attained, so these limits are sharp.

Q2. Yes, see W2.

We would be glad to answer any further questions.

Thanks the authors for their detailed response and additional experiments. While the revision addresses some of my concerns, I remain unconvinced about the novelty and significance of the theoretical contribution. I will maintain my original rating.

We are glad the revision addresses some of your concerns. Thank you for your two questions, which led to a tighter band whose encoder terms are all attained in the worst case, a constant from the data, and a world model that learns its encoder from pixels.

Reviewer_ZLhMOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This is an interesting theoretical paper that aims to characterize how world models differ from the underlying world. They formulate the relation between the world and the world model by defining: (1) a split, that splits a world state to the part that is relevant to state transitions and the part that is not (they call it nuisance), (2) an encoder, that maps a real-world state/observation to a state in the world model, and (3) a "learned" transition function that says how the world model estimates the real world changes after an action. Then, they define (1) how much the encoder may miss by how much states encoded to the same world model state are different from each other, and (2) how much the learned transition differs from the real transition. Parametrizing these differences allows deriving bounds on the relation between these differences and the actual potential inaccuracies when using the world model to predict transitions.

The authors then use their formalism to define several properties of this sort. They also demonstrate these properties empirically on several MuJoCo environments.

Critical strengths
  1. I found their formalism very natural and elegant.
  2. Studying how to bound the error of a world model and what type of assumptions one should make is very important and timely.
Critical weaknesses
  1. The paper is sometimes hard to follow. I am not sure if this is avoidable though given the topic.
  2. The practical relevance of their analysis could have been improved. Ideally, something about how one can derive the bounds on the encoder/transition model in real settings.
  3. I could not fully follow the argument about how future evaluation should occur.
Questions for authors

Q1: Page 2 lines 64-68: This very long sentence is not clear to me. Can you explain it? Q2: Page 3 line 152: why do you need to square the distance? Q3: Page 3, lines 154-157: I do not see the point of this discussion. Q4: What does it mean to "silently fail"? Q5: Can you explain more concretely how you envision future evaluations?

1 comment

Thank you for the thoughtful review and the precise questions. We are glad you found the formalism natural and elegant, and the problem important and timely. Your two main points, on clarity and on practical use, were well taken, and the other reviewers raised them too. In response, §3 now opens with a worked example, §4's first principle opens with what an evaluation should tell a planner, and the band's two bounds come without a simulator (W2 below). Changes are in blue in the revised PDF.

W1.

The paper is sometimes hard to follow.

§3 now opens with a worked example before any definition (p.3, l.112). It takes one recorded transition and a query near it, and shows how the constant turns the query's distance into a bound, and how the encoder sets that distance. The passages you name are revised (Q1, Q3, Q4).

W2.

Ideally, something about how one can derive the bounds on the encoder/transition model in real settings.

Thank you for this point, which the other reviewers raised too. It pointed to the missing piece, and answering it made the paper much stronger. The band needs two bounds: a constant on how fast the world changes, and the encoder's error. The submission took both as given, and the revision now obtains both without a simulator. It estimates the constant from the data, with a formal guarantee. A query lies off the data, so we imitate queries with edges of the data. We then raise the data's largest secant, a lower bound on the world's rate, by the most it grows when an edge is added back. With probability at least 0.99, this constant covers the world's rate at 90% of the queries or more, if all but a few queries are no steeper than the edges. This assumption holds on every dataset, so the guarantee applies there. Those few are at most 3.9% of the queries, within the 5% it allows. See §4 (p.6, l.301) and App G (p.28).

The revision reruns the paper's band experiment with this constant (§5, p.7, l.377), using the simulator only to find hard queries and score them. Its queries were searched for the largest error, where the band is most likely to fail. On 54 new models the band still holds at all but 9 of nearly 700,000 queries. Those 9 lie where the world turns steeper than the constant, as the guarantee allows. At its closest, the band is 0.9–4.1 times the worst error, below 1 only at those 9.

The encoder error bounds how far the nuisance alone moves an encoding. To measure it, we augment recorded observations with the nuisance and take the largest distance of the encodings from their mean. We test both inputs on a world model that learns its encoder from pixel frames of halfcheetah under varied lighting. Its band holds at all 26,976 queries off the data, under new lighting, and at its closest it is 2.3–3.4 times the worst error (§5, p.8, l.429, App G.3).

W3.

I could not fully follow the argument about how future evaluation should occur.

We envision evaluations that tell a planner, before it acts, how wrong each prediction can be. Today a world model's error is measured on held-out data. Models that score alike there can diverge at a planner's actions (Figure 1(a)). Beside that score, each prediction would come with an error bound that grows with its distance from the data (W2). A planner could then penalize actions with a large bound, which lowers one-step regret on all nine datasets (App E.2). The bound would also show what data to collect: widely varied actions, which lower the error far from the data (Figure 3). Over a long horizon, each step would get its own bound, while previous works assume one for all steps (§2). §4's first principle now opens with this (p.6, l.291).

Q1. It sums up the four principles of §4. Judge a prediction by how far its question lies from the data. Under two assumptions, on the world and the encoder, its error is at most a band that grows with that distance. If two models disagree, one of them is wrong by at least half their gap, with no assumption. To bring questions closer to the data, record varied actions, and let the encoder drop only the nuisance. A planner should penalize each action by its distance from the data. §1 now says this in short, plain sentences (p.2, l.69).

Q2. It is just the usual mean squared error. Any loss that grows with the distance and is zero at zero works too.

Q3. A model that predicts its own encodings can collapse to one point, and its loss can then be zero though it has learned nothing. A low loss therefore does not show a good encoder. Faithfulness makes collapse costly, because it asks the encoder to keep the relevant state, and a collapsed encoder keeps none. §4 recommends a term against collapse, and §3 now says why this matters (p.4, l.167).

Q4. It means a constant declared too low can make the band false while nothing in the data shows it (Proposition 2). §1 now says this plainly (p.2, l.63).

Q5. See W3.

We would be glad to answer any further questions.

Reviewer_CtAoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper asks what a world model can guarantee at actions absent from its training data, arguing that any such guarantee must rest on modeler-supplied assumptions: a Lipschitz constant on the dynamics and an encoder that keeps the relevant state. Under these assumptions, the authors bound the error of any model that fits the data and respects the constant by a term that grows linearly with the query's distance from the data, and argue that the encoder determines this distance. Experiments on MuJoCo datasets show that error grows with distance from the data and that the bound comes within a small factor of the worst error found.

Critical strengths
  1. The paper is rather candid about its own strengths and limitations. For example, it states up front that its inequalities are elementary, and it notes that a constant declared too low can make the bound fail. It is also somewhat well written.
  2. The guiding principles section (§4) is thoughtful, albeit many of the principles are already well known. Figure 3 is simple and clean, and makes a nice point: how widely the actions vary matters more than how many are varied.
Critical weaknesses

The weaknesses I would most like to see addressed are:

  1. The central claim seems rather modest and unsurprising: "the farther a query lies from the data, the less its answer can be trusted" (line 50). The paper itself says its inequalities are elementary and that, for states measured directly, the band is a form of the set-membership identification bound (lines 52–53). What it adds is an account of what the modeler must supply when the state is learned (lines 53–54), but I didn't find that this leads anywhere surprising. Penalizing uncertainty away from the data is also already the basis of offline model-based RL methods such as MOPO and MOReL. Relatedly, the paper has a high ratio of definitions to motivation or intuition (Definitions 1–9 in §3.1). This makes a fairly simple idea feel more technical than it needs to be, and I don't think the technicality buys surprising insights. In my opinion this is earnest scholarship, but I believe it is below the mark for acceptance.
  2. I'm not sure how useful the band is in practice. The data only bounds λS\lambda_S from below, and the paper suggests declaring it "from knowledge of the system where it exists" (line 280), but doesn't say where that knowledge would come from when the dynamics are known only through data. The experiments that test the band take λS\lambda_S from the simulator (line 344), which the paper notes a practitioner can't do (line 313). So they show that the band holds when the constant is known, not how to get the constant. The band also seems too conservative to use directly: in the planning experiment, at weight one it penalizes so heavily that the planner keeps the recorded action (line 1147), and §4 instead recommends a small tuned weight, which makes the penalty "a heuristic, not a bound" (line 306).
  3. Some of the figures make simple points but are confusingly worded. For example, Figure 1(a) claims that models that score alike on held-out data diverge with distance. That models trained on different splits of the same data can be wrong in different ways off support is a rather simple point.

Minor suggestions:

  • As the Limits paragraph acknowledges, the encoder is exact in every experiment except the encoder experiment (ε=0\varepsilon = 0), and the evidence comes from control benchmarks. Given the title and the models discussed in §1–2 (Dreamer 4, Genie, JEPA-style models), even one experiment with a learned encoder on image observations would make the encoder side of the theory much more convincing.
  • It is a bit odd that there is no conclusion section.
  • Please report the held-out losses and fit tolerances δ\delta of the six models in Figure 1(a).
Questions for authors
  1. How should a practitioner declare λS\lambda_S without access to a simulator?
  2. The paper notes that there is no shared definition of a world model (line 35), and relates its own definition to JEPA-style models (§4, Appendix F). How would it apply to models that predict in observation space, such as video-generative models, where the encoder is the identity and the nuisance stays in the state? Would ε\varepsilon, and so the band, become vacuous there?
3 comments

Thank you for the careful review, and for finding the paper candid, §4 thoughtful and Figure 3 simple and clean. Your Q1 led to the revision's main addition. Under one stated assumption, the data now gives a constant, with a proof, and the band built from it needs no simulator (App G, W2 below).

W1.

The central claim seems rather modest and unsurprising.

We agree that the intuition in line 50 is unsurprising, and that its bound is standard for states measured directly (lines 52–53). What the paper adds is not obvious, because with a learned state, distance itself can mislead. In Figure 4 an encoder trained by prediction keeps queries close to the data while their error grows. Proposition 1 says when a learned distance does bound the error. The constant from the data (W2) now makes such a bound computable without a simulator, even for a world model that learns its encoder from images (Learned encoder on images, below).

MOPO's and MOReL's guarantees assume an upper bound on the model's error, yet they penalize a predicted standard deviation or cut off on disagreement, and MOPO says its estimator "lacks theoretical guarantees". As §6 notes, a small disagreement does not imply a small error.

You are right that §3 gave definitions before intuition, and it now opens with a worked example. Every definition is used by a result in §3.

A general framework matters even for an unsurprising intuition, for practitioners and for future work.

W2.

I'm not sure how useful the band is in practice.

Thank you for this point and your Q1, which the other reviewers raised too. You pointed to the missing piece, and we are glad you did, since answering it made the paper much stronger. As you say, the data bounds λ_S only from below. But the band for one trained model needs the world's rate only between a query and its nearest recorded pair, and the revision estimates a constant for it from the data, with a formal guarantee. A query lies off the data, so we imitate queries with edges of the data. We then raise the data's largest secant, a lower bound on the world's rate, by the most it grows when an edge is added back. With probability at least 0.99, this constant covers the world's rate at 90% of the queries or more, if all but a few queries are no steeper than the edges. This assumption holds on every dataset, so the guarantee applies there. Those few are at most 3.9% of the queries, within the 5% it allows. §4 now states the constant from the data as its second principle (p.6, l.301), and App G gives the method, its proof and both experiments (p.28).

The submitted tests did take the constant from the simulator, so the revision reruns the paper's band experiment with the constant from the data (§5). Its queries were searched with the simulator for the largest error, where the band is most likely to fail. On 54 new models the band still holds at all but 9 of nearly 700,000 queries. Those 9 lie where the world turns steeper than the constant, as the guarantee allows. At its closest, the band is 0.9–4.1 times the worst error, against 1.7–4.2 with the simulator's constant. It is below 1 only at those 9.

We agree that at weight one the band is too conservative to plan with. Its ranking is still useful. At one step, distance alone sets the band, so distance ranks candidates by how large their error can be, and tuning sets only the weight. MOPO's and MOReL's penalties have no such bound (W1). With every penalty tuned the same way, distance lowers regret on all nine datasets, more than a MOReL cut-off on each and a MOPO spread on eight (App E.2).

W3.

Some of the figures make simple points but are confusingly worded.

Figure 1(a)'s point is indeed simple. It measures the problem the paper starts from, and panel (b) answers it with the band, which comes within a small factor of the worst error found. The six models in (a) share architecture, initialization and about 80% of their training rows, and still diverge. App C.2 now gives the numbers behind "score alike" (Held-out losses, below).

Learned encoder on images. Added. A JEPA-style world model now learns its encoder from pixel frames of halfcheetah, and its band, encoder error included, needs no simulator. The band holds at all 26,976 queries under lighting the data never had, and at its closest it is 2.3–3.4 times the worst error (§5, App G.3).

Conclusion. Added (§7).

Held-out losses. Added to App C, with δ. The six score alike, within 8% of each other on average over the nine datasets and 23% at most.

Q1. See W2.

Q2. Yes, when the nuisance changes much of the observation, since ε is at least half the largest distance between two frames of one situation. For example, that distance is large when lighting or background vary, and small with a fixed camera and scene. The band thus charges a model for the nuisance it keeps, and a model state that drops only the nuisance narrows it (§4).

We would be glad to answer any further questions.

I thank the authors for their thoughtful responses to my questions. I do believe they have made the paper stronger, however I still have lingering concerns about the salience of the results and so will maintain my score.

We are glad you find that our responses made the paper stronger. Thank you for the care you brought to the review, which led to the revision's main additions: a constant from the data and a world model that learns its encoder from pixels.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response and summary of revisions

We thank the three reviewers for their careful reviews. We are glad they found the formalism "very natural and elegant" (ZLhM) and "clear" (RKu6), the problem "very important and timely" (ZLhM), the paper candid and its guiding principles thoughtful (CtAo), and the experiments "systematic", with a "useful connection to planning" (RKu6). Their questions led to the revision's main additions. The band's two inputs now come without a simulator: a constant from the data, with a proof under one stated assumption, and the encoder error from augmented recorded observations. We also sharpen one encoder term of the band, and show that every encoder term is attained in the worst case. The revised PDF marks each change in blue, the list below gives its place, and we answer each review in its own reply.

A constant from the data. All three reviewers asked how the band's constant can be found in practice, without a simulator. We are grateful for this question. It pointed to the missing piece, and answering it made the paper much stronger. For one trained model, the revision estimates the constant from the data, with a formal guarantee. A query lies off the data, so we imitate queries with edges of the data. We then raise the data's largest secant, a lower bound on the world's rate, by the most it grows when an edge is added back. With probability at least 0.99, this constant covers the world's rate at 90% of the queries or more, if all but a few queries are no steeper than the edges. This assumption holds on every dataset, so the guarantee applies there. Those few are at most 3.9% of the queries, within the 5% it allows (App G.1, Figure 17). §4 now states the constant from the data as its second principle (p.6, l.301), and App G gives the method, its proof and both experiments (p.28).

Summary of revisions

Theory

  • A constant from the data (above), with new Algorithm 1, Proposition 3, Assumption 2 and Corollary 2, and the proofs in G.4 (p.32), as asked by CtAo, ZLhM and RKu6.
  • Every encoder term attained (Propositions 1(ii) and 2(ii), Corollary 1(ii), Remark 2, Remark 5 in App B, proofs in App A), as asked by RKu6. Proposition 1(ii)'s encoder term sharpens from (1+4λ_S)ε to (1+2λ_S)ε.

Experiments

  • The band experiment with the constant from the data (§5, p.7, l.377, App G.2 and Figure 18), as asked by CtAo and RKu6. On 54 new models the band holds at all but 9 of nearly 700,000 searched queries, and at its closest it is 0.9–4.1 times the worst error.
  • A world model that learns its encoder from pixels (§5, p.8, l.429, App G.3 and Figure 19), as asked by CtAo, ZLhM and RKu6. Its band needs no simulator and holds at all 26,976 drawn queries, and at its closest it is 2.3–3.4 times the worst error.
  • Held-out losses and δ for the models of Figure 1(a) (App C.2 and Table 1), as asked by CtAo.

Prior work

  • Suh et al., Tennenholtz & Mannor and Qiao et al. (SUMO) cited (§1, p.1, l.53, and §6, p.10, l.511), as asked by RKu6.
  • §1 and §6 say what is standard and what is new, as asked by CtAo and RKu6. The inequality is standard (§1), and distance as a penalty is prior work (§6). The guarantee through a learned state, its sharp encoder terms and the constant from the data are new (§1, p.2, l.54).

Presentation

  • A worked example opening §3 (p.3, l.112), as asked by CtAo and ZLhM. It takes one recorded transition and a query near it, and shows how the constant turns the query's distance into a bound.
  • §1's contributions in plain sentences (p.2, l.60), with "silently" explained, as asked by ZLhM.
  • §4's first principle now opens with what an evaluation should tell a planner (p.6, l.291), as asked by ZLhM.
  • The collapse passage in §3 says why it matters (p.4, l.167), as asked by ZLhM.
  • A conclusion (§7), as asked by CtAo.
  • Sentences the new results change (§3 at p.4, l.195, §5 at p.7, l.337, and §6 at p.10, l.522).

We thank the reviewers again for their patience. We wanted to answer every point thoroughly, with new experiments and proofs, and this took until now. We know little time remains, and we would be grateful for any further comments.

AuthorsOpenReview ↗

Summary of the discussion

As the discussion period approaches its end, here is a short summary of where it stands.

Why we think the paper matters. World models are used to plan at actions their data lacks, yet they are evaluated by their error on recorded behavior, which says nothing on its own about such actions. There, distance from the data should govern how far a prediction is trusted. Bounds in that distance exist when the input space is given, but a world model learns its input space, in which distance can mislead. Our band covers that case: under stated assumptions, it bounds the error through a learned distance and charges the encoder for what it loses or distorts, so it stays valid where distance alone falls short. Every encoder term is attained in the worst case, so no amount of data removes the floor an encoder's error sets. World models have no shared definition, and our framework offers one to build on. The theory also leads to practice, from how to collect data to how to plan.

What the reviewers valued. ZLhM found the problem "very important and timely" and the formalism "very natural and elegant". RKu6 found the theoretical formulation "clear", with "both an upper bound and a worst-case tightness result", the experiments "systematic" and the connection to planning "useful". CtAo had "no substantive concerns" about soundness, and found the paper "rather candid" and its guiding principles "thoughtful".

The shared question. All three reviewers were concerned with obtaining the band's inputs in practice. The revision addresses this with a practical method, backed by a formal guarantee for the constant. Under one stated assumption, a constant from the data covers the world's rate at 90% of queries or more, with probability at least 0.99 (Proposition 3). The resulting band holds at all but 9 of nearly 700,000 adversarially searched queries, as the guarantee allows, and is non-vacuous for a world model that learns its encoder from pixels (§5).

After the rebuttal. CtAo and RKu6 kept their scores over novelty and significance, which we take up above, yet both found that the revision helped: CtAo wrote that our responses "have made the paper stronger", and RKu6 that "the revision addresses some of my concerns".

We are grateful to the reviewers, whose questions led to the revision's main additions, and to the AC for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.