acceptodds
Under review as a conference paper at ICLR 2027

Beyond State Sufficiency: The Dimension of Gain-Bounded Predictive Realization

Abstract

Once a representation fully specifies the state, can regular prediction still require additional dimensions? We establish a quantitative separation between state identification and regular predictive realization. Transition structure can impose an additional dimensional requirement, even when no state information is missing. We define predictive realization dimension and derive verifiable certificates for exact prediction under a shared regularity bound. These certificates rule out all state-preserving encodings below certified thresholds and permit non-invertible transitions. Grid experiments preserve every state identity after dimension reduction, yet the walled systems require more dimensions even when small expansions are allowed. Our results identify a limit of information-based dimensionality assessment: coordinates can be redundant for state identification but necessary for predictive geometry.

Primary area:
learning theory
Keywords:
representation learning, latent dimension, world models, predictive realization, Lipschitz regularity, Euclidean geometry

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GGrnOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that even when a representation fully preserves state information, additional dimensions may be necessary to support regular prediction under bounded distance expansion across all actions. Theoretical lower bounds and synthetic experiments show that state identifiability alone does not guarantee a sufficiently regular predictive geometry.

Critical strengths

The paper shows that representing state information and regular prediction impose distinct requirements. Therefore, successful state recovery alone does not guarantee that a representation captures the predictive structure needed for the task.

Critical weaknesses
  1. The paper does not provide a formal definition of state recovery in general. How can we determine whether a representation fully specifies the underlying state, and how robust is state recovery to noise in observations or measurements?

  2. How this finiding can be expanded to recurrent models, e.g., in state-space models where state recovery is more challenging?

Questions for authors

The significance of the work and its impact on real-world applications seem limited. There is only one set of synthetic experimental results, which would be great if it could be elaborated/extended to real-world use cases.

1 comment

We sincerely thank the reviewer for the constructive feedback and for recognizing the distinction between state information preservation and predictive regularity. Your comments have helped us better clarify our theoretical contributions and their scope.

Our central question and contributions

Our work is motivated by a fundamental question: Why do world models, which aim to learn compact state representations, often use latent dimensions substantially larger than the intrinsic degrees of freedom of the underlying system? What roles might these seemingly redundant dimensions play?

Previous studies have explored explanations including overparameterization, feature expansion, Koopman representations, and Neural ODE expressivity.

We identify and rigorously establish a different mechanism: even when all state information is preserved, realizing action-conditioned transitions under a prescribed Lipschitz gain may require additional dimensions because all actions share the same latent representation.

Our theoretical contributions are:

  1. Formalizing this dimensional requirement.
  2. Proving strict dimensional separation.
  3. Deriving encoding-independent dimension certificates.

1. State recovery and noise robustness

We define exact state recovery through injectivity, equivalently, the existence of an inverse decoder on the represented state space (clarified in Section 3.1).

For example, a continuous two-dimensional state space (\mathcal X=[0,1]^2) cannot be losslessly encoded into (\mathbb R) by a continuous encoder. This illustrates the role of state-space structure; our present theorems require injectivity without assuming continuity.

We agree that robust recovery from noisy observations is important, but it is outside our scope. Our contribution concerns the additional dimensions that regular prediction may require even when state recovery is perfect.

2. Extension to recurrent models

Our framework is architecture-independent and also applies to deterministic recurrent hidden-state updates treated as input-conditioned transitions.

Recurrence alone does not remove the dimensional constraints imposed by gain-bounded dynamics. Recovering sufficient states from partially observable histories is a separate problem; when the hidden state is not sufficient, our analysis concerns its update geometry rather than recovery of the underlying physical state.

3. Practical significance

We have added experiments using trained LeWorldModel representations on PushT and Cube across five latent dimensions.

We observe that larger representations generally exhibit lower upper-tail transition expansion and prediction errors, accompanied by improved task performance. These findings extend our analysis beyond synthetic systems and support the practical relevance of predictive geometry, although they do not establish a causal dimensional requirement.

Reviewer_qY9TOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Introduces predictive realization dimension, minimum latent dimension for exact, gain-bounded transitions. Develops Gram-matrix and pair-recurrence-based dimension lower bounds.

Strength

  • Sound mathematical framework combining PSD Gram matrices, distance variation, and transition recurrence.
  • Applicable to non-invertible dynamics and relaxed Lipschitz gains ((L>1)).
  • Clear theoretical insight and verifiable evidence

Weakness

  • Limited connection to practical learned world models.
  • Loose dimension–gain certificates beyond unit gain: Bounds deteriorate sharply: 29/47/52 at (L=1), versus 5/3/3 at (L=1.02).
Critical strengths
  • Sound mathematical framework combining PSD Gram matrices, distance variation, and transition recurrence.
  • Applicable to non-invertible dynamics and relaxed Lipschitz gains ((L>1)).
  • Clear theoretical insight and verifiable evidence
Critical weaknesses
  • Limited connection to practical learned world models.
  • Loose dimension–gain certificates beyond unit gain: Bounds deteriorate sharply: 29/47/52 at (L=1), versus 5/3/3 at (L=1.02).
Questions for authors

Q1. Can the authors characterize a tighter dimension–gain profile for one non-invertible system? Q2. How much looseness arises from witness selection versus distance-variance bounds? Q3. Can dimension certificates survive bounded prediction error (\varepsilon), with minimum state separation (\delta)? Q4. Which gain-one permutation results extend beyond established invariant Gram-matrix and group-symmetry formulations?

3 comments

We thank the reviewer for the careful and insightful review. We especially appreciate Q2's distinction between witness selection and distance-variance slack; it identified an important weakness in our original relaxed-gain analysis.

Q1. Tighter dimension–gain profile.

We cannot yet characterize the exact profile. However, gain-aware witness search substantially strengthens the certificates for our non-invertible mazes. At L=1.02L=1.02, the lower bounds for A/B/C improve from 5/3/3 to 8/8/10, using verified witnesses (k,h)=(19,14),(18,13),(17,10)(k,h)=(19,14),(18,13),(17,10). At L=1.01L=1.01, they improve from 13/8/9 to 15/16/19. These are valid lower bounds, not exact dimensions or globally optimal witnesses.

Q2. Sources of looseness.

Motivated by the reviewer's precise distinction, we separately ablated witness size kk and recurrence horizon hh. For Maze C at L=1.02L=1.02, fixing k=17k=17 and reducing h=30,22,10h=30,22,10 yields bounds 3,4,10; fixing h=10h=10 and increasing k=3,8,13,17k=3,8,13,17 yields bounds 2,6,8,10. Mazes A/B show the same trend. Every witness and path length was independently verified in the full pair graph.

The distance-variance inequality is unchanged, so witness selection demonstrably explains part of the original looseness. We cannot yet quantify the remaining slack from the distance-ratio and variance-to-rank reductions. We also conjecture that a single worst-case horizon and global variance bound may obscure stronger constraints within smaller, dynamically coupled subsets; our experiments suggest, but do not prove, this local explanation.

Q3. Bounded prediction error.

Our main results concern exact deterministic prediction. Under a uniform one-step latent prediction error bound and a prescribed minimum state separation, the triangle inequality introduces an additional error term into each pairwise transition constraint.

Accumulating these errors along recurrence paths yields a relaxed distance-ratio bound, which can be substituted into our existing dimension certificate. Thus, nontrivial dimension lower bounds can persist when the prediction error is sufficiently small relative to state separation.

The separation assumption is essential: without a fixed scale, states can be placed arbitrarily close together in one dimension, allowing a constant predictor to satisfy any positive absolute error budget.

We provide the formal extension and its proof in Appendix B.

Q4. Classical symmetry.

We agree that gain-one permutation examples follow established invariant Gram-matrix and group-symmetry ideas; we do not claim a new group-theoretic result. Their role is to illustrate a sharp separation. Our broader framework and quantitative pair-recurrence certificates also cover non-invertible actions and gains L>1L>1.

Revisions.

Section 5.2 now reports gain-aware bounds; Appendix G.2 adds verified witnesses and controlled k/hk/h ablations; Appendix B gives the bounded-error result; Section 6 discusses global-versus-local looseness and its limitations. We thank the reviewer for motivating these substantive improvements.

I thank the authors for their response to my questions and for providing additional explaination and experimental results as asked. I also appreciate the changes made to the paper.

Thank you for your thoughtful feedback and for acknowledging our revisions and additional results. We particularly appreciate your insightful questions about the dimension–gain trade-off and the sources of certificate looseness, which have helped us substantially strengthen our theoretical analysis. We sincerely appreciate your time and constructive suggestions.

Reviewer_EQEVOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This work studies the minimum Euclidean dimension needed to represent each state distinctly while ensuring that all deterministic action-conditioned transitions satisfy some Lipschitz gain bound. The authors derive dimension lower bounds by translating transition constraints into restrictions on pairwise distances, then they use the relationship between these distances and the rank of the representation Gram matrix. Their experiments on finite transition systems demonstrate that preserving state identities does not necessarily guarantee a low-dimensional representation satisfying the required gain bound.

Critical strengths

1- The question is interesting, and the theoretical analysis looks sound. The provided experiments on finite-state systems support the main message of the paper that preserving the distinctness of states does not guarantee a low-dimensional representation satisfying a given gain bound.

Critical weaknesses

1- The related work is too abstract and brief, and does not sufficiently clarify the paper's novelty relative to prior works. Several research directions are summarized through broad statements and lists of citations, without identifying the closest existing results or explaining precisely what limitations the present work addresses.

Also, some terminology is hard to follow, like high and low family used in section 4.2 without prior definition.

2- The contribution to the representation learning community remains unclear beyond the theoretical insight that preserving state identities does not guarantee a low-dimensional representation satisfying a given gain bound.

Questions for authors

1- The theoretical analysis appears mathematically sound, but the results seem to rely largely on classical tools from distance geometry, Gram-matrix rank analysis, and dynamical systems. The paper would benefit from a clearer explanation of its novel theoretical contributions. In particular, which aspects of the proposed dimension certificates go beyond existing techniques, and what previously unresolved problem do they address?

2- What implications do the theoretical results have for learned world models, given that the main experiments use finite-state systems and coordinate optimization rather than training neural models?

3- The paper treats the gain bound as given, but in practical learning problems, this bound may not be known beforehand. How can the current analysis inform the choice of embedding dimension in such settings?

3 comments

We sincerely thank the reviewer for the constructive and insightful feedback. Your comments helped us recognize that the original presentation did not sufficiently emphasize our central research question and theoretical contributions. We have therefore reorganized the Introduction and Related Work to clarify our motivation and distinguish our results from prior work. We have also clarified the definitions and terminology of the high/low families in Section 4.2.

Q1. Novel theoretical contributions and previously unresolved problems

Our work is motivated by a fundamental question: Why do world models, which aim to learn compact, low-dimensional state representations, often employ latent embeddings whose dimensions substantially exceed the intrinsic degrees of freedom of the underlying system? What roles might these seemingly redundant dimensions play?

This motivation is not entirely new. Previous studies have explored different explanations, including overparameterization, nonlinear feature expansion, Koopman representations, and expressivity limitations of Neural ODEs.

Our contribution is to identify and rigorously establish a different mechanism for additional dimensional requirements: even when a representation preserves all state information, realizing action-conditioned transitions under a prescribed Lipschitz gain may require additional dimensions because all actions must share the same latent representation.

Unlike explanations relying on linear predictors, ODE flows, or particular neural architectures, our analysis identifies dimensional constraints imposed by the transition structure itself, even when arbitrary nonlinear predictors are allowed.

We agree that distance geometry and Gram-matrix rank analysis are classical mathematical tools. Our novel theoretical contributions are:

  1. Formalizing this dimensional requirement.

  2. Proving strict dimensional separation.

  3. Deriving encoding-independent dimension certificates.

Thus, our contribution goes beyond applying classical mathematical techniques: we formulate a previously unaddressed joint requirement on representation dimension and predictive regularity, characterize its feasibility, and prove that it can impose unavoidable dimensional constraints.

Q2. Implications for learned world models

Our results provide a new perspective for evaluating learned representations: recovering the state does not necessarily imply that the latent dimension is sufficient for prediction under a prescribed regularity requirement.

There are two implications:

  1. Latent compression and predictive regularity should not be considered independently. Representation dimension, the Lipschitz gain budget, and transition structure jointly determine the feasibility of exact regular prediction.

  2. Our certificates provide encoding-independent necessary conditions. Under our assumptions of deterministic transitions, exact realization, and a prescribed gain budget, if the latent dimension falls below a certified lower bound, no encoder, training method, or nonlinear predictor can simultaneously satisfy the exact transition requirements and that gain.

Regarding our experimental design, we directly optimize the coordinates of 64 states to isolate the geometric feasibility problem from the optimization difficulties of neural-network training. Our theory does not assume that standard neural training automatically approaches the minimum achievable Lipschitz gain. Consequently, neural prediction error alone may not directly reveal the theoretical dimensional obstruction.

To further connect our analysis with practical world models, we have added experiments using LeWorldModel on two robotic manipulation tasks, PushT and Cube. Across five latent dimensions, we examine transition expansion, unrealized expansion, prediction error, and task success, together with a shuffled-successor control. Larger representations generally exhibit lower upper-tail transition expansion and prediction errors, accompanied by improved task performance.

These findings provide empirical evidence that predictive geometry is relevant to learned world models. However, we do not interpret them as a causal demonstration of dimensional necessity, since model capacity also changes with latent dimension and continuous action sequences are only approximately matched.

Q3. Unknown gain bound in practice

We clarify that the Lipschitz gain LL is a task-dependent design budget, rather than an intrinsic environmental constant that must be known beforehand. When an appropriate budget is uncertain, our dimension certificates can be evaluated across different values of LL, revealing the trade-off between latent dimension and predictive regularity.

I thank the authors for their response to my questions and for providing additional experimental results to the paper. I also appreciate the major changes made to the Related Work section. I maintain my score of 2.

Thank you for reconsidering our paper and acknowledging the revisions and additional experiments.

We would respectfully emphasize that the revised manuscript directly addresses the central concerns raised in your review. In particular, our contribution goes beyond proposing dimension certificates: we provide a new, provable explanation for why world models may require additional latent dimensions beyond those needed to preserve state information. We have also added experiments on learned world models to strengthen the connection to practical settings.

We understand that these revisions may not fully change your assessment. However, if a specific concern remains decisive for your rating, we would greatly appreciate your clarification, as it would help us better understand and address the remaining limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.