acceptodds
Under review as a conference paper at ICLR 2027

Equivariance Transports, Closure Completes: Gauge Obstructions in Latent World Models

Abstract

Equivariance does not make an exact latent-prediction optimum a predictive state. Two finite constructions prove this: a persistent covariant copy under autonomous self-prediction, and a hidden trajectory gauge despite canonical innovation, hard transport, a proper anchor, and a supplied pose–quotient reconstructor. In a free, observation-aligned Borel class, we prove that deterministic equivariant emission driven by the world's specified joint innovation exists if and only if the invariant code carries the point gauge. This specializes standard factorization and is distinct from full-future-law sufficiency with auxiliary noise. We separate the assumptions for quotient-future and full-future sufficiency. Controlled studies then test how supplied information enters learned dynamics. With supplied physical calibration and known association, calibrated routing with multi-step training attains mean H8 NMSE 0.127500 versus 0.675110 for persistence, winning all 20 frozen-encoder blocks. Readout calibration and one-step retraining fail their tested new-object criteria. A separate matched-update study reverses the advantage of root supervision, and sensor-mounting Pendulum favors system identification over learned dynamics. These contrasts locate calibration's benefits without claiming generic predictive superiority.

Primary area:
learning theory
Keywords:
equivariance, world models, predictive sufficiency, gauge ambiguity, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_G9a8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a fundamental question in equivariant latent world models: whether an exact optimum of an equivariant latent-prediction objective must be a predictive state or sufficient for future prediction. The authors argue that equivariance alone does not guarantee predictive sufficiency because of a residual latent gauge ambiguity. Two finite counterexamples are constructed showing that exact optimization of common latent-prediction objectives can still fail to produce predictive states (a persistent covariant copy, and a hidden "trajectory gauge" that survives canonical innovation, hard transport and a supplied reconstructor). The paper then introduces a formal notion of point gauge closure and proves an exact equivalence between gauge closure and deterministic equivariant emission within a specified class of latent world models. Additional theoretical results separate quotient-future sufficiency from full-future sufficiency and characterize interfaces that can restore predictive closure. Extensive synthetic experiments investigate the practical consequences of calibration, routing, synchronization, and latent alignment in learned predictors. Theorems 5.1 and 5.2 give conditional sufficiency results. Experiments on a synthetic SO(2) point-cloud world and a sensor-mounting Pendulum test supplied calibration.

Critical strengths
  1. Deep and original theoretical contribution.

The paper identifies a subtle limitation of equivariant latent world models that appears largely overlooked in prior JEPA and equivariant-representation literature. The notion of residual gauge ambiguity and the distinction between transport and predictive closure provide a new conceptual framework for analyzing latent sufficiency. The finite counterexamples and necessity results are mathematically interesting and significantly advance understanding of representation identifiability in world models.

  1. Strong formal development with precise assumptions.

Unlike many theoretical works that rely on informal reasoning, this paper carefully states assumptions, contracts, measurable-space requirements, and sufficiency conditions. The distinction between quotient-future prediction and full-future prediction is clearly formalized, and several nontrivial results are proved rigorously.

  1. Thm H.1 gives a clean reason why orientation information is needed. Without it, risk is ≥ 0.737 at H8, against 0.1275 achieved with calibration.
Critical weaknesses
  1. Excessive complexity and limited accessibility.

The presentation is extremely dense. The paper introduces a large number of concepts, assumptions, interfaces, contracts, gauges, kernels, and auxiliary definitions before providing intuition. Even readers familiar with equivariant representation learning and world models will likely require substantial effort to follow the main argument. This significantly reduces the practical impact of otherwise interesting theoretical contributions.

  1. Limited empirical relevance to modern world-model benchmarks.

Although the paper includes many controlled synthetic studies, the experiments do not convincingly demonstrate that the identified gauge obstructions occur in widely used large-scale world-model architectures (e.g., JEPA, V-JEPA, TD-JEPA, Dreamer-like systems, or transformer-based world models). Consequently, the practical significance of the theoretical findings remains somewhat uncertain.

  1. The theory–experiment link is weak.
  • The theory covers autonomous equivariant worlds with exact contracts. No experiment measures the innovation kernel, emission, point closure or k_inv (Sec. 7 admits this).

  • The testbed has global-frame actions with q⁺ = q·exp(i f(qa)). That is not equivariant under rotating the state alone, and the gauge matters there because actions are in the world frame. That is a different mechanism from the theorem's.

  • The AR transition is trained on exact physical next-step targets (H.2, H.3), not a latent-prediction loss. The positive result is therefore supervised dynamics in a calibrated frame, not JEPA-style latent prediction.

Questions for authors

1.The paper focuses on exact global minima and exact interface contracts. To what extent do the identified gauge obstructions appear in practical neural-network training regimes where optimization is approximate and finite-data effects dominate?

2.Can the authors provide experimental evidence on contemporary visual world-model benchmarks demonstrating that the proposed gauge ambiguities materially affect downstream prediction or planning performance?

3.Several recent works study latent identifiability and factorized JEPA representations. Recent work such as JEPA+LIFS [1] explicitly promotes structured and identifiable latent representations. Can the authors discuss whether the gauge obstruction identified in this paper persists under JEPA+LIFS, and ideally provide a quantitative comparison? Such a comparison would help determine whether gauge closure addresses a limitation that remains unresolved by existing latent-factorization approaches or whether related mechanisms already mitigate the issue in practice. [1]Belloulata, 'Contractive MASO-Generalized Predictors for Stable Latent-Space Learning in JEPA, in Transactions on Machine Learning Research (TMLR), 2026, https://openreview.net/forum?id=k2Z2gPOtlq.

4.The paper distinguishes quotient-future sufficiency from full-future sufficiency. Can the authors provide a practical diagnostic that would allow practitioners to determine empirically whether a learned representation suffers from the predicted gauge obstruction?

  1. Does a standard latent-prediction (JEPA-style) equivariant world model, trained without physical-label targets, show the gauge or copy failure, and does closure fix it? If closure is the missing piece, why does ROOT hurt in Table 11? What is the empirical C_t? How does the SO(2) study change if observation-plus-action history is allowed? Does Thm 4.2 survive if K(V_t;·) is only approximately V-measurable?

What would move this up: a non-toy, self-supervised demonstration with the gauge measured directly, scaled object diversity, and a main text that a non-specialist can read.

4 comments
AuthorsOpenReview ↗

Response to Reviewer G9a8 (part 1 of 2)

(a) We trained on point clouds alone. No physical pose label entered any loss. With first-glimpse orientation, self-predictive forecasts became less sensitive to evidence in the second glimpse. Adding a loss for the next observed direction kept that evidence readable, including with a moving-average target encoder.

Task. Each glimpse can be reversed by a half turn with probability 1/4. Both encoders train to predict their next code and the next cloud rotated to align its observed direction with a fixed axis. This removes its overall orientation. One also predicts the next observed direction in the original observation coordinates. Under the two conventions the encoder applies a learned rotation to either the first or the current observed direction. We score frozen readouts identically under both conventions. Readout error is one quarter of the mean squared distance from its forecast to the conditional mean of the third observed direction given the first two.

Two-sided gradients (Appendix L). The first glimpse set orientation. We tracked the readouts at the weaker ridge penalty (0.001). From 500 to 32,000 updates the median self-predictive readout error rose from 0.0010 to 0.0099 (first-glimpse optimum 0.009375). The median forecast difference between agreeing and opposing glimpses fell from 0.180 to 0.001 (optimal 0.2). At this penalty, error rose and the difference fell in every block. With the added direction loss the final median difference was 0.201 and the median error was 0.0008. The preregistered primary test required a gain over both self-prediction and the first-glimpse optimum of more than 0.0025 at both penalties (0.001 and 0.1) in at least 12/16 blocks. All 16 passed (exact one-sided p=1/65536p=1/65536).

Exponential-moving-average (EMA) targets (Appendix O). Each block had six fits with first-glimpse orientation and a constant learning rate of 0.001. We trained with and without the direction loss under two-sided gradients and under EMA targets. The two Self-only controls used a detached online target and an EMA target with a variance penalty, respectively. Some of the eight pilot blocks still had readable second-glimpse evidence after 32,000 updates of EMA self-prediction. We then chose 128,000 and preregistered it before drawing 16 new blocks. We counted paired wins when the added direction loss beat both self-prediction and the first-glimpse optimum by the required margin at both penalties. With EMA targets, 16/16 blocks passed at 128,000 (12 required, exact one-sided p=1/65536p=1/65536). The descriptive paired-win count at 32,000 was 13/16.

We counted Self readouts at the first-glimpse level or worse when their gain over that optimum was below 0.0025 at both penalties. At 8,000 updates all 16 two-sided-gradient controls trained within the EMA study met this condition, versus none of their matched EMA fits. This EMA Self count rose to 11/16 at 32,000, 15/16 at 64,000 and 16/16 at 128,000. For the stop-gradient control (no averaging) the Self count at 128,000 was 13/16. These counts do not isolate averaging as the cause.

Continuous rotation noise (Appendix M). With two-sided gradients and orientation set by the first glimpse, the added loss beat both comparators by more than 25% of the first-glimpse error at all three training-set sizes (8, 32 and 128 shapes) and both penalties in 16/16 new blocks (preregistered, exact one-sided p=1/65536p=1/65536). With orientation set by the current glimpse, 4/16 met the same margins (descriptive).

What fell short (Appendices L–M). The original half-turn study used a constant rate of 0.001. Once the first-glimpse primary test passed, its preregistered nested test required success under both conventions in at least 12 of the same 16 blocks. The joint count was 11/16 at 32,000 updates. The longer replication drew 16 new blocks, distinct from those in the EMA study, and used the same two-sided gradients and constant rate for 128,000 updates. First-glimpse fits alone passed in 16/16 new blocks. Its preregistered joint test reached 10/16. All 16 current-glimpse direction-loss fits beat the first-glimpse optimum by the required margin at both penalties. In the six misses the current-glimpse self-predictive readout still carried enough evidence at the weaker penalty to keep the paired gain below the margin.

With learning-rate decay the joint count was 16/16 (descriptive). We scored the current-glimpse Self readout's advantage over the first-glimpse optimum at the weaker penalty, flooring it at zero. Under decay the advantage fell by more than 1/800 in nine blocks and rose by more than 1/800 in none. The change stayed within the preregistered margin in seven. The preregistered two-tailed test required 14 in either direction and was inconclusive. For hidden pose origins we have a finite example. A corresponding learning experiment and tests on visual benchmarks remain open.

AuthorsOpenReview ↗

Response to Reviewer G9a8 (part 2 of 2)

(b) Actions. Our theorems concern observation histories. In the calibration study actions are in world coordinates. Rotating the state while keeping the action fixed changes their relative angle, so the transition needs a reference for that action. We supplied the reference. Inferring it from actions and their effects would require assumptions on the dynamics and excitation. The two-glimpse task uses no actions.

(c) Targets and the ranking reversal. In the original two-stage study, the transition trained on latent targets and physical labels trained the readout. The later autoregressive study trained the transition on exact physical next-step targets. Under that calibrated recipe, extra relative supervision beat ROOT in every matched block. Point closure concerns equivariant emission. It does not rank these supervision recipes at a fixed training budget.

A practical diagnostic. Hold the first observation fixed, vary a later one that changes the future forecast, and compare a readout of the frozen code with a predictor that uses only the first observation (Appendix L). For pose, test a known-view correction on separate views of the same object. Appendix N reports this measurement for 60 frozen encoders.

Approximate innovation laws. The proof uses an exact conditional joint law and a common deterministic emission map to show that the conditional offset distribution is a point mass. A quantitative extension would need a metric on kernels, control of error through emission and assumptions linking a small distributional error to pose recovery.

JEPA+LIFS. “Contractive MASO-Generalized Predictors for Stable Latent-Space Learning in JEPA” (TMLR, 2026) retains the encoder and objective and replaces the predictor with a recursively applied, spectrally constrained mixture of affine maps. Its contraction result concerns predictor dynamics. Our first witness uses an identity predictor outside a strict contraction class. Extending that witness to strictly contractive predictors is open. A contraction by itself supplies no physical-frame reference.

Heads-up: no revised PDF came through with this message, only the authors' reply. So this is based on the reply text alone. I can't verify the numbers in Appendices L–O, and if the authors change the PDF I'll check them. My assessment from the previous turn still holds.

The reply on substance

•	Persistent-copy half, answered well. The label-free two-glimpse study shows the drift toward the copy: forecast sensitivity falls from 0.18 to 0.001, and an added next-direction loss repairs it. They also report that EMA targets only delay the drift, and that the stop-gradient control reached 13/16.
•	Gauge half, still open. They concede a label-free learning experiment for hidden pose origins is "open", and that the actions are in world coordinates (which is a different mechanism from the theorem's).
•	Results are mixed under the current-glimpse convention. The preregistered joint tests reached 11/16 and 10/16 against the 12/16 required, and continuous noise gave 4/16. The 128,000-update horizon was chosen after pilots.
•	Still unanswered: why ROOT hurts is waved off as "closure doesn't rank recipes". The theorem-side novelty and the SID/REL comparison got no response. There is no evidence beyond toy tasks, and the approximate-kernel extension is deferred even though a short total-variation argument looks possible.
•	Rating: I'd keep it at Weak rejection. If the scale allows, one notch up is defensible for the label-free experiment and the candour.
AuthorsOpenReview ↗

Reply to the follow-up of Reviewer G9a8

Thank you for the follow-up. The revised PDF and supplement, uploaded on 11 November, are the files at the top of this page. Page references:

  • forecast difference between agreeing and opposing glimpses, 0.180 → 0.001, and the primary 16/16 test: Section 7 (p. 8) and Appendix L (pp. 37–40);
  • the EMA study, with 13/16 detached-target Self readouts at the first-glimpse level or worse at 128,000: Appendix O, Table 21 (p. 44);
  • joint counts 11/16 (Appendix L, p. 38) and 10/16 with its six misses (Appendix M, pp. 41–42, Table 19);
  • continuous noise, 16/16 and 4/16: Appendix M (p. 41).

EMA endpoint chosen after pilots. Appendix O ("Why 128,000 updates?") reports this. In a four-block paired pilot at 32,000 updates, two blocks cleared both margins, below the three required to proceed. We then followed the eight blocks of both pilots to 128,000 updates: six passed at 32,000 and all eight at 64,000 and 128,000. We chose 128,000 after seeing these runs. Before the 16 new formal blocks were drawn, we fixed all six checkpoints and designated only 128,000 for the primary test. At 32,000 updates the formal blocks show 13/16 paired wins (descriptive, Table 21).

ROOT in Table 11. ROOT loses in the calibrated autoregressive regime and wins under the original recipe. In the calibrated regime every arm receives the supplied calibration, so the contrast is between encoder supervisions, not between having and lacking alignment. Two measurements point to the frozen encoder's pose output. On the encoders behind Table 11, the same system-identification transition run from each calibrated encoder has H8 NMSE 0.0538 for ROOT and 0.00048 for REL and EXTRA_REL (Table 12, p. 31). On the Appendix F encoders, which use the same architecture and 1,600-update recipe, one calibration view leaves an angular error of 0.0735 rad for ROOT, 0.0741 for the shuffled-root control and 0.0078 for REL (Table 20, p. 43). Offsets vary more across views of an object for ROOT than for REL (concentration 0.99689 against 0.99998), consistent with one calibration transferring less accurately. These experiments do not isolate why root supervision has this effect.

Theorem-side novelty. The factorization and conditional-expectation steps are standard (Bloem-Reddy and Teh, 2020, Lemma 8; Section 4 and Appendix G). Theorem 3.1: a persistent copy can remain an exact optimum when a strictly proper quotient anchor is added to equivariant self-prediction, and it merges histories with different futures; the persistent-copy shortcut of Sobal et al. (2022) has no such anchor. Theorem 3.2: a hidden trajectory gauge survives canonical innovation, hard transport and a supplied reconstructor. Theorem 4.2: with the joint innovation law, deterministic equivariant emission exists if and only if the invariant code determines the offset; Table 1 shows that the two exact marginals without their coupling do not force this.

SID against learned heads. The observed-moment comparator has the correct six-feature structure and physical labels, and it is more accurate than every learned head (H8 NMSE 0.00038 against at best 0.012; Table 12). We report it as the reference for a correct structural prior; the learned heads are far from it in this world.

Beyond synthetic tasks. All our learning studies use synthetic data. Section 9 states that transfer to real data requires separate evidence.

Approximate kernels. Your total-variation suggestion works; for exact offset recovery it needs less than our 11 November reply said. Keep Definition 4.1 except for exactness: take any Borel kernel KK and any equivariant Borel emitter E\mathcal E, with ρ\rho built from E\mathcal E as in Appendix B and D(v)D(v) the law of ρ(v,J)\rho(v,J) for J∼K(v;⋅)J\sim K(v;\cdot). At a governed time let ε=E∥law(Jt+1∣Ft)−K(Vt;⋅)∥TV\varepsilon=\mathbb{E}\Vert\mathrm{law}(J_{t+1}\mid\mathcal F_t)-K(V_t;\cdot)\Vert_{\mathrm{TV}} and α=P(Ot+1≠E((Ut,Vt),Jt+1))\alpha=P(O_{t+1}\neq\mathcal E((U_t,V_t),J_{t+1})). Then P(Ct≠χ(Vt))≤2(α+ε)P(C_t\neq\chi(V_t))\le 2(\alpha+\varepsilon), where χ(v)\chi(v) is the atom of D(v)D(v) with mass above 1/21/2 (the identity if there is none; Borel by Lusin–Souslin). This χ\chi depends only on KK and E\mathcal E, so it is the same at every governed time. Proof: by the identities in Appendix B, Ct=ρ(Vt,Jt+1)C_t=\rho(V_t,J_{t+1}) almost surely on the event that emission succeeds, so P(Ct=ρ(Vt,Jt+1))≥1−αP(C_t=\rho(V_t,J_{t+1}))\ge 1-\alpha. CtC_t and VtV_t are Ft\mathcal F_t-measurable, so replacing the true innovation law by KK changes this probability by at most ε\varepsilon; hence E[D(Vt)({Ct})]≥1−α−ε\mathbb{E}[D(V_t)(\lbrace C_t\rbrace)]\ge 1-\alpha-\varepsilon, and Markov's inequality gives the bound. Conversely, for any Borel χ\chi the emitter built from χ\chi in Appendix B fails with probability at most P(Ct≠χ(Vt))P(C_t\neq\chi(V_t)). At α=ε=0\alpha=\varepsilon=0 the first statement is the forward direction of Theorem 4.2; the second, with P(Ct≠χ(Vt))=0P(C_t\neq\chi(V_t))=0, is its converse. Both count exact events: a small pose error is a failure.

Reviewer_DYdWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper seems to study the gap between equivariant representations and their prediction power.

Critical strengths

None

Critical weaknesses
  • This is clearly a low-quality LLM-generated paper with few human involvement. I stopped reading at 3 page followed by a quick overall glance. Some signs are:
    • Unexplained jargon chains in Abstract like: "quotient-/full-future-law sufficiency", "physical calibration and known association", "Readout calibration," "root supervision".
    • Super thin Introduction, only 3 very short paragraphs. And still rough, unclear sentences like: "Our solution concept is the exact global minimum of the declared forward scalar objective, not stationary points or arbitrary training dynamics." What is "declared forward scalar objective"?
    • Some highly specialized, unexplained, and usually unnecessary math concepts: free orbit (in Introduction), locally compact second-countable group (at the beginning of SETUP). To be clear, I know those math to a large extent, but find them super confusing to appear in the current paper.
    • Misplaced and inadequate definitions: Figure 1 cannot be understood (U,VU, V and F+ class appear before definition). What is the difference between latent code UU and VV? What is ZinvZ_{inv}?
Questions for authors

Maybe the authors can at least confirm whether my one-sentence summary of paper is roughly correct. Also, what does the "Equivariance Transports, Closure Completes" mean in the title?

1 comment
AuthorsOpenReview ↗

Response to Reviewer DYdW

Yes. Your summary identifies the question we intended to ask. Consider a hidden arrow equally likely to point left or right. Each noisy report gives its true direction with probability 3/4. After two rightward reports, the next report points right with probability 0.7. After a rightward report and a leftward one, that probability is 0.5. An encoder can remember the first report forever, reverse its code whenever all reports reverse, and predict its own next code perfectly. It gives those two histories the same forecast even though the second report matters.

This is what the title means. Equivariance tells us how the code moves when the observations move. A prediction target selects what the code must retain. Closure answers a further question: can one recover the pose offset needed to turn a predicted change into an observation? Under the assumptions stated in the paper, that alignment, together with a recursive update and the observation law, gives a forecast of the full sequence. We now open with the arrow, define the pose and invariant coordinates before Figure 1, and pair the general notation with the arrow example in §2.

We disclosed AI use in the submission form and Appendix J. We made substantial use of AI tools to refine hypotheses and experimental designs, develop mathematical arguments, implement data generators and methods, interpret results, draft the manuscript and verification code, and review the work for consistency. We take responsibility for the paper, including every AI-assisted part. In the supplement, checks/verify_arrow.py enumerates all 16 models in the specified one-transition class and returns the exact next-report probabilities. For the first two-glimpse study (Appendix L), checks/reeval.py regenerates the evaluation panels from the recorded seeds, re-encodes the panels using the frozen weights and refits the readouts. For the moving-average target study (Appendix O), checks/tier3/replay_endpoint.py reconstructs the evaluation panels and reproduces the preregistered 16/16 result from all 96 final online weights. Both replays refit the readouts without retraining any encoder.

Reviewer_7E9gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This theory focused paper mainly addresses the question of whether a model trained to predict latent representations (like world models) is able to necessarily preserver the information that it needs to predict future representations, especially once you consider that information is severely compressed into a lower dimensional state. The paper mainly focuses on the rotational aspect of this question, defining a representation to respond consistently of the world is rotated/otherwise transformed. The key insight that this submission provides is that even idealized and perfect training struggle to guarantee this exact characteristic.

The paper used two specific toy examples to demonstrate the main point of contention:

  1. One representation is able to remember the first noisy input while ignoring evidence that is provided to it at a later stage;
  2. Another representation that is able to track the changing motion but uses a hidden orientation offset;

showing that different historic observations can share on representation even it they would have produces significantly different future predictions.

To address this problem the authors find conditions where removing this missing rotational alignment makes the representation sufficient for predicting futures, where their strongest equivariance applies when reconstructing next observations using actual changes in the next step. Using simulated rotating objects and ideal pendulum the authors try to validate their theory on toy models. Using calibrations inside the dynamics paired with multi-step prediction , the model is able to reduce prediction error significantly. IMO, the contribution of the submission is a precise account of missing information in latent world models and conditional remedies under specific assumptions, with practical benefits established only in controlled/synthetic/toy settings.

Critical strengths
  1. Th paper show cases a situation where the training objective fails even if the optimization is perfectly executed. The toy counterexamples presented in the paper are able to satisfy the declared objectives exactly while merging histories with different future predictions. Furthermore these examples are not trivial such as constant inputs, making the results consequantial as it showcases that missing information can be a structural problem within the objective, rather than insufficient data or non-converging training runs.

  2. The paper further proposes a precise fix for the problem that it identified. The authors further characterizes when such alignment is necessary for the specified task, and supplies examples showing why key assumptions matter, and separates that task from predicting future probabilities more generally. This allows readers to not only understand the failure but also have a principled approach in resolving such insufficiency.

Critical weaknesses
  1. The theoretical work that this paper presents requires significant and strong assumptions, so it is not entirely clear to me what the implications for actual trained models would be. Specifically, as I understand, the strongest equivalence requires an unambiguous orientation, a transformation-respecting deterministic reconstruction, and an exact joint prediction model. Can the authors expand on the gap between the theoretical guarantee and how this would be implemented for an actual model?

  2. I think the paper is missing a little bit of discussion and motivation on how much of this failure mode actually occurs in real world models, since the given examples are toy and synthetic in nature.

Questions for authors
  1. Does existing, and which if so, world models satisfy theorem 3.2? This is loosely related to the weakness in that there should be more link between the failure mode identified and real world models.

  2. Do any experiments measure whether the residual gauge C_t is recoverable from the invariant code of the learned models, as Theorem 5.2 requires? If not can the authors suggest potential experiments to address this?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7E9g

We do not know a deployed world model for which the assumptions of Theorem 3.2 have all been verified. In our point-cloud encoder the code determines the learned offset from a direction visible in the cloud. A first view of a new object leaves its orientation in the world unknown.

Which offset can the code recover? The calibration-study encoder learns a rotation relative to the observed moment direction m(x)m(x). Its orientation output is U(x)=m(x)exp⁡(if(d(x)))U(x)=m(x)\exp(i f(d(x))), so the invariant descriptor d(x)d(x) determines the offset from that observed direction by construction (Appendix F). For the physical reference, Theorem H.1(1) assumes a fresh object drawn from the stated rotation-invariant population. Its initial physical pose Q0Q_0 is uniform on the circle even conditional on the initial cloud X0X_0 and the given actions. Once X0X_0 is fixed, the frozen output U(X0)U(X_0) is fixed too. Thus the physical offset Q0‾U(X0)\overline{Q_0}U(X_0) is still uniform and cannot be a function of the descriptor of that cloud.

A known-orientation view supplies the missing reference. We used one such view of a held-out object and tested on separate trajectories of the same object. Across 60 frozen models (20 per method), mean angular errors after calibration were 0.0078, 0.0735 and 0.0741 radians for REL, ROOT and SHUFFLED. With four views the errors were 0.0062, 0.0577 and 0.0569. REL's lower error is consistent with its offsets varying less across noisy views of the same object (Appendix N, Table 20). The mathematical statement above concerns the first uncalibrated view of a randomly drawn object. Table 20 measures how well supplied calibration transfers across views.

How can one test whether a learned code uses a later observation? In the paired diagnostic of Appendix L, hold the first glimpse fixed and change the second in a way that changes the optimal forecast of the third glimpse's direction. Freeze the encoder and compare its forecast with the best forecast using just the first glimpse. This test needs no physical pose labels. In the two-glimpse experiments, both encoders train to predict their next code and the next cloud in its own observed frame. Only one also trains to predict the next observed direction.

With orientation set by the first glimpse, adding that direction loss beat self-prediction and the best first-glimpse forecast at both readout penalties in 16/16 blocks in the preregistered first-glimpse test of each of three studies. Two used gradients through both sides of the latent loss, one with half-turn (reversal) noise and one with continuous rotation noise at three training-set sizes. The third used reversal noise and a moving-average target encoder, with orientation set only by the first glimpse. The original reversal study and its longer replication used two-sided gradients and a constant learning rate. The first-glimpse fits passed in all 16 blocks of each study. Their stricter preregistered joint test required success under both conventions in the same block. It reached 11/16 at 32,000 updates and 10/16 in the new blocks at 128,000 (12 required). In the misses, the current-glimpse self-predictive readout still used later evidence. Under continuous noise, the current-glimpse count was 4/16 by the same margins used for that study's first-glimpse fits (descriptive). Our reply to Reviewer G9a8 (part 1) and Appendices L–O give the thresholds and details.

Thanks for the clarifcations and the additional experiments. The observation-only studies do strengthen the evidence that self-training can lose sensitivty to useful later observations, so that part is more convincing to me now. That said, I still don’t think the rebuttal really establishes whether the hidden gauge obstruction is a meaningful limitation of existing world models in practice. The distinction between observable and physical offsets is helpful, but Table 20 is evaluating externally supplied calibration rather than whether the physical offset can actually be recovered from the the invariant code alone.

So overall I’d say my concerns are partially addressed. The empirical support has improved, but the connection between the exact theoretcal guarantees and practical learned models still feels fairly limited to me.

AuthorsOpenReview ↗

Reply to the follow-up of Reviewer 7E9g

Thank you for the follow-up, and for the remark on Table 20.

Existing world models. We agree: the paper does not show whether existing world models are limited in this way. All our learning studies are synthetic, and Section 9 states that transfer to real data requires separate evidence. The calibration studies measure the value of calibration; they do not test the theorems' hypotheses (Section 7). In a new synthetic moving-arrow study run on 17 November, none of 16 trained encoders met our diagnostic for a trajectory-specific hidden origin (see below).

Recovery from the code alone. You are right that Table 20 does not test it. It concerns the physical offset; the next paragraph says how that differs from the offset in Theorems 4.2 and 5.2. Under the exact-real fresh-object law of Appendix H.5, which idealizes the calibration-study generator, objects come from a rotation-invariant population, and the physical orientation of a new object is independent of its first cloud (Theorem H.1(1), p. 32). Hence no frozen encoder can recover the physical offset from that view: given anything computed from the cloud, including the invariant descriptor, the offset is uniform on the circle, and a readout of the descriptor fitted on other objects is a blind guess with mean angular error π/2≈1.57\pi/2\approx 1.57 rad. The "No calibration" column of Table 20 (p. 43) is consistent with this: the uncorrected output of the single-frame encoders has errors of 1.50, 1.58 and 1.65 rad for REL, ROOT and SHUFFLED. It is a marginal check, not a readout from the code. The "One view" and "Four views" columns measure the remedy, known views of the same object.

The offset in Theorems 4.2 and 5.2. The theorems' offset is measured from a pose the model can see; Table 20's is measured from an orientation that no first view reveals. Theorem 5.2 assumes point gauge closure, and Theorem 4.2 shows it equivalent to deterministic equivariant emission in the class of Definition 4.1. Closure concerns the offset of the learned pose from a reference pose available from the history (notation table, p. 2; Definition 2.1, p. 4). Definition 4.1 reads that reference off the current observation. In the finite world of Theorem 3.2 the observation shows the pose itself (the "physical pose" of p. 5). In the calibration-study encoders the reference is the observed moment direction, and the architecture makes the offset from it a function of the descriptor, frame by frame (Appendix N). The true orientation qq of Appendix N, against which the physical offset is measured, is not such a reference for a first view. So the physical offset above is information missing from a first view (Section 7; Appendix H.5), not the hidden gauge of Theorem 3.2. The architectural identity alone does not verify the theorems' other assumptions.

Exact guarantees and learned models. We agree that this link is limited: Section 9 states that the theory concerns exact global optima, not high-dimensional optimization. Our reply to the follow-up of Reviewer G9a8 takes one step toward inexact models: in Definition 4.1 without exactness, it bounds the probability that the offset is not exactly recovered when the joint innovation kernel is inexact and the equivariant emitter sometimes fails. It counts exact events, so a small pose error counts as a failure.

A test of the kind your second question asks for. In the 17 November study the criteria were fixed in a locally hash-locked protocol before data generation. Seeds were reduced modulo 2632^{63}, not the specified 263−12^{63}-1. The recurrent encoder has separate phase and invariant outputs. Its pose output is the observed arrow direction turned by the learned phase, and it is trained to track observed turns, without pose labels. The task permits a trajectory-specific origin but does not require the encoder to adopt one. We counted a block as positive when three conditions held together on its test trajectories: (1) the pose output tracks the observed turns; (2) the offset from the observed direction is stable within a trajectory, differs across trajectories and is poorly recovered by two fixed readouts of the invariant code; (3) correcting the origin with the trajectory's own first frame improves a frozen forecast by a set margin over no correction, one shared correction and another trajectory's correction. All 16 trained encoders tracked the turns, and none met the other two conditions (12 of 16 were required). In every block the offset was nearly shared across trajectories (dispersion below 0.0003; cutoff 0.10). Both readouts predicted it (test MSE below 1e-5), and the mean gain over a shared correction was below 0.0003 (margin 0.10). The result is specific to this task, architecture, objective and 1,200-update budget. It is not in the PDF or the supplement. We have not run it on the two-glimpse encoders of Appendix L, which also have separate phase and invariant outputs (p. 37), or on an existing world model.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

To clarify the theory, we open with a hidden arrow seen through noise. Section 2 links it to the notation, and Figure 1 gives the assumptions. Section 7 adds experiments trained on point clouds alone, with no physical pose labels.

Both encoders train to predict their next code and the next cloud in its own observed frame. Under half-turn (reversal) noise, with first-glimpse orientation and two-sided gradients, self-predictive readouts moved toward the first-glimpse optimum in all 16 blocks. Adding a loss for the next observed direction beat self-prediction and that optimum. The preregistered first-glimpse counts were 16/16 under half-turn noise and 16/16 under continuous noise with two-sided gradients at three training-set sizes.

The moving-average target study used only first-glimpse orientation. Self-predictive readouts lost sensitivity more slowly than in matched two-sided-gradient fits. After pilots we fixed 128,000 updates before drawing 16 new blocks. All 16 passed the preregistered first-glimpse test.

Only the original half-turn study with two-sided gradients and its longer replication preregistered the stricter joint test requiring success under both orientations in the same blocks. They reached 11/16 at 32,000 updates and 10/16 at 128,000 (12 required). In the misses, current-glimpse self-prediction retained readable evidence. With current-glimpse orientation under continuous noise, 4/16 met that study's first-glimpse margins (descriptive). The learning-rate decay test was inconclusive.

Appendices L–O cover protocols, 60-model calibration and longer histories. The supplement replays the arrow and scores. Appendix J and the submission form disclose AI use.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.