- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a fundamental question in equivariant latent world models: whether an exact optimum of an equivariant latent-prediction objective must be a predictive state or sufficient for future prediction. The authors argue that equivariance alone does not guarantee predictive sufficiency because of a residual latent gauge ambiguity. Two finite counterexamples are constructed showing that exact optimization of common latent-prediction objectives can still fail to produce predictive states (a persistent covariant copy, and a hidden "trajectory gauge" that survives canonical innovation, hard transport and a supplied reconstructor). The paper then introduces a formal notion of point gauge closure and proves an exact equivalence between gauge closure and deterministic equivariant emission within a specified class of latent world models. Additional theoretical results separate quotient-future sufficiency from full-future sufficiency and characterize interfaces that can restore predictive closure. Extensive synthetic experiments investigate the practical consequences of calibration, routing, synchronization, and latent alignment in learned predictors. Theorems 5.1 and 5.2 give conditional sufficiency results. Experiments on a synthetic SO(2) point-cloud world and a sensor-mounting Pendulum test supplied calibration.
Critical strengths
- Deep and original theoretical contribution.
The paper identifies a subtle limitation of equivariant latent world models that appears largely overlooked in prior JEPA and equivariant-representation literature. The notion of residual gauge ambiguity and the distinction between transport and predictive closure provide a new conceptual framework for analyzing latent sufficiency. The finite counterexamples and necessity results are mathematically interesting and significantly advance understanding of representation identifiability in world models.
- Strong formal development with precise assumptions.
Unlike many theoretical works that rely on informal reasoning, this paper carefully states assumptions, contracts, measurable-space requirements, and sufficiency conditions. The distinction between quotient-future prediction and full-future prediction is clearly formalized, and several nontrivial results are proved rigorously.
- Thm H.1 gives a clean reason why orientation information is needed. Without it, risk is ≥ 0.737 at H8, against 0.1275 achieved with calibration.
Critical weaknesses
- Excessive complexity and limited accessibility.
The presentation is extremely dense. The paper introduces a large number of concepts, assumptions, interfaces, contracts, gauges, kernels, and auxiliary definitions before providing intuition. Even readers familiar with equivariant representation learning and world models will likely require substantial effort to follow the main argument. This significantly reduces the practical impact of otherwise interesting theoretical contributions.
- Limited empirical relevance to modern world-model benchmarks.
Although the paper includes many controlled synthetic studies, the experiments do not convincingly demonstrate that the identified gauge obstructions occur in widely used large-scale world-model architectures (e.g., JEPA, V-JEPA, TD-JEPA, Dreamer-like systems, or transformer-based world models). Consequently, the practical significance of the theoretical findings remains somewhat uncertain.
- The theory–experiment link is weak.
-
The theory covers autonomous equivariant worlds with exact contracts. No experiment measures the innovation kernel, emission, point closure or k_inv (Sec. 7 admits this).
-
The testbed has global-frame actions with q⁺ = q·exp(i f(qa)). That is not equivariant under rotating the state alone, and the gauge matters there because actions are in the world frame. That is a different mechanism from the theorem's.
-
The AR transition is trained on exact physical next-step targets (H.2, H.3), not a latent-prediction loss. The positive result is therefore supervised dynamics in a calibrated frame, not JEPA-style latent prediction.
Questions for authors
1.The paper focuses on exact global minima and exact interface contracts. To what extent do the identified gauge obstructions appear in practical neural-network training regimes where optimization is approximate and finite-data effects dominate?
2.Can the authors provide experimental evidence on contemporary visual world-model benchmarks demonstrating that the proposed gauge ambiguities materially affect downstream prediction or planning performance?
3.Several recent works study latent identifiability and factorized JEPA representations. Recent work such as JEPA+LIFS [1] explicitly promotes structured and identifiable latent representations. Can the authors discuss whether the gauge obstruction identified in this paper persists under JEPA+LIFS, and ideally provide a quantitative comparison? Such a comparison would help determine whether gauge closure addresses a limitation that remains unresolved by existing latent-factorization approaches or whether related mechanisms already mitigate the issue in practice. [1]Belloulata, 'Contractive MASO-Generalized Predictors for Stable Latent-Space Learning in JEPA, in Transactions on Machine Learning Research (TMLR), 2026, https://openreview.net/forum?id=k2Z2gPOtlq.
4.The paper distinguishes quotient-future sufficiency from full-future sufficiency. Can the authors provide a practical diagnostic that would allow practitioners to determine empirically whether a learned representation suffers from the predicted gauge obstruction?
- Does a standard latent-prediction (JEPA-style) equivariant world model, trained without physical-label targets, show the gauge or copy failure, and does closure fix it? If closure is the missing piece, why does ROOT hurt in Table 11? What is the empirical C_t? How does the SO(2) study change if observation-plus-action history is allowed? Does Thm 4.2 survive if K(V_t;·) is only approximately V-measurable?
What would move this up: a non-toy, self-supervised demonstration with the gauge measured directly, scaled object diversity, and a main text that a non-specialist can read.
4 comments
Response to Reviewer G9a8 (part 2 of 2)
(b) Actions. Our theorems concern observation histories. In the calibration study actions are in world coordinates. Rotating the state while keeping the action fixed changes their relative angle, so the transition needs a reference for that action. We supplied the reference. Inferring it from actions and their effects would require assumptions on the dynamics and excitation. The two-glimpse task uses no actions.
(c) Targets and the ranking reversal. In the original two-stage study, the transition trained on latent targets and physical labels trained the readout. The later autoregressive study trained the transition on exact physical next-step targets. Under that calibrated recipe, extra relative supervision beat ROOT in every matched block. Point closure concerns equivariant emission. It does not rank these supervision recipes at a fixed training budget.
A practical diagnostic. Hold the first observation fixed, vary a later one that changes the future forecast, and compare a readout of the frozen code with a predictor that uses only the first observation (Appendix L). For pose, test a known-view correction on separate views of the same object. Appendix N reports this measurement for 60 frozen encoders.
Approximate innovation laws. The proof uses an exact conditional joint law and a common deterministic emission map to show that the conditional offset distribution is a point mass. A quantitative extension would need a metric on kernels, control of error through emission and assumptions linking a small distributional error to pose recovery.
JEPA+LIFS. “Contractive MASO-Generalized Predictors for Stable Latent-Space Learning in JEPA” (TMLR, 2026) retains the encoder and objective and replaces the predictor with a recursively applied, spectrally constrained mixture of affine maps. Its contraction result concerns predictor dynamics. Our first witness uses an identity predictor outside a strict contraction class. Extending that witness to strictly contractive predictors is open. A contraction by itself supplies no physical-frame reference.
Heads-up: no revised PDF came through with this message, only the authors' reply. So this is based on the reply text alone. I can't verify the numbers in Appendices L–O, and if the authors change the PDF I'll check them. My assessment from the previous turn still holds.
The reply on substance
• Persistent-copy half, answered well. The label-free two-glimpse study shows the drift toward the copy: forecast sensitivity falls from 0.18 to 0.001, and an added next-direction loss repairs it. They also report that EMA targets only delay the drift, and that the stop-gradient control reached 13/16.
• Gauge half, still open. They concede a label-free learning experiment for hidden pose origins is "open", and that the actions are in world coordinates (which is a different mechanism from the theorem's).
• Results are mixed under the current-glimpse convention. The preregistered joint tests reached 11/16 and 10/16 against the 12/16 required, and continuous noise gave 4/16. The 128,000-update horizon was chosen after pilots.
• Still unanswered: why ROOT hurts is waved off as "closure doesn't rank recipes". The theorem-side novelty and the SID/REL comparison got no response. There is no evidence beyond toy tasks, and the approximate-kernel extension is deferred even though a short total-variation argument looks possible.
• Rating: I'd keep it at Weak rejection. If the scale allows, one notch up is defensible for the label-free experiment and the candour.
Reply to the follow-up of Reviewer G9a8
Thank you for the follow-up. The revised PDF and supplement, uploaded on 11 November, are the files at the top of this page. Page references:
- forecast difference between agreeing and opposing glimpses, 0.180 → 0.001, and the primary 16/16 test: Section 7 (p. 8) and Appendix L (pp. 37–40);
- the EMA study, with 13/16 detached-target Self readouts at the first-glimpse level or worse at 128,000: Appendix O, Table 21 (p. 44);
- joint counts 11/16 (Appendix L, p. 38) and 10/16 with its six misses (Appendix M, pp. 41–42, Table 19);
- continuous noise, 16/16 and 4/16: Appendix M (p. 41).
EMA endpoint chosen after pilots. Appendix O ("Why 128,000 updates?") reports this. In a four-block paired pilot at 32,000 updates, two blocks cleared both margins, below the three required to proceed. We then followed the eight blocks of both pilots to 128,000 updates: six passed at 32,000 and all eight at 64,000 and 128,000. We chose 128,000 after seeing these runs. Before the 16 new formal blocks were drawn, we fixed all six checkpoints and designated only 128,000 for the primary test. At 32,000 updates the formal blocks show 13/16 paired wins (descriptive, Table 21).
ROOT in Table 11. ROOT loses in the calibrated autoregressive regime and wins under the original recipe. In the calibrated regime every arm receives the supplied calibration, so the contrast is between encoder supervisions, not between having and lacking alignment. Two measurements point to the frozen encoder's pose output. On the encoders behind Table 11, the same system-identification transition run from each calibrated encoder has H8 NMSE 0.0538 for ROOT and 0.00048 for REL and EXTRA_REL (Table 12, p. 31). On the Appendix F encoders, which use the same architecture and 1,600-update recipe, one calibration view leaves an angular error of 0.0735 rad for ROOT, 0.0741 for the shuffled-root control and 0.0078 for REL (Table 20, p. 43). Offsets vary more across views of an object for ROOT than for REL (concentration 0.99689 against 0.99998), consistent with one calibration transferring less accurately. These experiments do not isolate why root supervision has this effect.
Theorem-side novelty. The factorization and conditional-expectation steps are standard (Bloem-Reddy and Teh, 2020, Lemma 8; Section 4 and Appendix G). Theorem 3.1: a persistent copy can remain an exact optimum when a strictly proper quotient anchor is added to equivariant self-prediction, and it merges histories with different futures; the persistent-copy shortcut of Sobal et al. (2022) has no such anchor. Theorem 3.2: a hidden trajectory gauge survives canonical innovation, hard transport and a supplied reconstructor. Theorem 4.2: with the joint innovation law, deterministic equivariant emission exists if and only if the invariant code determines the offset; Table 1 shows that the two exact marginals without their coupling do not force this.
SID against learned heads. The observed-moment comparator has the correct six-feature structure and physical labels, and it is more accurate than every learned head (H8 NMSE 0.00038 against at best 0.012; Table 12). We report it as the reference for a correct structural prior; the learned heads are far from it in this world.
Beyond synthetic tasks. All our learning studies use synthetic data. Section 9 states that transfer to real data requires separate evidence.
Approximate kernels. Your total-variation suggestion works; for exact offset recovery it needs less than our 11 November reply said. Keep Definition 4.1 except for exactness: take any Borel kernel and any equivariant Borel emitter , with built from as in Appendix B and the law of for . At a governed time let and . Then , where is the atom of with mass above (the identity if there is none; Borel by Lusin–Souslin). This depends only on and , so it is the same at every governed time. Proof: by the identities in Appendix B, almost surely on the event that emission succeeds, so . and are -measurable, so replacing the true innovation law by changes this probability by at most ; hence , and Markov's inequality gives the bound. Conversely, for any Borel the emitter built from in Appendix B fails with probability at most . At the first statement is the forward direction of Theorem 4.2; the second, with , is its converse. Both count exact events: a small pose error is a failure.
Response to Reviewer G9a8 (part 1 of 2)
(a) We trained on point clouds alone. No physical pose label entered any loss. With first-glimpse orientation, self-predictive forecasts became less sensitive to evidence in the second glimpse. Adding a loss for the next observed direction kept that evidence readable, including with a moving-average target encoder.
Task. Each glimpse can be reversed by a half turn with probability 1/4. Both encoders train to predict their next code and the next cloud rotated to align its observed direction with a fixed axis. This removes its overall orientation. One also predicts the next observed direction in the original observation coordinates. Under the two conventions the encoder applies a learned rotation to either the first or the current observed direction. We score frozen readouts identically under both conventions. Readout error is one quarter of the mean squared distance from its forecast to the conditional mean of the third observed direction given the first two.
Two-sided gradients (Appendix L). The first glimpse set orientation. We tracked the readouts at the weaker ridge penalty (0.001). From 500 to 32,000 updates the median self-predictive readout error rose from 0.0010 to 0.0099 (first-glimpse optimum 0.009375). The median forecast difference between agreeing and opposing glimpses fell from 0.180 to 0.001 (optimal 0.2). At this penalty, error rose and the difference fell in every block. With the added direction loss the final median difference was 0.201 and the median error was 0.0008. The preregistered primary test required a gain over both self-prediction and the first-glimpse optimum of more than 0.0025 at both penalties (0.001 and 0.1) in at least 12/16 blocks. All 16 passed (exact one-sided p=1/65536).
Exponential-moving-average (EMA) targets (Appendix O). Each block had six fits with first-glimpse orientation and a constant learning rate of 0.001. We trained with and without the direction loss under two-sided gradients and under EMA targets. The two Self-only controls used a detached online target and an EMA target with a variance penalty, respectively. Some of the eight pilot blocks still had readable second-glimpse evidence after 32,000 updates of EMA self-prediction. We then chose 128,000 and preregistered it before drawing 16 new blocks. We counted paired wins when the added direction loss beat both self-prediction and the first-glimpse optimum by the required margin at both penalties. With EMA targets, 16/16 blocks passed at 128,000 (12 required, exact one-sided p=1/65536). The descriptive paired-win count at 32,000 was 13/16.
We counted Self readouts at the first-glimpse level or worse when their gain over that optimum was below 0.0025 at both penalties. At 8,000 updates all 16 two-sided-gradient controls trained within the EMA study met this condition, versus none of their matched EMA fits. This EMA Self count rose to 11/16 at 32,000, 15/16 at 64,000 and 16/16 at 128,000. For the stop-gradient control (no averaging) the Self count at 128,000 was 13/16. These counts do not isolate averaging as the cause.
Continuous rotation noise (Appendix M). With two-sided gradients and orientation set by the first glimpse, the added loss beat both comparators by more than 25% of the first-glimpse error at all three training-set sizes (8, 32 and 128 shapes) and both penalties in 16/16 new blocks (preregistered, exact one-sided p=1/65536). With orientation set by the current glimpse, 4/16 met the same margins (descriptive).
What fell short (Appendices L–M). The original half-turn study used a constant rate of 0.001. Once the first-glimpse primary test passed, its preregistered nested test required success under both conventions in at least 12 of the same 16 blocks. The joint count was 11/16 at 32,000 updates. The longer replication drew 16 new blocks, distinct from those in the EMA study, and used the same two-sided gradients and constant rate for 128,000 updates. First-glimpse fits alone passed in 16/16 new blocks. Its preregistered joint test reached 10/16. All 16 current-glimpse direction-loss fits beat the first-glimpse optimum by the required margin at both penalties. In the six misses the current-glimpse self-predictive readout still carried enough evidence at the weaker penalty to keep the paired gain below the margin.
With learning-rate decay the joint count was 16/16 (descriptive). We scored the current-glimpse Self readout's advantage over the first-glimpse optimum at the weaker penalty, flooring it at zero. Under decay the advantage fell by more than 1/800 in nine blocks and rose by more than 1/800 in none. The change stayed within the preregistered margin in seven. The preregistered two-tailed test required 14 in either direction and was inconclusive. For hidden pose origins we have a finite example. A corresponding learning experiment and tests on visual benchmarks remain open.