- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies representation selection in capacity-limited LeJEPA and shows that Gaussianity still permits latent source reuse. It introduces a transition-closure constraint and establishes exact linear recovery and stable near linearity under the stated assumptions, supported by controlled experiments.
Critical strengths
The paper addresses a meaningful question and proposes a well-motivated transition-closure constraint. The theoretical results on exact linear recovery and stable near linearity are valuable, and the experiments provide targeted evidence for the mechanisms identified by the theory.
Critical weaknesses
The connection between the theoretical conditions and the implemented training procedure is not sufficiently validated. The recovery guarantees require control of Gaussianity error and transition-closure error, together with a uniform predictability floor. However, the implementation uses soft Gaussian regularization and a finite critic, whose small objective does not necessarily imply a small true closure defect. Although the experiments demonstrate recovery in some settings, they do not sufficiently examine its sensitivity to critic capacity and optimization budget or explain the settings where recovery fails. Consequently, the reliability of the proposed correction remains insufficiently supported.
If the authors can adequately address these validation concerns in the rebuttal, I would consider raising my score to 3.
By the way, the fonts in some figures are excessively large. Reducing their size would improve the presentation. This is only feedback from reading the paper and does not affect my score.
Questions for authors
- Which regimes of the original Gaussian-constrained objective are fully characterized, and what remains open under unequal transition rates?
- The synthetic CSV contains negative
closure_defectvalues despite the nonnegative clamp inzcm_rf, as well as fields absent from the current output code. Which code version and post-processing procedure generated these results?
5 comments
Scope of the Gaussian-Constrained Objective and Code Provenance (Q1–Q2)
Q1. Characterization of the Gaussian-Constrained Objective
We organize the regimes in which the Gaussian-constrained objective is fully characterized into three levels.
-
For the whitening-only problem, which requires only zero mean and identity covariance, we have fully characterized top-m selection through the complete Hermite spectrum. The selected modes may be the latent variables themselves or the squares and higher-order functions of the same slow latent variable. This level is fully characterized, including cases with tied eigenvalues.
-
Under the exact Gaussian constraint, we have a complete characterization for equal latent transition rates, representation dimension , and a heterogeneous-rate regime in which the linear factors are sufficiently dominant. Ordering the rates as , this regime satisfies
In other words, the weakest retained linear factor is at least as predictable as both any omitted linear factor and the strongest competing second- or higher-order mode. In these cases, any Gaussian representation attaining the whitening spectral optimum must be linear, and the remaining ambiguity comes only from ties among linear transition rates.
The problem becomes substantially harder when is smaller than higher-order modes of a slow latent variable. Completing this broader characterization would require establishing global optimality for arbitrary measurable Gaussian representations or constructing the full family of optimizers. The existing source-reuse construction provides a counterexample showing that a leading linear representation is not optimal in some cases, but it does not fully characterize the optimizer family or provide an exact decomposition throughout this entire difficult regime.
- When Gaussianity and exact transition closure hold simultaneously, we characterize the feasible representations as linear -invariant subspaces. Under a strict spectral cutoff, alignment further selects the leading linear subspace. This is the central result of our proposed correction and does not depend on first solving for all optimal representations without closure in the general unresolved regime.
We will clearly distinguish the regimes that have been resolved from those that remain open, making the scope of the theoretical results more explicit.
Q2. CSV Fields and Code-Version Consistency
Thank you for checking this carefully. After tracing the discrepancy, we confirmed that it arose from a version mismatch introduced during code cleanup.
The closure_defect field in the original CSV corresponds to zcm_eps_norm in the original results. It is a normalized, signed latent-Hermite held-out dual and can therefore take negative values. During cleanup of the submitted code for anonymous review, the same-named field was connected instead to the observation random-feature diagnostic after nonnegative clamping.
Our item-by-item check confirms that all results reported in the paper are consistent with the original experimental records. We will carefully correct the code to match the original experimental implementation.
Thank you for the additional experiments. They substantially address my concern about the absence of critic-capacity and optimization-budget studies. My original review raised the question of how sensitive recovery is to the critic’s optimization budget. The additional results in your response to Reviewer PZ8g directly bear on this concern. At the same critic width, increasing the number of updates per step from 10 to 20 or 40 substantially worsens both target recovery and the independently estimated closure error. Could you explain this deterioration and clarify how the update budget should be chosen in practice?
On a separate note, I was disappointed that the authors did not respond at all to my earlier comment about the oversized fonts in some figures, even though this issue still does not affect my rating.
Empirical Validation of Theoretical Conditions (W1)
W1. Connecting Theoretical Conditions to the Training Procedure
Thank you for the suggestion. We have added experiments for independent measurement of the theoretical conditions, comparisons of critic capacity and optimization budget, and analysis of failure cases.
In this longer training run, both the target recovery error and the independently measured closure error improve, while Δ4 increases slightly.
These results show that an undersized critic or insufficient updates can impair recovery. The default configuration, with width 128 and three updates per step, already performs well. Further increasing the capacity or the number of updates brings limited gains. The independent closure error improves alongside the recovery error, indicating that the effect is not merely a reduction in the score reported by the training critic.
The results suggest that a critic that is too small may underestimate the closure defect. Once its expressive capacity is adequate, however, increasing it further brings no clear benefit. This indicates that a suitably configured critic has a sufficiently small approximation gap to serve as an optimization surrogate during training.
The goal of this paper is to recover the linear latent space, rather than to optimize task relevance, so task relevance is not our focus here. Future work could combine task-relevance methods with task-specific information and modify alignment accordingly, allowing it to select latent variables that better match the task.
These experiments address the original manuscript’s lack of direct condition checks and sensitivity evidence, while clarifying both the issues that have been improved and the remaining limitations.