- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes an "order spectrum" to analyze the polynomial degree required to extract target information from learned representations. For Gaussian representations, the paper demonstrates that the regression function of a target can be decomposed into Hermite components; specifically, the of an optimal affine probe corresponds to the first-order energy, while the incremental performance gain of an optimal polynomial probe at each degree aligns with the energy of that corresponding order. For non-Gaussian representations, the paper provides bounds on the error of this relationship by utilizing the Hermite-feature Gram deviation and the target-dependent approximation residual.
The authors numerically validate these bounds within controlled settings and in the posterior mixture distributions of an MNIST β-VAE. Furthermore, they examine cases—such as in OthelloGPT, a pendulum-based JEPA, and models trained on modular addition—where quadratic probes outperform affine probes in predicting targets, and they analyze how the accessibility of this information evolves throughout the training process.
Critical strengths
-
The conditions for the theory’s application are clearly stated. The error that arises when relating the performance gain of an optimal polynomial probe—within a non-Gaussian representation—to a Gaussian-referenced Hermite energy is bounded by the "Gram deviation" and a target-dependent approximation residual. This formalizes the conditions and limitations under which the interpretation holds.
-
Numerical validation of the proposed bound is provided. The components of the bound are evaluated using the posterior mixture distribution of an MNIST β-VAE alongside the actual digit labels. Notably, the observation that the bound is violated when the residual term is omitted confirms the necessity of that term.
-
The scope for interpreting the experimental results is carefully constrained. While predictive performance is compared across different orders in various representation learning settings, the scores obtained from fitted probes are not equated with the calibrated energy ratios. This distinction ensures that the experimental results are evaluated without overinterpretation.
Critical weaknesses
-
The significance of the additional contributions relative to existing research is not sufficiently compelling. It is already well-established that the failure of a linear probe does not necessarily imply an absence of information, and that additional signals can be extracted using polynomial readouts. Furthermore, the energy relationships across different orders in the Gaussian case are rooted in classical orthogonal expansions and projection theory. Consequently, while the primary technical innovation appears to be the derivation of error bounds for non-Gaussian distributions, it remains to be clearly demonstrated what significant insights or analytical capabilities these bounds offer beyond existing polynomial probing methods.
-
There is a gap between the validation of the non-Gaussian bounds and their practical utility. While the authors successfully validate these bounds using a posterior mixture distribution from an MNIST -VAE—thereby demonstrating the necessity of the residual term—the precision provided by these bounds in real-world analytical scenarios is less clear. For instance, at , the mean discrepancy is , whereas the mean bound is , indicating a highly conservative estimate. In the case studies involving OthelloGPT, JEPA, and modular addition, the bounds do not serve to guarantee an energy-based interpretation; rather, they primarily function to diagnose the limitations of such interpretations. While this is a meaningful contribution, the significance of the work would be more persuasive if it included examples where these bounds lead to tangible improvements in analytical judgments—judgments that would otherwise be difficult to reach using conventional probing techniques alone.
-
Greater control is required to disentangle the effects of "accessibility by order" from the influence of the fitting procedure on actual probe performance. The experiments in the paper evaluate probes that are fitted using regularization on finite samples, rather than utilizing the optimal population-level predictor. Therefore, it is crucial to determine to what extent the observed performance differences reflect the underlying order-based structure of the representations, as opposed to being artifacts of the chosen coordinate system, conditioning, or regularization strategy. As an exploratory check, I conducted a small-scale experiment using three GELU-based models—each with a width of 128 and four hidden layers—trained on the 10-bit full parity task. I compared the representation immediately following the second activation layer with the output of the subsequent affine layer. All three models achieved 100% classification accuracy on 50 held-out test samples. Following the procedure specified in Appendix G.2 of the paper, I standardized the inputs and employed a quadratic kernel, . The penalty parameter was selected based on internal validation at the final checkpoint, then refitted to the entire probe-fitting dataset and held constant for earlier checkpoints. Using the sum-of-squared-errors criterion, the quadratic test for the earlier representation was and across the three random seeds, while for the later representation, it was higher: and . The respective means were approximately and . However, substituting into the quadratic readout for the later representation yields a function that is also quadratic with respect to . By transforming the coefficients, I was able to apply this to the earlier representation and perfectly replicate the predictions obtained from the later one. This indicates that a successful quadratic readout for the earlier representation did exist, but the standard fitting procedure failed to discover it. The observed disparity between the layers persisted even when varying the conventions for the penalty term (sum vs. mean of errors) and the handling of the quadratic intercept. These findings demonstrate that the scores obtained from fitted probes can depend significantly not only on the polynomial degree but also on the choice of coordinate system and regularization. Consequently, when interpreting differences between representations based on performance at various polynomial degrees, conducting sensitivity analyses regarding affine transformations and regularization serves as a valuable supplementary control. Since the authors explicitly state that a low fitted score does not imply the failure of the entire function family, this result does not constitute a counterexample to the theorem. The core of the critique is not whether the authors acknowledged these limitations, but whether they sufficiently evaluated the magnitude of their impact in actual experiments. It should be noted, however, that this additional experiment does not represent a full replication of the authors' implementation; it is an exploratory result limited to three initializations and a single data split.
Questions for authors
-
Can you provide examples where non-Gaussian bounds improve the assessment of learned representations? While I acknowledge that the necessity of bounds—and the residual terms dependent on the target—has been validated for MNIST β-VAE, in some settings, these bounds appear significantly more conservative than the actual discrepancy. Beyond merely confirming numerical validity, under what conditions do these bounds provide a quantitative guarantee for interpreting probe performance at different polynomial degrees, or enable insights that would be difficult to obtain through standard polynomial probing alone?
-
Is it possible to demonstrate applicability to higher-order interactions using tasks where the interaction order can be controlled, such as the parity problem? Current empirical results focus on low-order polynomials, specifically comparing affine and quadratic models. For tasks like parity, one could distinguish between the order of the input-target relationship and the order of the readout used for the learned representation, thereby testing whether higher-order probes can recover additional signals. Analyzing how sample size and computational cost scale with the order—along with the behavior of , , and the validity of the bounds—would help clarify the scope of the proposed framework.
-
How can the influences of the coordinate system, conditioning, and regularization be disentangled from the fitted performance at each order? I agree with the authors' point that a low fitted score does not necessarily imply the failure of an entire family of polynomials. Given this, what controls do you recommend for assessing the reliability of observed performance gains when using this framework as a diagnostic tool? Specifically, can you confirm that your primary findings and interpretations remain robust when accounting for factors such as affine transformations of the representations, data whitening, or sensitivity to regularization?
2 comments
Thank you for the response and the added control protocol. While I appreciate the more transparent framing, my core concerns remain unresolved:
- Practical utility of Proposition 2: The non-Gaussian bound remains loose in real-world settings and still only serves as a negative diagnostic, while the new OthelloGPT sign-product bound does not use Proposition 2 at all.
- Fitting sensitivity: While the final grokking gap and the existence of the OthelloGPT gap persist, the added controls still confirm strong sensitivity of the estimates to coordinates and regularization (e.g., up to 0.54 shifts in pre-transition grokking scores under affine maps, and large magnitude drops under whitening in OthelloGPT), and the transport check was not run on the paper's models.
- Higher-order scope: Empirical evaluation on trained networks remains limited to degree-2, with higher-order tasks deferred to future work.
Accordingly, I will maintain my current rating.
Thank you for the careful review, and especially for running your own parity experiment. Section numbers refer to the revised PDF.
W1. The contribution over existing work isn't compelling. A failed linear probe is known not to imply missing information, and the Gaussian energy relations are classical.
We agree with both points, and the revision mentions this. Section 2 now attributes Proposition 1 to classical Gaussian analysis (Brillinger, 1982), and we don't claim that "linear fails, nonlinear succeeds" is new. What we add is a measurement and its conditions. The order spectrum is now defined as the increments Δk between nested population polynomial probes under the representation's own law, so it assumes nothing about the distribution. We then say exactly when these increments equal Gaussian Hermite energies, bound how far they can move off the Gaussian, and give the controls fitted probes need before such increments are read (Section 2.1, Appendix H). The OthelloGPT result under W2 is an example of what this gives beyond a higher score.
W2 / Q1. There's a gap between validating the bound and its practical use. It's conservative (at β=2 a discrepancy of 0.0091 against a bound of 0.2564), and in the case studies it mainly diagnoses limits. Where does it change a judgment that standard probing couldn't reach?
We agree with your reading, and the revision now says it directly. On the OthelloGPT and pendulum activations the bound is loose, so we use Proposition 2 as a stability result relative to the Gaussian reference and not as a certificate for fitted scores (Section 3). It still changes one judgment that would otherwise be made. Gaussian-regularized representations such as the SIGReg JEPA are easy to read as near-Gaussian, but their measured Gram deviation is 6.230 to 8.698, far above one (Section 7), so a Hermite-energy reading of their probe increments isn't justified even though the regularizer targets the Gaussian. We also added a case where theory gives a number that probing alone can't. A sign-product proposition for OthelloGPT (Section 6.1, Appendix E.7), which needs no Gaussian assumption, bounds degree-two accessibility of board color from below at 0.74, 0.81 and 0.82 at layers 2 to 4, while the fitted quadratic probe scores 0.56, 0.64 and 0.66. So the fitted probe loses 0.16 to 0.18 to estimation, which no fitted score could show on its own.
W3 / Q3. Accessibility by degree must be separated from the fitting procedure. What controls do you recommend, and do the main findings survive affine maps, whitening and regularization?
Your experiment shows exactly the failure mode the revision now tests for, and transport through a known map, as in your experiment, is step 5 of the protocol for fitted probes in Appendix H. Each main finding is checked against the controls in Table 10, where W means the claim holds but a number moves by 0.05 or more, or the selected penalty sits at a grid edge with validation still improving. Cells without a reported number are marked as such.
We haven't run the transport check on our own models, and the protocol explains it (Table 10 has no transport column).
Q2. Can the framework be shown on higher-order interactions with a task of controlled order, such as parity, including how sample size, cost, ρk, τk and the bound scale with order?
We didn't add a parity experiment in this revision, but agree it would be a good test of degrees above two and note it as future work. Within the current paper, degrees above two are covered by the controlled checks in Section 4 and Appendix C, where the nested ladder recovers a target with known energies up to degree three and the degree-four increment stays near zero (−0.002). Appendix H.1 now also reports cost, i.e., the degree-k feature count (kd+k), the fitting times and memory, and why sample size, not compute, limits the full-space quadratic probe.