acceptodds
Under review as a conference paper at ICLR 2027

Spectral Unit Tests for Predictive Representation Models: A Fault Atlas and Its Scope Conditions

Abstract

Predictive representation models are trained and monitored through one scalar loss, which can hide a predictor that loses to the unconditional mean of its targets in some directions. We audit the residual spectrally: whitening its second moment against the target covariance makes any generalized eigenvalue above one a certificate of regret against the mean; a covariance ceiling attributes that regret to the predictor, not to irreducible noise. A LayerNorm-aware rank bound predicts the unit-eigenvalue plateau a normalized affine head forces; five public checkpoints saturate it, and excising it makes the spectrum interpretable. On the official I-JEPA head the scalar and the spectrum disagree: its NMSE is below that of a ridge head refit on the audit data, yet its plateau-excluded regret mass is 2.2 times that of the ridge head, a gap that official-loss refits narrow by only 4-10%. A calibrated gate fires at full power on a five percent rise in scalar error for rank, bias and noise faults, and reaches faults confined to low-variance directions, which no calibrated NMSE gate reaches on this source budget. NMSE-matched injected faults leave distinct spectral signatures consistent with the theory's bounds, and rules fit on one checkpoint transfer blind to two others under conditions we state and measure. The audit also caught the HuggingFace I-JEPA export carrying the online encoder rather than the EMA target.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, representation learning, JEPA, model diagnostics, spectral analysis, evaluation, failure detection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6MD4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks a simple question: what does a scalar prediction loss fail to see in models such as JEPA? The main tool is elegant: the residual second moment whitened by the target covariance, D=Ch†/2CeCh†/2D=C_h^{\dagger/2}C_eC_h^{\dagger/2}. Eigenvalues above one then flag exactly the directions in which the predictor does worse than the unconditional mean, which is a much more informative failure signature than a single number. The paper also derives a rank-inertia bound for affine heads, and a LayerNorm-aware refinement that nicely explains the large unit-eigenvalue plateau. Around these tools, the authors build a practical audit based on support projection, plateau exclusion, cross-fitting, and healthy-resplit calibration. The experiments are extensive and well chosen, covering public checkpoints, loss-matched injected faults, training-induced faults, natural misconfigurations, and checkpoint exports. The main empirical claims are threefold: disagreement between scalar and directional error, predictable architectural plateaus, and transferable fault fingerprints.

Critical strengths
  1. The architecture-aware correction is, for my money, the strongest part of the paper. I particularly like that the authors do not simply plot a whitened spectrum and treat every bump as a failure. The rank-inertia argument explains an expected structural feature rather than a pathology, and the LayerNorm refinement sharpens that explanation enough to actually check on real heads. Support projection and plateau exclusion then guard against two important ways a naive implementation could go wrong. The underlying linear algebra is admittedly not new in itself, but this is a useful and well-motivated application of it.

  2. There is also meaningful evidence beyond the synthetic fault grids. The official I-JEPA head showing lower NMSE but substantially higher directional regret than the refit null is, to me, one of the most interesting results in the paper. The held-out comparisons and official-loss controls make it much harder to wave this away as a simple fitting-objective artifact. I additionally appreciate the authors' willingness to report where the method does not help. The failed transfers and negative pruning/repair results put useful boundaries around the claims. They do not by themselves settle the contribution question, but they make the paper considerably more credible.

Critical weaknesses
  1. I am not yet convinced that the spectral machinery earns its additional complexity for the main diagnosis task. Appendix I reports that prediction-rank and residual-mean statistics classify all 149 primary injected arms, compared with 143/149 for the spectral rulebook. That makes the headline fault-classification results less compelling than they first appear, since the injected families are largely identifiable from the very properties used to construct them. The natural-fault results do not fully close this gap either. NMSE alone achieves perfect separation in the preregistered held-out experiment, and the spectrum does not consistently improve ranking, pruning, or repair selection. To be fair, I am not asking the method to win on every downstream task. But once those applications are set aside, the acceptance case rests mainly on directional interpretation and localization, and the practical benefit of those needs to be demonstrated more directly. A unified regret readout is a nice property to have; it is not, on its own, enough to establish a substantial diagnostic advance.

  2. The validation of the final, calibrated procedure is still too narrow. Much of the evidence comes from reusing the same caches under new partitions, and the natural-fault test contains only a few independent fault families. The excess false positives at the looser target also suggest that the tail calibration is not entirely settled. There is a related selection issue in Appendix C: several normalizers are compared on the same reported test arrays, with one of them giving 0/200 false positives. I could not tell whether that normalizer had been fixed before those test results were examined. More broadly, the fingerprints need their construction-specific qualifications spelled out. Misalignment changes the prediction-target cross-covariance and does not generally leave the spectrum unchanged; the paper's own severe-misalignment failures illustrate this point nicely. The appendix is careful about such limits, but the headline descriptions are sometimes less careful. Taken together with the multiple rulebook versions, this leaves me uncertain about exactly what has been validated as a transferable unit test.

Questions for authors
  1. What is the strongest existing evidence for incremental utility? Which result demonstrates a practical advantage over a combined prediction-rank/residual-mean diagnostic and a two-sided NMSE gate, given comparable healthy-baseline access? Please distinguish detection, morphology classification, and directional localization, since this distinction would materially affect how I judge the contribution.

  2. Which rulebook produced the training-induced-fault result? Appendix I describes the 40/40 result as using a frozen six-feature rulebook, whereas the main results use five features. Does the final five-feature procedure also achieve 40/40 on those same arms? If not, I would like to know its actual result.

  3. What was fixed before the calibration test was examined? Was the normalizer used for the single-statistic gate in Figure 5 chosen before looking at the 200 healthy test resplits? Also, should the 1% false-positive claim be read strictly as a conditional-on-cache empirical target, rather than a guarantee for new source data?

  4. Are the inversion and resplit discrepancies only reporting errors? Figure 8 gives r^=(k−nunit)/2−1\hat r=(k-n_{\rm unit})/2-1, while the stated truncated-head signature (r+1,r)(r+1,r) implies r=(k−nunit−1)/2r=(k-n_{\rm unit}-1)/2 when the plateau saturates. Please state what the code actually computes, and distinguish projection rank, centered rank, and head width. Table 22 also says 30 resplits but uses denominators of 5. Please confirm the actual repetition count, and whether correcting either issue changes any reported conclusion.

3 comments

We are grateful to the reviewer for the careful reading, appendices included.

Q1 and W1. Localization is the incremental utility we can demonstrate (§4.1, Table 1B). For detection, a combined non-spectral gate matches the spectrum; for typing, a non-spectral classifier trained on fitted faults matches it, so the revision claims no advantage in either.

  • Detection: recalibrated on the same 500 healthy resplits, a combined non-spectral gate (two-sided NMSE, residual-mean zz, prediction rank, span) gives 72 false positives in 3,000 against 64 for the spectral gate and the same decision on every resplit of the 12 injected conditions (Appendix C).
  • Typing: the rank-and-mean rule types all 149 injected faults. Both fixed rule sets were developed on injected faults; on 240 training-induced faults from disjoint images and 20 new seeds, the spectral rules type all 240 and the rank-and-mean rule 120, reading noisy targets and re-pairing as bias. A logistic regression trained on fitted faults types all 240 without spectral features. Noise-free small-sample refits also get the noise label, so this family does not isolate target corruption (Appendix I).
  • Localization: replacing the released I-JEPA head's predictions by the target mean along the 64 directions with the largest eigenvalues, chosen on separate images, raises the STL-10 score of a classifier, trained on target tokens and held fixed, by 0.84 ± 0.16 points, 0.30 ± 0.09 more than removing the mean residual (± is the half-width of the paired 95% bootstrap interval over 32-image batches). The count is not tuned: the 22 with eigenvalue above 2 give +0.78; the 379 above 1 give +0.92. On the head recentered on the fitting images, where removing the mean residual changes the score by −0.01, its own 64 directions still gain 0.32 ± 0.09. Along 64 principal directions ranked by error-to-variance ratio, the correction gives +0.10; random sets lower the score, and unwhitened residual directions cost 18 points. A classifier retrained on the predictions does not gain from the correction, so the result concerns consumers trained on target features (Appendix D).

W2. What transfers as a unit test: frozen thresholds within a source, with recalibration on a new source (Q3), and rules fixed on I-JEPA, which type 93 of 99 injected faults on V-JEPA and V-JEPA2 and all 240 fitted faults on new images. Natural faults still come from few families, as the limitations state. With 500 calibration resplits, the realized rate at the nominal 1% is 0.40% to 2.00% across seven further blocks (median 1.37%); the 5% excess comes from the submission's block, whose eigenvalue channel realizes 5.06% against 1.87% to 3.03% for the others (Appendix C). Re-pairing changes the prediction-target cross-covariance, and the severe cases fail by leaving the alignment rule's plateau window (Appendix L). One deployed procedure is validated (Table 3); the frozen held-out scorer differs only in when it disables the spike rule, which changes no route but labels two leakage-type families on the 40k run set as bias instead of noise (Appendix P).

Q2. Yes. Re-scoring the same 50 stored feature vectors with the five-feature procedure gives all 40 intended labels and reads the 10 undertrained heads as rank; regenerating the 40 conditions gives the same and labels 200 healthy refits healthy (Appendix I).

Q3. No: the single statistic's normalizer was chosen after the 200 test resplits were scored; Figure 5 is now labeled exploratory and Table 4 reports the full grid. The deployed gate does not use it: its thresholds come from the 500 calibration resplits alone. Yes, the 1% is an empirical target conditional on the source and the calibration block, a nominal level with a design rate of about 1.4%. Frozen thresholds give 16 of 1,500 on a disjoint STL-10 pool (1.07%); on class-balanced Imagenette they fire on 35.2%, and recalibration on that source gives 1.2%.

Q4. The Table 22 discrepancy was a reporting error. Figure 8 plotted the wrong estimand, (k−nunit)/2−1(k-n_{\rm unit})/2-1, Theorem 1's lower bound on rank(WW) before rounding; fixing it narrows the width-recovery claim. Appendix G separates the estimands: if the retained support preserves a saturated signature (q+1,q)(q+1,q), then q=(k−nunit−1)/2q=(k-n_{\rm unit}-1)/2; a head truncated to rank rr after normalization, with an injective retained span, has q=rq=r, and a full-rank LayerNorm head of width ww has q=w−1q=w-1, so w=(k−nunit+1)/2w=(k-n_{\rm unit}+1)/2. This recovers widths 48 and 96; at 192 and 384 the support cannot identify the width. Section 2 distinguishes ww, rank(WW), qq and rar_a: for the I-JEPA head w=rank(W)=384w={\rm rank}(W)=384, q=383q=383, and rar_a is 384 or 385.

Table 22 (Table 25 in the revision) used 5 resplits per cell. Rerun with 30, every I-JEPA cell keeps its modal label: each operation is typed correctly in 22 of 30 resplits at a 5% NMSE rise (4 of 5 in the submission), 30 of 30 at 10% and 20%, and 0 to 4 of 30 at 2%; video cells keep 5.

The response is effective and addresses some of my concerns, overall I keep my weak accept decision.

Thank you for taking the time to read our response and for your continued support. Your thoughtful feedback helped us clarify our claims and strengthen the paper. We’ll make sure the clarifications from our discussion are reflected in the final revision.

Reviewer_PsjFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study addresses a limitation in the evaluation of predictive representation models: the standard mean prediction error metric can overlook prediction issues occurring along specific directions within the representation space. To tackle this, the authors propose a method to evaluate prediction errors for individual directions by performing a generalized eigenvalue analysis on the second-moment matrix of the prediction residuals and the covariance matrix of the target representations. Experiments using models such as I-JEPA and V-JEPA demonstrate that this approach can detect issues that are difficult to capture with scalar-based prediction error metrics, and that the theoretically derived eigenvalue structures are indeed observable in practice. Furthermore, the study analyzes how the eigenvalue distribution shifts when experimental settings are altered, showing that these shifts can be used to classify the nature of the changes. The paper also examines differences between various implementations of publicly available models.

Critical strengths

The paper is commendable for reconsidering the standard practice of using scalar prediction error to evaluate predictive representation models.

Furthermore, the paper is praiseworthy for providing a theoretical characterization of the proposed evaluation method, supported by formal theorems, and for demonstrating the corresponding results through experiments.

Critical weaknesses
  1. The practical utility and positioning of the proposed method remain unclear.

Assuming the use of a predictive representation model, it is unclear where the problems actually arise. Furthermore, while this paper opts for JEPA-based models, such as I-JEPA and V-JEPA, the reason for choosing these particular models is not explicitly explained. In particular, LeJEPA, which employs SIGReg, is likely to see increased adoption, as it does not require mechanisms like EMA and allows for a simpler training process. Furthermore, given that SIGReg imposes isotropy on latent representations, it is highly relevant to the problem addressed in this paper. Although the relationship with SIGReg is discussed in Appendix L, the experimental validation remains insufficient. It would be beneficial to conduct additional experiments using SIGReg to more clearly demonstrate the effectiveness and positioning of the proposed method, even if only in the appendix.

  1. A review of the terminology and expressions used is necessary.

Some of the expressions used, such as "rig," "fault atlas," and "rank stamp", are not typical for an academic paper; therefore, I believe the text should be revised in its entirety.

Questions for authors

Do you have any idea about practical scenarios in which JEPA with scalar loss fails? In terms of SIGReg, how does the proposed spectral audit behave?

4 comments

We thank the reviewer for the helpful comments and suggestions. We added the SIGReg experiment you asked for, and the revision answers your practical question with three cases (§4.1, §4.4, Appendix L).

Where the scalar loss fails (your first question).

(1) Training runs whose loss improves while the representation degrades: in our run sets of small I-JEPA training runs, six harmful settings (among them high weight decay, no warmup, a slow or frozen EMA target and a high learning rate) lower NMSE below the healthy runs while the final probe drops by 8.9 to 41.0 points, because the target gets easier to predict as the encoder degrades. A check that treats lower loss as better calls all of them healthy. A two-sided gate, read post hoc on these run sets, flags all 18 runs, and on a later held-out set, two-sided scores separate nine harmful from nine healthy runs (AUC 1.00; §4.4, Appendix P).

(2) SIGReg itself: training TwoRoom at weight 0.0009 gives the lowest training prediction loss of all runs (5.6×10⁻⁷) while planning succeeds in 29.5% of episodes, against 86.5% at the default weight. Eval-mode NMSE (1.2×10⁶) and prediction rank (28 of 192) expose the collapse, as does the regret mass (1.00).

(3) Choosing between the released I-JEPA prediction head and a ridge refit for a module trained on target features and held fixed: I-JEPA's training loss (and NMSE on STL-10) ranks the refit last, the regret mass ranks it first, and the module scores the refit 1.09 points higher on STL-10 and 1.01 on Imagenette (§4.1; reply to GsXM).

In (1) and (2), NMSE read in both directions or in evaluation mode catches the failure. In (3), normalizing each error direction by its target variance changes the choice, and the spectrum adds where to correct: on STL-10, correcting the released head along its top 64 spectral directions raises the fixed classifier's score by 0.84 points, 0.30 ± 0.09 more than removing the mean error (± is the half-width of the paired 95% bootstrap interval over 32-image batches).

How the audit behaves under SIGReg (your second question).

This is a good question. LeWorldModel is a SIGReg-trained JEPA world model whose release includes the predictor with its affine output layer and the encoder that produces its targets; we audited the released predictors on PushT and TwoRoom (Table 7). The regret reading is defined on the retained target support for any target covariance, isotropic or not. What changes is the architecture: the head maps 2,048 inputs to 192 outputs without a bottleneck, so neither the healthy head nor any tested rank condition has a forced unit-eigenvalue plateau, and no rank condition is labeled rank.

Over the four injected families (5 rank conditions and 15 each of bias, noise and re-pairing), the rules fixed on I-JEPA type 37 of 50 on PushT and 15 of 50 on the released TwoRoom checkpoint (re-pairing 15 of 15 in both), whose evaluation-mode BatchNorm statistics raise its healthy error; re-estimating them on 25,600 training windows, a post hoc modification reported separately, lowers healthy NMSE from 0.067 to 0.012 and gives 43 of 50.

Why these models (your first weakness).

The audit reads a released predictor's affine output head with the encoder that produces its targets. I-JEPA, V-JEPA and V-JEPA2 release both, as does LeWorldModel, which uses its predictor for planning. LeJEPA recommends training without a predictor (Balestriero and LeCun, 2025), so a LeJEPA model trained as recommended has no prediction head to audit. The public LeVJEPA checkpoint contains only the encoder, so its prediction-target pair cannot be built; Appendix L still reads the encoder's local view against its global view (55 of 239 directions with eigenvalue above one). Section 4 now states this criterion, and Appendix L.4 adds the LeJEPA case.

Terminology (your second weakness).

As you suggested, we rewrote the main text, and the revision replaces the three terms you named and the others listed in the general comment. The title keeps its submitted subtitle, "A Fault Atlas and Its Scope Conditions," through the discussion period; we will drop the subtitle in the first version after it.

Dear Reviewer PsjF,

Thank you again for your review. As the discussion period ends on November 18 (AoE), we briefly recap how our response addresses your two questions.

Your first question asked for practical scenarios in which JEPA with a scalar loss fails. The response gives three cases:

  • Training settings: in small I-JEPA training, six groups of runs with harmful settings, such as high weight decay, no warmup or a frozen EMA target, drive NMSE below that of the healthy runs while their mean final probe accuracy is 8.9 to 41.0 points lower (Section 4.4).
  • The SIGReg weight: among ten-epoch LeWorldModel runs on TwoRoom, weight 0.0009 gives the lowest training prediction loss, yet planning succeeds in 29.5% of episodes, against 86.5% and 88.0% for the two runs at the default weight (Appendix L.1).
  • The prediction head: I-JEPA's training loss keeps the released head, while the regret mass picks a ridge refit. A fixed classifier trained on target tokens is more accurate on the refit's predictions than on the released head's, by 1.09 points of token accuracy on STL-10 and 1.01 on Imagenette (Section 4.1).

For your second question, on how the audit behaves under SIGReg, we audited the released SIGReg-trained LeWorldModel predictors on PushT and TwoRoom (Table 7).

The revision also states why these models were chosen (Section 4) and replaces the terms you named in the body.

If any point remains open, or if a specific number would help, we would be glad to answer it before the discussion closes.

Thank you for the detailed response and the additional experiments. My main concerns have been addressed, and I will keep my current assessment.

Thank you for the follow-up and for confirming that your main concerns are addressed. Your question on SIGReg-trained models was especially helpful, and we appreciate the time and care you gave to our paper.

Reviewer_GsXMOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper does a residual analysis on the linear head of JEPA predictors. It whitens the error covariance by the target covariance and reads the eigenvalues as per-direction residual variance against predicting the mean. Since the head is a 384-to-1280 linear map, most eigenvalues are 1 by construction. The paper counts these from the architecture and removes them. What remains shows the predictor is worse than the mean in a third of the directions, which are low-variance, with little effect downstream effect. They build a change detector of the head with resampled thresholds and if-then rules to catch injected faults.

Critical strengths

The handling of the eigenvalues forced to one by the narrow head is correct and was checked on five public models.

The paper shows a case where the official head has lower total error than a refit head, yet more of its error sits in directions where it does worse than guessing the mean.

Critical weaknesses

The main issue is significance. The central empirical finding is that the JEPA predictor is worse than the constant predictor in about a third of the whitened directions. But Appendix D shows that these directions carry only about 2.5% of the error once weighted by target variance and that removing them does not hurt the linear probes. So the paper’s own evidence says the loss can ignore them. The paper gives no example where looking at the spectrum would change a decision you’d have made from the loss alone. In the one experiment with real problems, where the authors trained many small models with deliberately bad settings and tried to predict partway through which ones would end up good, the ordinary loss predicted final accuracy as well as any eigenvalue-based quantity (Table 16). The experiments on choosing a repair and on pruning hyperparameter sweeps are both negative (Appendices N, Q). In the injected-fault battery, where the four fault types are separable by construction (Section 4.3), Appendix I shows that two simple checks, the rank of the head’s output and whether the average error is zero, classify all 149 cases correctly without any of the spectral machinery. The mathematics is standard (canonical correlation and reduced-rank regression).

The second issue is presentation. The paper is 41 pages and the main text cannot be read without the appendices. Throughout, claims are supported by citing internal script and result files by name (p0p_power_fresh.json, review_0923_measure.json, elementary_typing.py, critical_review_checks.py and many others) as if they were evidence. The text reads extremely poorly, as drafted by a language model, with every claim qualified in several directions at once, so it is hard to say in one sentence what the paper claims. The finding that the HuggingFace copy of I-JEPA is the online encoder, instead of the EMA encoder used in the original paper, is not related to the proposed method.

Questions for authors
  • Is there a case, on a real model or training run, where reading the spectrum leads to a different decision than reading the loss, and the spectrum’s decision is the right one?
3 comments

We thank the reviewer for the careful review. Yes, for two decisions on the public I-JEPA ViT-H/14 predictor:

  • Which output head to keep: I-JEPA's own training loss (SmoothL1 on layer-normalized targets) and NMSE (on STL-10; a tie on Imagenette) keep the released head, the regret mass switches to a ridge refit, and a classifier, trained on real target tokens and held fixed, scores the refit higher on both image sets (§4.1, Table 1).
  • Which directions to correct: the spectrum names them, and correcting them helps the same classifier (below).

This extends the disagreement you list as a strength.

We compare the released head, the same head with its mean error on the fitting images removed from the bias (recentered), and the refit, listed in that order in the table. A logistic regression trained on real target tokens scores each head's predictions unchanged; an image's score is the share of its predicted tokens assigned to its class. Fitting, measurement, classifier training and scoring use four disjoint image sets. In a ± b, b is the half-width of the paired 95% bootstrap interval over 32-image batches.

MeasurementSTL-10Imagenette
NMSE0.565 / 0.564 / 0.5720.533 / 0.533 / 0.534
Regret mass0.141 / 0.137 / 0.0660.148 / 0.144 / 0.037
Classifier score, points91.46 / 92.01 / 92.5592.47 / 92.79 / 93.47
Refit − released, points+1.09 ± 0.25+1.01 ± 0.49
Refit − recentered, points+0.53 ± 0.20+0.69 ± 0.39

On STL-10 the released head's NMSE is lower than the refit's by 0.0072 ± 0.0011, so the regret mass overrules NMSE; on Imagenette the refit's NMSE is 0.07% ± 0.12% above the released head's, a tie, and the regret mass makes the call. The recentered head shows that the choice is not an intercept effect: NMSE prefers it to the refit, while the regret mass and the classifier prefer the refit. Scored by image accuracy (a classifier on per-image mean features), the refit gains 0.54 ± 0.28 points on STL-10.

Normalizing each error direction or coordinate by its target variance is what changes the head choice: whitened NMSE (the mean of the spectrum's eigenvalues) and a per-coordinate standardized NMSE also rank the refit first.

The eigendecomposition is what locates the error. On STL-10, replacing the released head's predictions by the target mean along the 64 directions with the largest eigenvalues raises the score by 0.84 ± 0.16 points, 0.30 ± 0.09 more than removing the head's mean error. The same correction gives +0.10 along 64 principal directions ranked by error-to-variance ratio, lowers the score along every random set of 64, and costs 18 points along unwhitened residual directions. On the recentered head, where removing the mean error changes the score by −0.01, its own 64 directions still gain 0.32 ± 0.09.

Thank you for the close look at Appendix D. We respectfully disagree that it lets the loss ignore these directions: its probe was retrained after the directions were removed, so it measures how much the target needs them, not what prediction error in them costs a fixed consumer. Removing the 64 flagged directions from real target tokens costs the classifier 0.28 points, about as much as the 64 principal directions (0.33); correcting the predictions along them gains 0.74 ± 0.15 points more than correcting along the principal ones.

The refit's classifier score gain over the released head holds over four further STL-10 fitting splits (+1.10 to +1.21 points), STL-10 classifier L2 penalties from 10⁻⁷ to 10⁻¹ (+0.48 to +1.32, all 95% intervals above zero) and raw or standardized classifier inputs on both datasets. A classifier retrained on each head's own predictions nearly closes the gap (+0.01 points on STL-10, +0.08 on Imagenette): the case concerns a consumer trained on target features and kept fixed. This choice arises in feature distillation that reuses a teacher's classifier (SimKD; Chen et al., CVPR 2022); our test builds such a consumer on a released predictor without running SimKD.

We agree with your reading of Table 16 (Table 18 in the revision), Appendices N and Q, and the 149 injected cases; the revision claims no detection or typing advantage for the spectrum. We also agree that the foundations are classical (the covariance ceiling, Proposition 1; the CCA residual spectrum and mean-relative skill scores, §5); the paper adds an inertia bound that holds for any trained affine head, bias included, and the LayerNorm counts behind the plateau handling you found correct.

On presentation, we revised the main text so that each result is stated where it is used and repeated qualifications are stated once; proofs, protocols and further checks are in the appendix, mapped to the main-text questions by an overview. Script and result file names are removed; the HuggingFace result is now a provenance check in Appendix S.

Thank you for the response and the revision of the writing. My remaining concern is still significance, as the new example shows that normalizing the error by target variance can change which head you pick, but simpler normalized losses can make the same choice without spectral analysis, thus in the end the spectrum adds a small gain in a narrow setting. As the detection and typing advantages were withdrawn, I find the practical contribution limited. I am raising my rating to 2 to reflect the improvements, but I do not think the paper is ready for acceptance.

Thank you for raising your rating and for stating the remaining concern so clearly; it led us to the new analyses below. A normalized scalar does pick the better head, and the spectrum adds which directions to correct: correcting only the 64 directions it flags, out of about 800, closes 48% to 93% of the score gap between the released head and the refit for the logistic classifiers, against at most 40% for the 64 target principal directions with the largest error-to-variance ratios. On a second public I-JEPA checkpoint, the spectral directions also close more of the gap than these principal directions for both logistic classifiers.

Closing the gap to the refit. By classifier, the 64 spectral directions close 77% and 93% of the gap on STL-10 and 48% and 85% on Imagenette, for the raw and standardized classifiers. The principal directions above close 9% to 40% (averaged over all retained principal directions, their error-to-variance ratios give whitened NMSE), and removing the mean error closes 15% to 58%. On the second checkpoint, I-JEPA ViT-H/14 pretrained on ImageNet-22K, the spectral directions close 58% and 75% of the gap for the two classifiers on STL-10, against 2% and 30% for the principal directions and 44% and 55% for mean removal.

Breadth. On the first checkpoint we also swept two image sources, the released and recentered heads, and 16, 32, 64 and 128 directions (16 settings), with six classifiers trained on real target tokens. In the table, accuracy is each classifier's accuracy on real target tokens (STL-10 / Imagenette); gain is the range, over the 16 settings, of the score change from correcting the spectral directions; margin is the range of that gain minus the larger gain of the two simpler corrections in the same setting (removing the mean error, or correcting the same number of principal directions); wins counts the settings in which the paired 95% intervals of both differences are above zero.

ClassifierAccuracy on real tokens (%)Gain (points)Margin (points)Wins (of 16)
Logistic regression97.5 / 97.2+0.19 to +0.86+0.13 to +0.3216
Logistic regression, standardized tokens98.4 / 97.7+0.37 to +3.00+0.16 to +1.1115
Nearest class mean, Euclidean82.7 / 87.0−0.03 to +0.23−0.09 to +0.203
Nearest class mean, standardized tokens82.8 / 87.3+2.19 to +5.64+0.94 to +3.4716
Nearest class mean, Mahalanobis97.5 / 96.9+0.33 to +2.49+0.15 to +0.8115
LDA (equal priors)97.1 / 96.4+0.67 to +2.60+0.38 to +1.2416

Against the principal directions alone, the spectral directions are ahead with the interval above zero in 88 of the 96 settings, and random directions lower the score in 479 of 480 cases. On the second checkpoint (STL-10, 8 settings per classifier), the spectral correction wins in 8 of 8 settings for the standardized logistic classifier and the three reweighted rules, in 3 of 8 for the logistic classifier, whose gap to the refit there is 0.24 points, and in 0 of 8 for the Euclidean rule.

Why one classifier barely moves. On the first checkpoint, the spectral directions are ones along which the target varies little: setting the 64 of them to the mean on real target tokens would remove 0.8% of the tokens' variance on STL-10 and 0.5% on Imagenette, against 7.9% and 7.8% for 64 random directions. The Euclidean nearest-class-mean rule hardly uses them (they carry 0.15% to 0.29% of the variance of its class scores on real tokens), and the spectral correction changes its score by less than a quarter point across the sweep. Standardizing the rule leaves its score on real STL-10 tokens unchanged (82.8 against 82.7) but lowers its score on the released head's predictions from 84.7 to 76.6, and the 64 spectral directions recover 5.5 of those 8.1 points.

Practical contribution. The unit test's two-sided scalar check matters in practice because harmful JEPA settings can lower the loss: on the held-out run set all nine harmful runs end with NMSE 3% to 22% below the healthy mean, and the check flags all nine (Appendix P). This check does not use the spectrum.

We will add the full tables to Appendix D. We hope these results address your remaining concern about significance, and we would be glad to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and summary of the revision

We sincerely thank the reviewers for their thoughtful and constructive comments. All three asked what the spectrum adds in practice; PsjF also asked how the audit behaves under SIGReg, and GsXM and PsjF asked for clearer writing. The revised PDF addresses each point as follows. Substantive additions, corrections and changes in the scope of the submission's claims are marked in blue, and expositional and stylistic edits are unmarked.

Claim. Whitening a predictor's residual by the target covariance, after excluding the unit eigenvalues its architecture forces, locates directions in which a trained predictor does worse than the target mean. On the public I-JEPA predictor, average loss ranks prediction heads the opposite way to this directional error, and on STL-10, correcting the located directions improves a classifier, trained on target tokens and held fixed, beyond removing the mean error. The same spectrum checks a predictor against a healthy reference.

Unchanged. We thank the reviewers for their positive feedback on several results, which the revision preserves:

  • The rank-inertia bound and its LayerNorm refinement, whose ambient rank ceiling five public checkpoints attain (§2, §4.3).
  • The official I-JEPA head, whose held-out NMSE is below a ridge refit's while its mean plateau-excluded regret mass is 2.2 times the refit's, a gap that SmoothL1 and ℓ1 refits of the comparison head narrow by only 4% and 10% (§4.2).

The calibrated gate (118 of 120 faulted resplits detected, 3 false alarms in 200 healthy resplits) and the rules fixed on I-JEPA that type 93 of 99 injected faults on V-JEPA and V-JEPA2 are also unchanged (§4.2, §4.4).

New.

  • Head choice and directional correction for a classifier trained on target tokens and held fixed (§4.1, Table 1, Appendix D; GsXM, PsjF, 6MD4 Q1). I-JEPA's SmoothL1 loss (on both image sets) and NMSE (on STL-10) prefer the released head, the regret mass prefers a ridge refit, and the classifier scores the refit higher on both image sets. Correcting the released head's predictions along 64 spectral directions chosen on separate images raises the STL-10 score by 0.84 points, 0.30 more than removing its mean error (Table 1).
  • Detection and fault typing against non-spectral baselines with the same healthy access (§4.2, §4.4, Appendices C and I; 6MD4 Q1). On 240 training-induced faults from new images and seeds, the spectral rules and a non-spectral classifier trained on fitted faults both type all 240; a fixed rank-and-mean rule types 120.
  • Calibration on a disjoint STL-10 pool and on class-balanced Imagenette (Appendix C; 6MD4 W2, Q3).
  • LeWorldModel, a SIGReg-trained JEPA world model (§4.4, Appendix L; PsjF): its released predictors audited on PushT and TwoRoom, and TwoRoom trained at four SIGReg weights; at the smallest weight the lowest training loss belongs to a collapsed model, which eval-mode NMSE, prediction rank and regret mass expose.
  • Table 22 (Table 25 in the revision) with 30 resplits in every I-JEPA cell (6MD4 Q4).

Corrections.

  • The strongest non-spectral baselines we tested match the spectrum at detecting and typing the tested faults, and the revision says so; what the spectrum adds is localization (6MD4 W1, GsXM).
  • The HuggingFace export finding came from output comparisons and tensor matching; it moves from the abstract to Appendix S (GsXM).
  • Figure 8 plotted Theorem 1's lower bound as an estimate; conditional inversion formulas replace it (6MD4 Q4).
  • The gate's 1% is a nominal level with a design rate of about 1.4%, and the exploratory single statistic's normalizer was chosen after the test resplits were scored; Table 4 reports the full grid (6MD4 Q3).
  • Re-pairing is no longer said to leave the spectrum unchanged: it changes the prediction-target cross-covariance, and Appendix H reports how far it lowers the skill of directions where the predictor beats the mean. Figure 3 now uses the official V-JEPA2 forward pass (6MD4 W2).
  • On Imagenette the regret gap is 8.5 points on a class-balanced cache; the submission's 18.1 came from a class-ordered subset (Appendix C).

Presentation.

  • The main text is rewritten around the reviewers' questions: what the spectrum adds in practice (§4.1), how it compares with non-spectral baselines (§4.2, §4.4) and how it behaves under SIGReg (§4.4).
  • Script and result file names are removed; an appendix overview maps each main-text question to its analyses.
  • Terms: fault atlas → catalog of fault signatures, rank stamp → rank signature, battery → fault suite, fleet → run set, rulebook → rule set; "rig" is gone.
  • The title keeps its submitted subtitle, "A Fault Atlas and Its Scope Conditions," through the discussion period; we will drop the subtitle in the first version after it.
  • The two routes that bypass fault typing are named by their triggers (loss-lowering, eigenvalue-only) throughout, appendix tables included.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.