- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks a simple question: what does a scalar prediction loss fail to see in models such as JEPA? The main tool is elegant: the residual second moment whitened by the target covariance, . Eigenvalues above one then flag exactly the directions in which the predictor does worse than the unconditional mean, which is a much more informative failure signature than a single number. The paper also derives a rank-inertia bound for affine heads, and a LayerNorm-aware refinement that nicely explains the large unit-eigenvalue plateau. Around these tools, the authors build a practical audit based on support projection, plateau exclusion, cross-fitting, and healthy-resplit calibration. The experiments are extensive and well chosen, covering public checkpoints, loss-matched injected faults, training-induced faults, natural misconfigurations, and checkpoint exports. The main empirical claims are threefold: disagreement between scalar and directional error, predictable architectural plateaus, and transferable fault fingerprints.
Critical strengths
-
The architecture-aware correction is, for my money, the strongest part of the paper. I particularly like that the authors do not simply plot a whitened spectrum and treat every bump as a failure. The rank-inertia argument explains an expected structural feature rather than a pathology, and the LayerNorm refinement sharpens that explanation enough to actually check on real heads. Support projection and plateau exclusion then guard against two important ways a naive implementation could go wrong. The underlying linear algebra is admittedly not new in itself, but this is a useful and well-motivated application of it.
-
There is also meaningful evidence beyond the synthetic fault grids. The official I-JEPA head showing lower NMSE but substantially higher directional regret than the refit null is, to me, one of the most interesting results in the paper. The held-out comparisons and official-loss controls make it much harder to wave this away as a simple fitting-objective artifact. I additionally appreciate the authors' willingness to report where the method does not help. The failed transfers and negative pruning/repair results put useful boundaries around the claims. They do not by themselves settle the contribution question, but they make the paper considerably more credible.
Critical weaknesses
-
I am not yet convinced that the spectral machinery earns its additional complexity for the main diagnosis task. Appendix I reports that prediction-rank and residual-mean statistics classify all 149 primary injected arms, compared with 143/149 for the spectral rulebook. That makes the headline fault-classification results less compelling than they first appear, since the injected families are largely identifiable from the very properties used to construct them. The natural-fault results do not fully close this gap either. NMSE alone achieves perfect separation in the preregistered held-out experiment, and the spectrum does not consistently improve ranking, pruning, or repair selection. To be fair, I am not asking the method to win on every downstream task. But once those applications are set aside, the acceptance case rests mainly on directional interpretation and localization, and the practical benefit of those needs to be demonstrated more directly. A unified regret readout is a nice property to have; it is not, on its own, enough to establish a substantial diagnostic advance.
-
The validation of the final, calibrated procedure is still too narrow. Much of the evidence comes from reusing the same caches under new partitions, and the natural-fault test contains only a few independent fault families. The excess false positives at the looser target also suggest that the tail calibration is not entirely settled. There is a related selection issue in Appendix C: several normalizers are compared on the same reported test arrays, with one of them giving 0/200 false positives. I could not tell whether that normalizer had been fixed before those test results were examined. More broadly, the fingerprints need their construction-specific qualifications spelled out. Misalignment changes the prediction-target cross-covariance and does not generally leave the spectrum unchanged; the paper's own severe-misalignment failures illustrate this point nicely. The appendix is careful about such limits, but the headline descriptions are sometimes less careful. Taken together with the multiple rulebook versions, this leaves me uncertain about exactly what has been validated as a transferable unit test.
Questions for authors
-
What is the strongest existing evidence for incremental utility? Which result demonstrates a practical advantage over a combined prediction-rank/residual-mean diagnostic and a two-sided NMSE gate, given comparable healthy-baseline access? Please distinguish detection, morphology classification, and directional localization, since this distinction would materially affect how I judge the contribution.
-
Which rulebook produced the training-induced-fault result? Appendix I describes the 40/40 result as using a frozen six-feature rulebook, whereas the main results use five features. Does the final five-feature procedure also achieve 40/40 on those same arms? If not, I would like to know its actual result.
-
What was fixed before the calibration test was examined? Was the normalizer used for the single-statistic gate in Figure 5 chosen before looking at the 200 healthy test resplits? Also, should the 1% false-positive claim be read strictly as a conditional-on-cache empirical target, rather than a guarantee for new source data?
-
Are the inversion and resplit discrepancies only reporting errors? Figure 8 gives , while the stated truncated-head signature implies when the plateau saturates. Please state what the code actually computes, and distinguish projection rank, centered rank, and head width. Table 22 also says 30 resplits but uses denominators of 5. Please confirm the actual repetition count, and whether correcting either issue changes any reported conclusion.
3 comments
The response is effective and addresses some of my concerns, overall I keep my weak accept decision.
Thank you for taking the time to read our response and for your continued support. Your thoughtful feedback helped us clarify our claims and strengthen the paper. We’ll make sure the clarifications from our discussion are reflected in the final revision.
We are grateful to the reviewer for the careful reading, appendices included.
Q1 and W1. Localization is the incremental utility we can demonstrate (§4.1, Table 1B). For detection, a combined non-spectral gate matches the spectrum; for typing, a non-spectral classifier trained on fitted faults matches it, so the revision claims no advantage in either.
W2. What transfers as a unit test: frozen thresholds within a source, with recalibration on a new source (Q3), and rules fixed on I-JEPA, which type 93 of 99 injected faults on V-JEPA and V-JEPA2 and all 240 fitted faults on new images. Natural faults still come from few families, as the limitations state. With 500 calibration resplits, the realized rate at the nominal 1% is 0.40% to 2.00% across seven further blocks (median 1.37%); the 5% excess comes from the submission's block, whose eigenvalue channel realizes 5.06% against 1.87% to 3.03% for the others (Appendix C). Re-pairing changes the prediction-target cross-covariance, and the severe cases fail by leaving the alignment rule's plateau window (Appendix L). One deployed procedure is validated (Table 3); the frozen held-out scorer differs only in when it disables the spike rule, which changes no route but labels two leakage-type families on the 40k run set as bias instead of noise (Appendix P).
Q2. Yes. Re-scoring the same 50 stored feature vectors with the five-feature procedure gives all 40 intended labels and reads the 10 undertrained heads as rank; regenerating the 40 conditions gives the same and labels 200 healthy refits healthy (Appendix I).
Q3. No: the single statistic's normalizer was chosen after the 200 test resplits were scored; Figure 5 is now labeled exploratory and Table 4 reports the full grid. The deployed gate does not use it: its thresholds come from the 500 calibration resplits alone. Yes, the 1% is an empirical target conditional on the source and the calibration block, a nominal level with a design rate of about 1.4%. Frozen thresholds give 16 of 1,500 on a disjoint STL-10 pool (1.07%); on class-balanced Imagenette they fire on 35.2%, and recalibration on that source gives 1.2%.
Q4. The Table 22 discrepancy was a reporting error. Figure 8 plotted the wrong estimand, (k−nunit)/2−1, Theorem 1's lower bound on rank(W) before rounding; fixing it narrows the width-recovery claim. Appendix G separates the estimands: if the retained support preserves a saturated signature (q+1,q), then q=(k−nunit−1)/2; a head truncated to rank r after normalization, with an injective retained span, has q=r, and a full-rank LayerNorm head of width w has q=w−1, so w=(k−nunit+1)/2. This recovers widths 48 and 96; at 192 and 384 the support cannot identify the width. Section 2 distinguishes w, rank(W), q and ra: for the I-JEPA head w=rank(W)=384, q=383, and ra is 384 or 385.
Table 22 (Table 25 in the revision) used 5 resplits per cell. Rerun with 30, every I-JEPA cell keeps its modal label: each operation is typed correctly in 22 of 30 resplits at a 5% NMSE rise (4 of 5 in the submission), 30 of 30 at 10% and 20%, and 0 to 4 of 30 at 2%; video cells keep 5.