acceptodds
Under review as a conference paper at ICLR 2027

Disentangling Regularization Effects in Self-Supervised Learning through Controlled Gradient Interventions

Abstract

Multi-objective self-supervised visual learning couples auxiliary losses and geometric regularizers through shared encoders, yet regularization gains need not translate into gains over a base-loss reference. Existing studies document configuration dependence, but removing an auxiliary loss changes updates to several parameter groups, obscuring the contribution of direct encoder updates. We therefore study how direct auxiliary encoder updates influence regularization effects. At the same computational state, we retain or remove only the auxiliary encoder gradient while matching forward losses and all other parameter gradients. Crossing this intervention with regularizer inclusion, alongside a base-loss reference, distinguishes regularization effects, combination effects, and update-path dependence. Paired experiments cover controlled-background predictive learning and natural-image self-distillation across regularizers, auxiliary weights, and training stages. Results show that removing auxiliary encoder gradients reverses the mean effect of Gram regularization on accuracy from +1.98 to −1.70 percentage points (pp) in five-fold linear evaluation of Caltech–UCSD Birds-200-2011 pretraining images after 20 epochs, consistently across six runs. This reversal also occurs with covariance regularization and with Gram at a lower auxiliary weight; these combinations remain below the base-loss reference. On Tiny ImageNet after 100 epochs, Kozachenko–Leonenko regularization lowers mean held-out accuracy by 0.50 pp with auxiliary encoder updates retained, although the full DINOv2-style combination exceeds the reference by 0.92 pp. On a fixed validation set, the interaction changes by −1.04 pp between epochs 20 and 100 (95% confidence interval: −1.84 to −0.25). These comparisons inform regularizer reuse and configuration selection under specified objectives, gradient paths, and training budgets.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-supervised learning, Regularization, Auxiliary losses, Gradient interventions, Training dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_i532OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • This paper investigates how the effects of geometric regularization in self-supervised learning depend on the gradient updates contributed by auxiliary objectives. The authors introduce a controlled gradient intervention that selectively removes auxiliary-loss gradients to the encoder while preserving forward losses and gradients to the predictor or projection head. By combining this intervention with regularizer on/off comparisons and a base-loss reference, the study distinguishes regularization effects, full-combination effects, and their dependence on encoder update paths. Experiments are conducted in an I-JEPA-style predictive learning setting with controlled backgrounds on CUB-200 and Oxford-IIIT Pet, as well as a DINOv2-style self-distillation setting on Tiny ImageNet. The results show that Gram and covariance regularization can improve accuracy when auxiliary encoder gradients are retained but degrade performance when these gradients are removed. Additional experiments with KoLeo regularization demonstrate that component-level improvements and full-combination performance can differ, and that gradient interactions may change across training stages.
Critical strengths
  • The paper presents a careful and well-controlled experimental design for analyzing the interaction between auxiliary losses and regularization. Unlike conventional ablations that remove entire objectives, the proposed intervention isolates direct auxiliary encoder gradients while preserving predictor updates at the same computational state. This provides a more informative comparison of gradient-path dependence and helps distinguish component-level gains from improvements over the base-loss reference.
  • The empirical analysis is relatively thorough, with paired experiments across multiple seeds, regularizers, auxiliary weights, and training stages. The authors also provide detailed implementation checks, statistical intervals, and supplementary results. I particularly appreciate that the paper explicitly distinguishes positive regularization effects from improvements over the base objective, rather than presenting relative gains within a weak configuration as overall performance improvements.
Critical weaknesses
  • While the experimental design is careful, the main finding is demonstrated in a rather limited and weakly performing learning configuration. On CUB-200, the unregularized full-auxiliary model (F0) achieves only 4.02% accuracy after 20 epochs, essentially showing no improvement over initialization, while the base-loss model (N0) reaches 12.80%. Although adding Gram regularization improves F0 to 6.00%, the resulting model still performs substantially worse than the base-loss reference. This raises concerns about how representative the observed reversal is of reasonably optimized self-supervised learning systems. The use of artificially constructed background states and a specialized auxiliary objective further limits the generality of this finding.
  • The broader significance of the results remains somewhat unclear. The main contribution is primarily a controlled experimental protocol rather than a new learning method or a substantially new theoretical explanation. While the reversal is consistent in the short CUB experiments, the evidence becomes less decisive in other settings. The interaction on Oxford-IIIT Pet has a confidence interval crossing zero, and the Tiny ImageNet experiment shows only a small negative interaction at 100 epochs, also with an interval crossing zero. Moreover, the natural-image experiments use a simplified DINOv2-style configuration rather than the standard full implementation. These results are useful observations, but they do not yet establish a broadly applicable understanding of when or why regularization benefits from particular encoder gradient paths.
Questions for authors
  • In the main CUB experiment, F0 barely improves over initialization and performs substantially worse than N0. What explains this large degradation when the auxiliary objective is introduced? Does the regularization reversal persist when the auxiliary objective is better optimized and produces competitive representations?
  • Table 1 shows that F1 and P1 achieve similar accuracy, while F0 and P0 differ considerably. What drives this asymmetry?
  • The natural-image experiment shows a relatively small interaction, with a confidence interval that includes zero. How well do these findings generalize to standard DINOv2 configurations or other natural-image datasets and regularizers?
1 comment
AuthorsOpenReview ↗

Response to Reviewer i532

Thank you for pointing out that assessing how representative the reversal is requires us to examine both how much F0 learns and how the full scheme compares with base training.

Q1. What explains this large degradation when the auxiliary objective is introduced? Does the regularization reversal persist when the auxiliary objective is better optimized and produces competitive representations?

A1. At the reference weights, F0 had already lost much of its feature variance by the midpoint of training, and its classification accuracy stayed near initialization. Its official-test accuracy was 5.21% at initialization, 5.00% at the midpoint, and 5.07% at the endpoint. Total feature variance fell from 17.07 to 1.03 and then to 0.85. From the same starting point, P0 reached 8.71%. N0, trained with the base loss alone, reached 14.43%. With initialization, training schedule, and evaluation protocol held fixed, blocking only the direct auxiliary gradient to the encoder gave P0 higher final accuracy than F0. The gap was already evident halfway through training.

When F0 learned a stronger representation for classification, the full scheme could outperform base training, but the original reversal did not recur. With both the auxiliary-loss weight and the covariance regularization strength at 0.01 times their reference values, F0 was 10.13 percentage points (pp) above initialization and 0.86 pp above N0. Regularization improved F by 1.36 pp and P by 3.44 pp. Despite the larger gain in P, F1 exceeded P1 in all six runs and finished 2.22 pp above N0 on average (95% interval [1.58, 2.86]). We added this comparison to Table 3 in Section 4.2 and to Appendix J.1.

Q2. Table 1 shows that F1 and P1 achieve similar accuracy, while F0 and P0 differ considerably. What drives this asymmetry?

A2. At the reference strength on controlled CUB, Gram improved accuracy in F, the weaker condition, and reduced it in P. In the five-fold evaluation in Table 1 of the original submission, P0 exceeded F0 by 3.58 pp. Adding Gram raised F by 1.98 pp to 6.00% and lowered P by 1.70 pp to 5.90%. The official test showed the same pattern. P0−F0\mathrm{P0}-\mathrm{F0} was +3.64 pp, and F1−P1\mathrm{F1}-\mathrm{P1} was only +0.68 pp.

When we adapted the model on the original CUB photographs without KoLeo, P0 exceeded F0 by 0.45 pp in ridge accuracy. With KoLeo, F1 exceeded P1 by 0.97 pp. The larger gap in this setting came after regularization. Equation (8) in the revision separates the two gaps, and Sections 4.1 and 4.4 give the results for the two settings.

Q3. How well do these findings generalize to standard DINOv2 configurations or other natural-image datasets and regularizers?

A3. We tested KoLeo both when training from scratch on Tiny ImageNet and when adapting a pretrained model on the original CUB photographs. The interaction differed in sign and magnitude across the two tasks. On Tiny ImageNet, after 100 epochs with KoLeo weight 0.10, mean DD across 16 paired runs was −0.41 pp for ridge classification (95% interval [−0.72, −0.10]) and −0.37 pp for weighted k-NN (95% interval [−0.82, +0.07]). The interval for DD also included zero on controlled Pet. Extending controlled CUB training to 75 epochs reproduced the original reversal in only four of six runs.

On the original CUB photographs, we ran six adaptations from the same pretrained DINOv2 ViT-S/14 checkpoint with separate image and patch heads. Both readouts gave a positive interaction in all six runs. Mean DD was +1.43 pp for ridge classification and +2.69 pp for k-NN, yet the full scheme finished 0.66 pp below base adaptation with ridge and 4.14 pp above it with k-NN. Thus the two readouts agreed on the sign of the interaction but differed on whether the full scheme improved accuracy. Both natural-image experiments tested KoLeo, and the dataset, initialization, and prediction heads changed together. The observed directions therefore apply to the respective training configurations and readouts. Extending them to standard-scale DINOv2 or other regularizers requires corresponding paired comparisons (Sections 4.3–4.4, Table 4, and Section 5).

Reviewer_6bRTOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the effect of a geometric regularizer in multi-objective SSL depends on the direct gradient that an auxiliary loss sends to the encoder. It proposes a "matched gradient intervention": at the same computational state, condition F keeps the full auxiliary gradient, while P replaces the encoder gradient with the cached gradient of the base loss plus regularizer, so forward losses and head/predictor gradients are identical and only the direct auxiliary-to-encoder gradient differs. A base-loss-only condition N serves as a reference. Crossing {N, F, P} with regularizer on/off gives six conditions, from which the paper defines a regularization effect δX\delta_X, an F/P interaction D=δF−δPD=\delta_F-\delta_P, and a combination effect G=YF,1−YN,0G = Y_{F,1}-Y_{N,0}. The main setting is I-JEPA-style predictive learning with a randomly initialized ViT-S/16 on CUB-200 (5,952 images) with synthetic sinusoidal backgrounds, a custom state-residual whitened alignment loss as the auxiliary term, and a random-group Gram (or covariance) regularizer, evaluated with ridge probes over six paired runs. At 20 epochs, Gram raises five-fold accuracy under F (+1.98 pp) and lowers it under P (−1.70 pp) in all six runs; the reversal also appears with a covariance regularizer and at α=0.25\alpha=0.25. A second setting, DINO + iBOT + KoLeo on Tiny ImageNet (eight runs, 100 epochs), shows KoLeo lowering accuracy under F (−0.50 pp) while the full combination beats the DINO-only reference (+0.92 pp); the interaction is not significant at the endpoint and changes sign across training stages.

Critical strengths

S1. A clean, well-verified intervention. Separating the direct auxiliary-to-encoder gradient from the auxiliary loss's effect on the predictor/head, while matching forward values, head gradients and optimizer updates at the same state (Appendix C, including bitwise update checks and separate per-group clipping), is a more precise ablation than the usual "drop the loss" comparison. The paper is also explicit that F/P measures cumulative training effects rather than matched trajectories (Sec. 3.2, Appendix G.2), which is honest and correct.

S2. Careful paired statistics and transparent reporting. Independent runs are the statistical unit, all per-run values are reported (Appendix F), intervals are paired t-intervals, repeated measurements of the same models are not treated as independent, and the paper includes label-shuffling, refitting and ridge-coefficient sensitivity checks. This level of reporting is better than typical SSL ablation papers and makes the weaknesses below verifiable.

Critical weaknesses

W1. The headline reversal is measured on near-random, collapsed representations and is explained by collapse rescue rather than by the gradient path.

a. Accuracy is barely above a random encoder. On the CUB official test set, a randomly initialized ViT-S/16 already gives 5.2–5.4% (Table 27, epoch 0). At the 20-epoch endpoint, F0 = 5.08%, F1 = 7.48%, P0 = 8.71%, P1 = 6.80% (Table 10). F0 is at or below random-initialization accuracy (also on five-fold: 4.02% vs. 4.10% at init, Table 16), and the full objective F1 is about 2 pp above it. For reference, an ImageNet-supervised ResNet-18 reaches 54.3% on the same rendered data (Table 23), and even the base loss alone reaches only 14.4%. The "20-epoch prespecified endpoint" is 7,440 steps with a 3,720-step warm-up and a peak learning rate of 3.1×10−53.1\times10^{-5} (Table 6), so half of training is warm-up. A ViT trained from scratch on ~6k images is known to be in a data-starved regime (e.g., Liu et al., NeurIPS 2021).

b. The auxiliary loss collapses the encoder, and Gram simply undoes it. F0 has effective rank 3.69 and total variance 0.88 (Table 4); Gram raises effective rank to 90 under F. The same Gram weight is strongly harmful wherever learning actually happens: δN=−6.85\delta_N = -6.85 pp (five-fold) and −7.52 pp (test). So the observation "Gram helps F, hurts P" is consistent with a simple explanation: a collapse-preventing regularizer helps only when the objective it is paired with causes dimensional collapse (Jing et al., 2021), and hurts otherwise.

c. Per-run data support this explanation directly. Across runs, δF\delta_F is almost perfectly negatively correlated with F0 accuracy: r = −0.95 (CUB 20 ep., Table 10), −0.98 (CUB 75 ep., Table 13), −0.98 (Pet, Table 14). In the only runs where F0 escapes collapse, the reversal disappears: CUB75 run 3 (F0 = 10.64%) gives δF=−2.24\delta_F=-2.24 pp and D=−0.33D=-0.33 pp; Pet run 5 (F0 = 17.24%) gives δF=−1.46\delta_F=-1.46 pp and D=−2.57D=-2.57 pp. (Part of this correlation is mechanical since δF=F1−F0\delta_F = F_1 - F_0, but F1 varies little across runs, so δF\delta_F is essentially determined by whether F0 collapsed.) The paper's attribution of the reversal to "direct auxiliary encoder updates" (Sec. 4.1) is therefore literally true as an intervention, but the mechanism is collapse, which is a property of this particular custom auxiliary loss, not a general lesson about regularizer reuse.

d. Regularizer weights are untuned. Gram is fixed at λ1=1\lambda_1=1 and the covariance weight is calibrated to Gram's gradient norm (Appendix B.2), so both are set at a strength that costs ~7 pp in the base configuration. An interaction measured at a single, clearly harmful weight says little about how practitioners would combine these components.

W2. The realistic setting does not support the claims, and the contribution is mostly a protocol whose usefulness is not demonstrated.

a. Effects in the natural-image setting are within noise. On Tiny ImageNet at 100 epochs, D=−0.454D = -0.454 pp with a 95% interval of [−0.910, 0.002], below the authors' own pre-specified 1 pp practical threshold (Sec. 4.3); δF=−0.50\delta_F=-0.50 pp barely excludes zero. The validation-set interaction is positive at 20 epochs (+0.78), negative at 60 (−0.30), and not distinguishable from zero at 80 and 100 (Table 21). At 20 epochs, its magnitude also roughly doubles (0.76 → 1.48 pp) depending only on the ridge coefficient (Table 24). This is better described as "no stable interaction" than as "stage-dependent interaction."

b. Other extensions do not replicate. On Pet, D=+2.16D = +2.16 pp with interval [−0.42, 4.75]; at CUB 75 epochs, δF\delta_F's interval is [−0.81, 5.32] and only 4/6 runs have δF>0\delta_F>0 (Table 2). The only consistently significant results are the CUB 20-epoch variants, i.e., the collapsed regime in W1. Many effects are tested without any multiplicity control, and the sign-test p = 0.03125 reported as evidence is the smallest value attainable with n = 6.

c. Limited novelty and no actionable guidance. Detaching an auxiliary loss from the encoder while keeping its head update is a standard stop-gradient ablation; replacing or removing gradient terms in SSL was studied by UniGrad (Tao et al., 2022), and gradient-level analysis of auxiliary losses goes back at least to Du et al. (2018). The 2 × 3 factorial with a base reference is a standard design. The conclusion ("component and combination effects require distinct references") is a methodological truism, and the paper gives no rule for when a regularizer will transfer. The geometry analysis (effective rank rising while accuracy falls) also needs to engage with RankMe (Garrido et al., ICML 2023), which proposed effective rank as a label-free SSL quality signal; the observed disconnect is relevant to that work but not discussed. KoLeo should be credited to Sablayrolles et al. (ICLR 2019).

Questions for authors

Clarifications (could change my assessment):

Q1. Is the positive δF\delta_F explained by collapse? Please report, for every run and setting, F0's effective rank together with δF\delta_F and DD, and explain CUB75 run 3 and Pet run 5, where F0 does not collapse and the reversal vanishes or flips.

Q2. Why is F0 at or below random-initialization accuracy at 20 epochs? Does the custom state-residual whitened alignment loss collapse the encoder by design (e.g., through the eigenvalue floor or the aggregated 4 × 384 term), and does it ever improve over N0 in any configuration?

Q3. Why was 20 epochs (7,440 steps, half of it warm-up) pre-specified as the primary endpoint, and why is five-fold accuracy on the pretraining images, rather than the official test split, the result quoted in the abstract? Was the pre-specification recorded anywhere (e.g., a registration or timestamped plan)?

Q4. How sensitive is the reversal to the regularizer weight? Does it persist when Gram and cov weights are chosen so that δN≥0\delta_N \geq 0 (i.e., the regularizer is not harmful in the base configuration)?

Experiments needed to support the claims:

Q5. The CUB/Pet analysis in a regime where the encoder actually learns, e.g., starting from a pretrained I-JEPA/DINO checkpoint, training on a larger dataset, or extending training until N0 approaches typical I-JEPA probe accuracy, with a standard auxiliary objective rather than the custom one.

Q6. In the Tiny ImageNet setting, more runs or a larger effect size test so that the endpoint interaction can be distinguished from zero, plus at least one more KoLeo weight (DINOv2 uses 0.1, but the interaction may be weight-dependent).

Q7. A correction for multiple comparisons across the reported effects, or an explicit separation between the single confirmatory test and exploratory analyses.

Q8. A discussion of how the results relate to RankMe-style effective-rank diagnostics, since the paper's own Tables 4 and 25 show effective rank and accuracy moving in opposite directions.

Suggestion for future work (not affecting my score): applying the intervention to widely used multi-objective recipes (e.g., DINOv2 with separate image/patch heads, or I-JEPA + VICReg-style regularization as in Mo & Tong, 2024) at standard scale would make the protocol much more useful to practitioners.

References -

Liu, Y., Sangineto, E., Bi, W., Sebe, N., Lepri, B., De Nadai, M. Efficient Training of Visual Transformers with Small Datasets. NeurIPS 2021.

Jing, L., Vincent, P., LeCun, Y., Tian, Y. Understanding Dimensional Collapse in Contrastive Self-supervised Learning. ICLR 2022 (arXiv 2021).

Garrido, Q., Balestriero, R., Najman, L., LeCun, Y. RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank. ICML 2023.

Sablayrolles, A., Douze, M., Schmid, C., Jégou, H. Spreading Vectors for Similarity Search. ICLR 2019.

Du, Y., Czarnecki, W. M., Jayakumar, S. M., Farajtabar, M., Pascanu, R., Lakshminarayanan, B. Adapting Auxiliary Losses Using Gradient Similarity. arXiv:1812.02224, 2018.

Tao, C. et al. Exploring the Equivalence of Siamese Self-Supervised Learning via a Unified Gradient Framework (UniGrad). CVPR 2022.

Mo, S., Tong, S. Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning. NeurIPS 2024.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6bRT (1/2)

Thank you for identifying a possible link between the weak baseline, contraction of the representations and regularization gains in individual runs. We paired each run’s effective rank with its accuracy change and retested the reversal at weights that improve base training.

Q1. “The auxiliary loss collapses the encoder, and Gram simply undoes it.” […] “CUB75 run 3 and Pet run 5, where F0 does not collapse and the reversal vanishes or flips.”

A1. The reference-weight CUB results are consistent with Gram partly offsetting a weak baseline. Neither run you identified showed the original reversal. In CUB run 3 at epoch 75, F0 accuracy was 10.64% and effective rank was 5.41. Gram lowered F and P accuracy by 2.24 and 1.91 percentage points (pp). In Pet run 5, F0 reached 17.24% at rank 1.97. F fell 1.46 pp and P rose 1.10 pp. Pet run 6 had an even lower rank (1.40), yet F rose 2.98 pp. Both Pet runs had low rank, with opposite effects on F accuracy.

Since δF=F1−F0\delta_F=\mathrm{F1}-\mathrm{F0}, lower F0 gives a larger δF\delta_F when F1 is similar across runs. Original Tables 13–14 give these counterexamples. Appendix J.5 lists F0 accuracy, effective rank, total variance, δF\delta_F, δP\delta_P and DD for each run in the controlled CUB and Pet series reported there and the first eight Tiny runs.

Q2. Why is F0 at or below random-initialization accuracy at 20 epochs? Does the custom state-residual whitened alignment loss collapse the encoder by design (e.g., through the eigenvalue floor or the aggregated 4×3844\times384 term), and does it ever improve over N0 in any configuration?

A2. At the high auxiliary weight, F0 had lower prediction loss than N0, but its representation variance fell from 17.07 to 0.85 and official-test accuracy from 5.21% to 5.07%. With the same small sample and warm-up schedule, P0 rose from the same starting point to 8.71%. At auxiliary weight 0.01 in the weak covariance experiment, unregularized F0 exceeded initialization by 10.13 pp and N0 by 0.86 pp. The same auxiliary objective can thus improve accuracy at a lower weight. We scaled the entire auxiliary term, retaining the whitening floor based on the largest eigenvalue of the target-residual Gram matrix and the same aggregation rule for the 4×3844\times384 batch-mean residual (Section 3.1). Appendix J.1 gives the trajectories. The externally supervised pretrained ResNet-18 in Table 23 of the original submission achieved about 54.3% accuracy, confirming that the constructed images retained class information.

Q3. Why was 20 epochs (7,440 steps, half of it warm-up) pre-specified as the primary endpoint, and why is five-fold accuracy on the pretraining images, rather than the official test split, the result quoted in the abstract? Was the pre-specification recorded anywhere (e.g., a registration or timestamped plan)?

A3. Our internal plan set epoch 20 as an early comparison point (step 7,440, with the first 3,720 steps used for warm-up, original Table 6). The supplement’s configs/cub_matched/SCIENTIFIC_CONTRACT.json has an internal creation date of September 9, 2026 and specifies six paired runs, the step-7,440 endpoint, image-grouped ridge classification, the primary contrast DD and the two-sided sign test.

The original abstract reported five-fold accuracy because it was the primary evaluation. Those images were used in self-supervised pretraining. The revised abstract reports the official-test results from original Table 1 (Gram raised F by 2.40 pp and lowered P by 1.91 pp). Section 4.1 retains the five-fold results for the same models so readers can check that the effect directions agree.

Q4. How sensitive is the reversal to the regularizer weight? […] “the regularizer is not harmful in the base configuration”

A4. The reversal persists at the original auxiliary weight even when weak Gram improves base training. We selected both Gram and covariance at 0.01 times their reference weights using only training-side validation data from two N runs. On the official test, N gained 1.61 and 1.69 pp, respectively, across six paired runs and 1.91 and 1.68 pp across the four held out from selection.

At α=1\alpha=1, F0 remained weak. Weak Gram raised F by 0.39 pp and lowered P by 2.18 pp. Weak covariance raised F and P by 2.86 and 1.10 pp. At α=0.01\alpha=0.01, both weak regularizers raised mean accuracy in both conditions. Thus, even at a strength that benefits base training, the effect of the same regularizer depends on the auxiliary weight in F and P. Revised Table 3 gives the crossed comparisons and Appendix J.1 the selection rule and four-run check.

AuthorsOpenReview ↗

Response to Reviewer 6bRT (2/2)

Q5. “starting from a pretrained I-JEPA/DINO checkpoint” […] “with a standard auxiliary objective rather than the custom one.”

A5. On the original CUB photographs, all six adaptations from the same pretrained DINOv2 ViT-S/14 gave positive interactions for both readouts. We trained for 20 epochs with DINO image and iBOT patch objectives and separate heads. Mean KoLeo DD was +1.43 pp (95% interval [1.24, 1.61]) for ridge and +2.69 pp [2.19, 3.18] for weighted k-NN. Starting accuracies (85.74% and 87.11%, respectively) exceeded every adaptation condition’s mean. For each readout, mean F and P effects had the same sign, and the F increment exceeded P’s in every run.

Revised Table 4 and Appendix J.2 report pretrained adaptation on natural CUB. Both trajectories extended to 150 epochs in original Appendix I were N0. With target-token pooling and a ridge classifier, mean official-test accuracy rose another 1.96 pp from epoch 75 to 150.

Q6. “the endpoint interaction can be distinguished from zero, plus at least one more KoLeo weight”

A6. Across 16 paired runs at weight 0.10 and epoch 100, holdout DD was −0.41 pp (95% interval [−0.72, −0.10]) for ridge and −0.37 pp [−0.82, +0.07] for weighted k-NN. Only the ridge interval excludes zero. Both lie within ±1\pm 1 pp. In the eight runs paired across weights, lowering the weight from 0.10 to 0.03 changed DD by +0.035 and +0.180 pp, respectively. Both intervals for these changes span zero (Table 4, Appendices J.3 and J.6).

Across eight runs, mean development-set DD went from +0.78 pp at epoch 20 to −0.26 pp at epoch 100. At epoch 20, varying only the ridge coefficient gave +0.75 to +1.48 pp (original Tables 21 and 24). We tracked training on the development set, then used the selected, frozen ridge classifier for the epoch-100 holdout evaluation (Section 4.3, Appendices H.3 and J.6).

Q7. A correction for multiple comparisons across the reported effects, or an explicit separation between the single confirmatory test and exploratory analyses.

A7. The two-sided sign test for five-fold DD at epoch 20 on controlled CUB was the only prespecified test. All six paired differences were positive (p=0.03125p=0.03125, the smallest two-sided value possible for six nonzero pairs). All other 95% paired intervals are exploratory and unadjusted for multiplicity. At 75 CUB epochs, only four of six runs showed the original reversal, with a δF\delta_F interval of [−0.81, +5.32] pp. On Pet, DD was +2.16 pp [−0.42, +4.75]. See p. 6 and Appendix E.3.

Q8. A discussion of how the results relate to RankMe-style effective-rank diagnostics, since the paper's own Tables 4 and 25 show effective rank and accuracy moving in opposite directions.

A8. In these settings, effective rank measures how evenly variance is distributed across spectral directions. Accuracy measures how well class information can be read out. In original Table 4, Gram increased mean effective rank in F, P and N on controlled CUB, but their five-fold accuracies changed by +1.98, −1.70 and −6.85 pp, respectively. RankMe uses the singular values of the embedding matrix. We use the eigenvalues of the centered covariance matrix, so the two measures weight spectral components differently. Sections 2 and 5 explain the distinction. Table 5 reports both metrics.

Q9. “Limited novelty and no actionable guidance.”

A9. We retain the auxiliary update to the prediction head and isolate how the effect of regularization depends on the auxiliary loss’s direct gradient to the encoder. In 16 same-state, single-step F/P checks (original Appendix C.2), the maximum absolute post-update differences in head parameters and optimizer states were both zero. With weak Gram and α=1\alpha=1, the four paired runs held out from weight selection gave δF−δN=−1.46\delta_F-\delta_N=-1.46 pp and δF−δP=+2.86\delta_F-\delta_P=+2.86 pp. N removes the entire auxiliary loss. P retains auxiliary head updates. The opposite signs on the same runs show that whole-loss ablation includes changes to both the head and the encoder. Comparing F1 with N0 tells us whether the full scheme improves on base training.

Section 2 compares our design with UniGrad’s gradient replacement and Du et al.’s use of main–auxiliary gradient similarity to select updates. Section 3.1 now credits Sablayrolles et al. (2019) for KoLeo. Section 4.3 gives the example above.

Reviewer_LypSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a geometric regularizer's benefit depends on the auxiliary loss's direct gradient into the encoder. F keeps that gradient and P drops it at the same computational state, with forward losses and predictor/head gradients matched. Crossing F/P with a regularizer switch, plus a base-loss-only reference N, defines the interaction D and the combination effect G (Eq. 4). On CUB with controlled backgrounds at 20 epochs, Gram helps under F (+1.98 pp) and hurts under P (-1.70 pp) in all six runs, and this repeats with a covariance regularizer and at α = 0.25. On DINOv2-style Tiny ImageNet, KoLeo gives δF = -0.50 but G = +0.92.

Critical strengths

I think the intervention is done carefully. P still gives the predictor the same auxiliary gradient, and App. C.2 reports 16 same-state update checks with a maximum absolute difference of 0 in predictor parameters and optimizer states. Everything is paired within run, with per-run tables in App. F, so I trust the numbers enough to argue with the interpretation.

I also like the N reference. It exposes what a leave-one-out ablation would hide: every regularized CUB combination sits well below base-only training (G = -6.80 in Table 1, -8.17 for cov, -7.46 at α = 0.25), while on Tiny ImageNet the signs flip (δF < 0, G > 0, Table 3).

Critical weaknesses

My main concern is that the headline interaction mostly comes from a baseline arm that never trains. D = δF - δP equals (F1 - P1) + (P0 - F0). In Table 1 (five-fold), F1 - P1 = 6.00 - 5.90 = 0.10 while P0 - F0 = 7.60 - 4.02 = 3.58, about 97% of D = 3.67 (84% on unseen images). F0 ends at 4.02% from a 4.10% initialization while N0 reaches 12.80% (p.6, Table 16). The paper notes this but never decomposes D, so I read the "reversal" as mostly Gram rescuing an F0 run that doesn't learn. Where the regularized arms do differ (F1 - P1 = 2.13 at CUB75, Table 13; 1.81 on Pet, Table 14), the reversal stops being clean: at CUB75 δF's interval crosses zero with 4/6 runs positive, and on Pet D's interval crosses zero (Table 2).

Second, I'm not convinced this generalizes. All CUB accuracies sit between 4% and 15% on synthetic-background composites, the Gram weight is never varied, and the only natural-image test gives D = -0.454 [-0.910, 0.002] (Table 3), below the paper's own 1 pp threshold (p.8).

Questions for authors
  1. Could you report D = (F1 - P1) + (P0 - F0) for every setting, and explain why F0 stays at initialization while P0 and N0 learn? If F0 is collapsing, is the CUB20 reversal more than Gram preventing that?
  2. Is there a setting (another Gram weight, α, or schedule) where F0 trains normally and the reversal holds with F1 - P1 clearly positive? That result would most change my score.
  3. I'm curious whether the Gram groups act as fixed per-image labels (App. B.1: "randomly generated and fixed within each independent training run"). If so, isn't Gram partly a random-label objective, which might explain why N1, F1 and P1 all land near 6% (Table 4)?
  4. KoLeo is judged only by a ridge probe on CLS features, while DINOv2 reports its gain mostly in nearest-neighbour search. Does the negative δF on Tiny ImageNet hold under k-NN?
3 comments
AuthorsOpenReview ↗

Response to Reviewer LypS

Thank you for pointing out that comparing the F–P gaps before and after regularization lets us assess how much of the CUB reversal comes from a weak F0.

Q1. Could you report D=(F1−P1)+(P0−F0)D=(\mathrm{F1}-\mathrm{P1})+(\mathrm{P0}-\mathrm{F0}) for every setting, and explain why F0 stays at initialization while P0 and N0 learn? If F0 is collapsing, is the CUB20 reversal more than Gram preventing that?

A1. The weak F0 accounts for most of the interaction. After 20 epochs on controlled CUB, P0 exceeded F0 by 3.58 percentage points (pp) in five-fold accuracy without Gram. With Gram, F1 exceeded P1 by only 0.10 pp. The corresponding gaps on the official test set were 3.64 and 0.68 pp, again with the larger gap before regularization.

F0’s official-test accuracy was 5.21% at initialization, 5.00% midway through training and 5.07% at the endpoint. Its total feature variance fell from 17.07 to 1.03 and then 0.85. From the same starting point, P0 reached 8.71%, and N0, trained on the base loss alone, reached 14.43%. Gram improved F and hurt P in all six paired runs, but F1 still finished 6.96 pp below N0.

Equation (8) expresses the interaction as D=(F1−P1)+(P0−F0)D=(\mathrm{F1}-\mathrm{P1})+(\mathrm{P0}-\mathrm{F0}). Appendix J.4 gives both gaps for the main settings and training stages. Table 3 covers the reduced weights, and Section 4.4 covers natural-image CUB. For Tiny ImageNet, Table 4 reports the pooled interaction and Appendix J.6 gives both gaps for runs 9–16.

Q2. Is there a setting (another Gram weight, α\alpha, or schedule) where F0 trains normally and the reversal holds with F1−P1\mathrm{F1}-\mathrm{P1} clearly positive?

A2. No controlled CUB setting we tested met all three conditions. With the auxiliary weight and covariance regularization strength each at 0.01 times its reference value, F0 exceeded its initial accuracy by 10.13 pp and N0 by 0.86 pp. F1 exceeded P1 in all six paired runs, by 1.18 pp on average (95% interval [0.87, 1.49]), and exceeded N0 by 2.22 pp. Both conditions benefited from regularization, with gains of 1.36 pp in F and 3.44 pp in P. At the original auxiliary weight, weakening Gram alone raised F by 0.39 pp and lowered P by 2.18 pp. In that setting F0 remained weak, and F1 finished 1.28 pp below P1 (Table 3, Appendix J.1).

F1 exceeded P1 by 2.13 pp on CUB at 75 epochs and by 1.81 pp on Pet. The reversal occurred in only four of six CUB runs, and the interval for DD on Pet included zero. Whether F1 outperforms P1 and whether regularization has opposite effects in F and P are distinct questions (Section 4.3).

Q3. I'm curious whether the Gram groups act as fixed per-image labels […] If so, isn't Gram partly a random-label objective, which might explain why N1, F1 and P1 all land near 6% (Table 4)?

A3. Each image does have a fixed random group assignment within a run, independent of its class. The Gram loss constrains the geometry of the four group means in feature space. It has no classification term that predicts an individual image’s group (original manuscript, Appendices B.1–B.2).

Across five runs with the base objective, mean five-fold accuracy was 12.93% without Gram, 5.55% with fixed groups shared across both directions, and 5.79% with balanced groups resampled independently for each direction at every step. The similar drop after resampling shows that persistent group assignments are unnecessary for the loss in accuracy under the base objective (Appendix J.1).

Q4. Does the negative δF\delta_F on Tiny ImageNet hold under k-NN?

A4. The negative effect did not consistently carry over to k-NN. At KoLeo weight 0.10, both readouts used the same final teacher CLS features at epoch 100 and the same 10,000 holdout images. Across the original eight runs, KoLeo reduced F’s mean ridge accuracy by 0.50 pp and increased its weighted cosine k-NN accuracy by 0.59 pp. Across all 16 runs, the effects were −0.54 pp for ridge (95% interval [−0.80, −0.29]) and +0.23 pp for k-NN (95% interval [−0.11, +0.56]). The corresponding interactions DD were −0.41 and −0.37 pp, with both intervals within ±1\pm 1 pp.

On the original CUB photographs, all six adaptations of the pretrained model gave positive interactions under both readouts. Mean DD was +1.43 pp for ridge and +2.69 pp for k-NN. KoLeo reduced mean accuracy in both F and P under ridge and increased it in both under k-NN. For these same models, a larger regularization effect in F than in P is distinct from a gain within F (Table 4, Appendices J.2 and J.6).

Thank you for the response and the extra runs, which answer my questions directly. They also confirm my main concern: the weak F0 accounts for most of the CUB interaction, no setting keeps the reversal once F0 trains normally (at 0.01x both arms gain from regularization), and the Tiny ImageNet effect does not hold under k-NN. I think based on the rebuttal and other reviewer's reviews, I will maintain my original score accordingly.

AuthorsOpenReview ↗

Clarification on the contribution and natural-image evidence

Thank you for reviewing our response and confirming that the additional experiments directly answered your questions.

On the original CUB photographs, we conducted six paired adaptations from the same pretrained DINOv2 encoder, using separate image and patch prediction heads. Both ridge classification and weighted k-NN yielded positive interactions in all six paired runs, with means of +1.43 percentage points (pp) (95% interval [1.24, 1.61]) and +2.69 pp (95% interval [2.19, 3.18]), respectively. For each readout, the mean regularization effects in F and P had the same sign but differed in magnitude. Under both readouts, the gap after regularization, F1−P1\mathrm{F1}-\mathrm{P1}, was larger than the gap before regularization, P0−F0\mathrm{P0}-\mathrm{F0}. Thus, dependence of regularization effects on update paths also occurs when adapting a pretrained model to natural images. In this setting, the larger part of the interaction comes from the gap after regularization (Section 4.4, Table 4).

Section 4.3 further shows how the choice of ablation affects the assessment. With weak Gram and an auxiliary weight of 1, the four paired runs not used for weight selection gave a difference in regularization effects of δF−δN=−1.46\delta_F-\delta_N=-1.46 pp under whole-loss ablation and δF−δP=+2.86\delta_F-\delta_P=+2.86 pp under the path intervention. N removes the entire auxiliary loss, eliminating its updates to both the encoder and the prediction head. P retains auxiliary training of the prediction head and blocks only the auxiliary loss's direct gradient to the encoder. The opposite signs in the same set of training runs show that these two ablations lead to different assessments of regularization effects.

Our matched-gradient intervention explicitly compares regularization effects with and without the direct auxiliary gradient to the encoder. We then compare F1 with N0 to evaluate the full scheme. These experiments provide empirical evidence for this distinction, allowing readers to identify the update conditions under which regularization gains occur and determine whether the full scheme improves classification accuracy over base training.

Thank you again for your careful review and specific suggestions, which have helped us improve the experimental analysis and clarify our conclusions and the conditions under which they hold.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.