Clarifications (could change my assessment):
Q1. Is the positive δF explained by collapse? Please report, for every run and setting, F0's effective rank together with δF and D, and explain CUB75 run 3 and Pet run 5, where F0 does not collapse and the reversal vanishes or flips.
Q2. Why is F0 at or below random-initialization accuracy at 20 epochs? Does the custom state-residual whitened alignment loss collapse the encoder by design (e.g., through the eigenvalue floor or the aggregated 4 × 384 term), and does it ever improve over N0 in any configuration?
Q3. Why was 20 epochs (7,440 steps, half of it warm-up) pre-specified as the primary endpoint, and why is five-fold accuracy on the pretraining images, rather than the official test split, the result quoted in the abstract? Was the pre-specification recorded anywhere (e.g., a registration or timestamped plan)?
Q4. How sensitive is the reversal to the regularizer weight? Does it persist when Gram and cov weights are chosen so that δN≥0 (i.e., the regularizer is not harmful in the base configuration)?
Experiments needed to support the claims:
Q5. The CUB/Pet analysis in a regime where the encoder actually learns, e.g., starting from a pretrained I-JEPA/DINO checkpoint, training on a larger dataset, or extending training until N0 approaches typical I-JEPA probe accuracy, with a standard auxiliary objective rather than the custom one.
Q6. In the Tiny ImageNet setting, more runs or a larger effect size test so that the endpoint interaction can be distinguished from zero, plus at least one more KoLeo weight (DINOv2 uses 0.1, but the interaction may be weight-dependent).
Q7. A correction for multiple comparisons across the reported effects, or an explicit separation between the single confirmatory test and exploratory analyses.
Q8. A discussion of how the results relate to RankMe-style effective-rank diagnostics, since the paper's own Tables 4 and 25 show effective rank and accuracy moving in opposite directions.
Suggestion for future work (not affecting my score): applying the intervention to widely used multi-objective recipes (e.g., DINOv2 with separate image/patch heads, or I-JEPA + VICReg-style regularization as in Mo & Tong, 2024) at standard scale would make the protocol much more useful to practitioners.
References -
Liu, Y., Sangineto, E., Bi, W., Sebe, N., Lepri, B., De Nadai, M. Efficient Training of Visual Transformers with Small Datasets. NeurIPS 2021.
Jing, L., Vincent, P., LeCun, Y., Tian, Y. Understanding Dimensional Collapse in Contrastive Self-supervised Learning. ICLR 2022 (arXiv 2021).
Garrido, Q., Balestriero, R., Najman, L., LeCun, Y. RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank. ICML 2023.
Sablayrolles, A., Douze, M., Schmid, C., Jégou, H. Spreading Vectors for Similarity Search. ICLR 2019.
Du, Y., Czarnecki, W. M., Jayakumar, S. M., Farajtabar, M., Pascanu, R., Lakshminarayanan, B. Adapting Auxiliary Losses Using Gradient Similarity. arXiv:1812.02224, 2018.
Tao, C. et al. Exploring the Equivalence of Siamese Self-Supervised Learning via a Unified Gradient Framework (UniGrad). CVPR 2022.
Mo, S., Tong, S. Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning. NeurIPS 2024.
Response to Reviewer i532
Thank you for pointing out that assessing how representative the reversal is requires us to examine both how much F0 learns and how the full scheme compares with base training.
Q1. What explains this large degradation when the auxiliary objective is introduced? Does the regularization reversal persist when the auxiliary objective is better optimized and produces competitive representations?
A1. At the reference weights, F0 had already lost much of its feature variance by the midpoint of training, and its classification accuracy stayed near initialization. Its official-test accuracy was 5.21% at initialization, 5.00% at the midpoint, and 5.07% at the endpoint. Total feature variance fell from 17.07 to 1.03 and then to 0.85. From the same starting point, P0 reached 8.71%. N0, trained with the base loss alone, reached 14.43%. With initialization, training schedule, and evaluation protocol held fixed, blocking only the direct auxiliary gradient to the encoder gave P0 higher final accuracy than F0. The gap was already evident halfway through training.
When F0 learned a stronger representation for classification, the full scheme could outperform base training, but the original reversal did not recur. With both the auxiliary-loss weight and the covariance regularization strength at 0.01 times their reference values, F0 was 10.13 percentage points (pp) above initialization and 0.86 pp above N0. Regularization improved F by 1.36 pp and P by 3.44 pp. Despite the larger gain in P, F1 exceeded P1 in all six runs and finished 2.22 pp above N0 on average (95% interval [1.58, 2.86]). We added this comparison to Table 3 in Section 4.2 and to Appendix J.1.
Q2. Table 1 shows that F1 and P1 achieve similar accuracy, while F0 and P0 differ considerably. What drives this asymmetry?
A2. At the reference strength on controlled CUB, Gram improved accuracy in F, the weaker condition, and reduced it in P. In the five-fold evaluation in Table 1 of the original submission, P0 exceeded F0 by 3.58 pp. Adding Gram raised F by 1.98 pp to 6.00% and lowered P by 1.70 pp to 5.90%. The official test showed the same pattern. P0−F0 was +3.64 pp, and F1−P1 was only +0.68 pp.
When we adapted the model on the original CUB photographs without KoLeo, P0 exceeded F0 by 0.45 pp in ridge accuracy. With KoLeo, F1 exceeded P1 by 0.97 pp. The larger gap in this setting came after regularization. Equation (8) in the revision separates the two gaps, and Sections 4.1 and 4.4 give the results for the two settings.
Q3. How well do these findings generalize to standard DINOv2 configurations or other natural-image datasets and regularizers?
A3. We tested KoLeo both when training from scratch on Tiny ImageNet and when adapting a pretrained model on the original CUB photographs. The interaction differed in sign and magnitude across the two tasks. On Tiny ImageNet, after 100 epochs with KoLeo weight 0.10, mean D across 16 paired runs was −0.41 pp for ridge classification (95% interval [−0.72, −0.10]) and −0.37 pp for weighted k-NN (95% interval [−0.82, +0.07]). The interval for D also included zero on controlled Pet. Extending controlled CUB training to 75 epochs reproduced the original reversal in only four of six runs.
On the original CUB photographs, we ran six adaptations from the same pretrained DINOv2 ViT-S/14 checkpoint with separate image and patch heads. Both readouts gave a positive interaction in all six runs. Mean D was +1.43 pp for ridge classification and +2.69 pp for k-NN, yet the full scheme finished 0.66 pp below base adaptation with ridge and 4.14 pp above it with k-NN. Thus the two readouts agreed on the sign of the interaction but differed on whether the full scheme improved accuracy. Both natural-image experiments tested KoLeo, and the dataset, initialization, and prediction heads changed together. The observed directions therefore apply to the respective training configurations and readouts. Extending them to standard-scale DINOv2 or other regularizers requires corresponding paired comparisons (Sections 4.3–4.4, Table 4, and Section 5).