- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SAG-VICReg, which is a random walk-guided method for self-supervised learning. Motivated by a spectral perspective, this paper constructs a similarity graph and samples positive from neighboring images. SAG-VICReg uses similarity-weighted invariance between sampled pairs. This paper also introduces representation comparison using LCA and cophenetic correlations. Experimental results show that the proposed method outperforms existing methods.
Critical strengths
-
The comprehensive experiments show the effectiveness of the proposed method.
-
This paper is easy to follow with clear writing.
Critical weaknesses
-
My main concern is the pairing mechanism using neighboring representations is well established, such as NNCLR and I-Con. I am curious whether the proposed SAG-VICReg would still hold an advantage if they were applied to VICReg.
-
The experimental results on ImageNet-1k are conducted only with 250 epochs. The authors should provide more epochs and recent SSL methods to validate the effectiveness of SAG-VICReg.
Questions for authors
See weakness.
5 comments
W2 (Part 1/2)
W2 — ImageNet-1k training duration and recent SSL methods
We agree that longer training is the common setup, and we acknowledge this limitation in the paper. Training all four methods (SimCLR, Barlow Twins, VICReg, and SAG-VICReg) for 1,000 epochs on ImageNet-1k, with multiple runs each, was beyond our computational budget. Therefore, all methods in Table 3 were trained for 250 epochs under the same protocol, ensuring a fair comparison. Importantly, our VICReg baseline already reaches 71.74% linear top-1 accuracy after 250 epochs, within 1.5 percentage points of the 73.2% reported by the original VICReg paper after 1,000 epochs. SAG-VICReg further improves this to 72.46%, demonstrating gains over an already competitive baseline. Moreover, SAG-VICReg improves VICReg across all seven ImageNet evaluation metrics in Table 3.
Regarding recent SSL methods, Table 5 already evaluates DINOv2, TIPSv2, and CAPI, showing improvements in all 20 configurations, with gains of up to +1.65 percentage points.
To further strengthen this evaluation, we conducted additional experiments with DINOv2 ViT-S/14 and TIPSv2 ViT-B/14 across six additional datasets: EuroSAT, Food-101, RESISC45, GTSRB, DTD, and Tiny-ImageNet. For each backbone, both variants use the same native objective, training data, and fine-tuning schedule; the only intervention is our random walk pairing. We evaluate both 20% and full-data fine-tuning, with five runs per configuration.
Table R1. Fine-tuning on 20% of the training data. Gains in k-NN top-1 accuracy (percentage points) over the corresponding vanilla fine-tune.
| Backbone | EuroSAT | Food-101 | RESISC45 | GTSRB |
|---|---|---|---|---|
| DINOv2 ViT-S/14 | +0.84 | +0.79 | +0.75 | +0.74 |
| TIPSv2 ViT-B/14 | +0.68 | +0.51 | +0.90 | +1.64 |
Table R2. Fine-tuning on the full training data. Gains in k-NN top-1 accuracy (percentage points) over the corresponding vanilla fine-tune.
| Backbone | DTD | GTSRB | Tiny-ImageNet |
|---|---|---|---|
| DINOv2 ViT-S/14 | +0.88 | +0.32 | +0.81 |
| TIPSv2 ViT-B/14 | +1.83 | +1.22 | +0.67 |
Random walk pairing improves all 14 new configurations, with gains of up to +1.64 points on GTSRB and +1.83 points on DTD for TIPSv2. Together with Table 5, these results provide further evidence that our approach benefits recent pretrained SSL models across different architectures, datasets, and both partial-data and full-data fine-tuning. We have included these new experiments in Appendix B.8 (Tables 18 and 19) of the revised manuscript.
While longer ImageNet-1k pretraining would provide additional validation, the matched ImageNet comparisons and the broader experiments with recent SSL models support the effectiveness and general applicability of our approach.
W2 (Part 2/2)
Additional robustness evaluation — Distribution shift on CIFAR-100-C
To further evaluate robustness under distribution shift, we tested our existing DINOv2 and TIPSv2 checkpoints on CIFAR-100-C, without retraining. The goal is to assess whether random walk pairing produces representations that generalize more reliably when the test distribution differs from the clean training distribution.
CIFAR-100-C introduces this distribution shift by applying 19 types of corruption (noise, blur, weather, and digital distortions) at five severity levels (1 = mild, 5 = severe) to CIFAR-100 test images. The models are trained and fine-tuned exclusively on clean images, while corruptions are introduced only at test time. We evaluate the frozen representations using k-NN classification against the clean CIFAR-100 training set ().
The table reports mean corrupted accuracy (mCA), averaged across corruption types and evaluated severity levels, alongside the gains at the lowest and highest severities (percentage points).
Table R3. Robustness under distribution shift on CIFAR-100-C.
| Backbone (fine-tuning data) | mCA (Vanilla → Ours) | mCA Gain | Gain (Sev. 1 → 5) |
|---|---|---|---|
| DINOv2 ViT-S/14 (10%) | 60.33 → 62.76 | +2.43 | +2.05 → +2.60 |
| DINOv2 ViT-S/14 (20%) | 62.28 → 64.49 | +2.22 | +1.60 → +2.65 |
| DINOv2 ViT-S/14 (100%) | 65.82 → 66.44 | +0.62 | +0.26 → +0.83 |
| TIPSv2 ViT-B/14 (20%) | 64.18 → 65.59 | +1.41 | +1.13 → +1.65 |
| TIPSv2 ViT-L/14 (20%) | 72.29 → 74.15 | +1.85 | +1.23 → +2.45 |
For TIPSv2, mCA is averaged over severities 1, 3, and 5; DINOv2 uses all five severities.
Random walk pairing improves robustness under distribution shift across all evaluated configurations, with gains of up to +2.43 percentage points in mean corrupted accuracy. The improvements generally increase with corruption severity and extend across corruption types: accuracy improves on 18/19 corruptions for DINOv2 at 10% and 20%, and on all 19 for both TIPSv2 models.
These results demonstrate that the benefits of random walk pairing extend beyond clean-data performance to generalization under distribution shift, where models must maintain accurate representations despite changes in the input distribution. We have included this evaluation in Section 5 (Table 6) of the revised manuscript, including the experimental setup, corruption severity analysis, and results across corruption types.
We hope the new NNCLR-style comparison, together with the clarification of I-Con and the additional evaluations of recent SSL methods, addresses the reviewer's concerns. We would be happy to answer any further questions and hope the reviewer will consider updating their assessment in light of these results.
W1
We thank the reviewer for the positive assessment of our experimental coverage and the clarity of the paper. We appreciate the suggestions regarding comparisons with existing pairing methods and longer ImageNet training, and address both concerns below, including new experimental evidence.
W1 — Would SAG-VICReg still hold an advantage if NNCLR / I-Con pairing were applied to VICReg
We thank the reviewer for this suggestion. We conducted a new controlled experiment applying NNCLR-style pairing within VICReg. Both variants use the same architecture, training schedule, warm-up, and base VICReg loss coefficients. The NNCLR-style variant deterministically retrieves the nearest neighbor from a queue of 16,384 past embeddings, whereas SAG-VICReg samples probabilistically from its online mini-batch affinity graph.
Hierarchical k-NN accuracy on CIFAR-100 (%):
SAG-VICReg maintains its advantage at all four hierarchy levels, including a 1.28-point gain at the finest level. This comparison evaluates the two pairing strategies under matched architectures and training conditions. We have included this experiment, together with the corresponding hierarchical linear evaluation, in Appendix B.5 of the revised manuscript.
The nearest-neighbor ablation in Appendix C.2 further supports this finding under a fixed candidate pool. With uniform weighting, random walk pairing outperforms deterministic nearest-neighbor selection (59.21% vs. 57.94%). The advantage persists with affinity weighting (60.01% vs. 59.27%), suggesting that the benefit extends beyond the choice of candidate pool.
Regarding I-Con, Theorem 4 establishes a connection to a simplified VICReg objective that excludes the covariance term, rather than the full VICReg objective used in our experiments. Moreover, its experimental neighbor propagation relies on pretrained DINO features and a precomputed global neighbor graph for clustering, rather than training VICReg from scratch. In contrast, our method constructs the graph online from current representations, uses locally scaled transition probabilities, and retains the full VICReg objective, including covariance regularization. Therefore, while the methods are conceptually related, I-Con's theoretical formulation and experimental results do not directly establish the effectiveness of its neighbor-propagation mechanism within full VICReg.
Overall, the new experiment demonstrates that SAG-VICReg outperforms NNCLR-style retrieval when both are applied within the same VICReg training setup, while requiring no memory queue or externally pretrained features.