- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work examines the changes in representational structure that occur when tuning pre-trained SSL models. The authors analyze various SSL models and tuning techniques, with metrics that focus on diverse kind of representational structures such as local, global or layer-wise structure. The results expose that the effects of post-training on the internal representational structure vary significantly across settings.
Critical strengths
In general, the methodology is sound, with diverse pre-trained models, datasets, metrics etc. Analyzing LoRA adaptation in vision models is also interesting, given that this tuning technique is largely absent from the visual SSL literature.
Critical weaknesses
The main limitation of this paper is the absence of key take-away. The manuscript provides a lot of technical analyses, but it remains unclear why they may be important to the SSL community. In line with this limitation, the introduction indeed clarifies that the analysis has never been conducted, but does not explain why that matters. Addressing this point likely requires to relate the present analytical results to downstream out-of-distribution/in-distribution accuracy. This could clarify how relevant a tuning technique is, why, and under what conditions.
A myriad of details make the paper hard to follow. For instance:
- the notion of class purity is not clearly explained in the main paper.
- the authors employ the word "target" to refer to a downstream "dataset".
- "[..] across all families", families of what ?
- The acronym GW is never properly introduced.
- In the result section, the objective of each paragraph is not properly stated before describing the results.
- Sometimes, the relevant Figure or Table is not cited prior to its description.
Questions for authors
- Why do these technical observations matter to the SSL community ?
- How the representational changes observed during post-training tuning relate to downstream accuracy ?
6 comments
Reminder to engage in reviewer-author discussion
Hi M3Yt,
As there are only two days left in the author-reviewer discussion period, I would like to remind you to kindly respond to the authors' rebuttal, posted last week. After November 18, reviewer responses are not be allowed and you will no longer be able to ask any follow-up questions. If you have no follow-ups, a short and polite response acknowledging you have read and understood the rebuttal and have no further questions is sufficient.
Thanks, AC
I thanks the authors for the answer.
Minor thing: why are there two numbers in the first Table.
The authors claim that "Across 114 adapted models, retaining local pretrained neighborhoods outside the adaptation domain consistently predicts retained accuracy, whereas preserving global geometry does not. " However, they write afterwards that "CKA is similarly informative" to Neighborhood retention. It is contradictory, as CKA is a measure of global geometry.
Regarding the correlation: the analysis may also encourage the introduction of methods that specifically promote a specific level of neighborhood retention. For clarification, is the correlation computed with off-target or on-target metrics ? I also wonder if the authors considered LoRA with r=1 and r=4 in the group: their Neighbours kept is high while their accuracy is "low". Could the authors discuss these results ?
Regarding the depth of divergence: there are confounding factors so that it is unclear whether that is the depth of divergence or just LoRA that matter. Doing more experiments with frozen layers (with LoRA and finetuning) could provide more evidence for this claim.
Regarding the preservation-capacity trade-off: that could be particularly interesting if the authors could describe concrete scenarios where off-target evaluations matter, and/or provide evidence showing that post-training can improve performance on the final off-target dataset.
In general, the additional analyses may reveal interesting directions of works. However, the integration of these analyses also requires substantial rewriting of the paper, which we can not review during the rebuttal or iterate on. Therefore, I encourage the authors to complete this analysis and refocus/restructure their study.
Addressing reviewer M3Yt concerns
Key takeaway
We agree that the paper’s central takeaway should have been stated more clearly. Across 114 adapted models, retaining local pretrained neighborhoods outside the adaptation domain consistently predicts retained accuracy, whereas preserving global geometry does not. Our additional analysis supports three conclusions:
Together, these results indicate that effective adaptation depends not only on how much representations change, but also on where and at what structural scale they change. This can guide future choices of adapter placement, updated components, and LoRA rank. We will make this takeaway explicit in the abstract, introduction, and conclusion of the camera-ready version.
How the representational changes observed during post-training tuning relate to downstream accuracy ?
On non-adaptation datasets, representational preservation predicts retained accuracy. Neighbour retention is positively correlated with linear-probe accuracy changes in 42/42 backbone–dataset groups (median ρ=0.90), k-NN accuracy in 40/42 (ρ=0.83), and robustness under corruption in 9/10. CKA is similarly informative, whereas Gromov–Wasserstein distance is not.
For LoRA, preservation comes at no target-accuracy cost up to r=16; larger ranks reduce both preservation and accuracy. This supports selecting the smallest rank within tolerance of the best target accuracy, with label-free neighbour retention serving as a practical selection signal. Full results are reported below.
Clarifications
The notion of class purity is not clearly explained in the main paper. → We acknowledge that the main paper explanation of class purity is quite succinct and, upon acceptance we will use the added space to move the explanation from the appendix in the main paper.
"[..] across all families", families of what? → Here families refer to the different families of vision SSL models analyzed in this paper. We will clarify terminology to improve clarity.
In the result section, the objective of each paragraph is not properly stated before describing the results. → Along the key take-away that will be added to better frame the scope of the full paper, we will add the key intuitions of each paragraph and how they relate overall to each other.