acceptodds
Under review as a conference paper at ICLR 2027

WHEN MORE ADAPTATION HURTS: DEGRADATION- SELECTIVE SENSITIVITY IN CLIP FINE-TUNING

Abstract

Fine-tuning on degraded data should improve robustness, but we find the opposite can occur. We uncover degradation-selective sensitivity in CLIP fine-tuning: robustness is not uniform, but depends jointly on adaptation depth and learning rate , an interaction concealed by aggregate accuracy. On a fine-grained bird benchmark with naturally occurring degradation, full-network adaptation at retains % accuracy on sharp images but drops to % on slight blur, while partial adaptation stays stable; across three domains and three seeds, high-dose full adaptation consistently enters a high-risk regime. Mechanistically, failure reflects how adaptation directions interact with decision geometry, not how far representations move: drift magnitude is matched across strata, and norm-matched counterfactuals show learned drift is protective yet less so on degraded inputs. A dose sweep shows that adaptation continues after error recovery saturates, while a paired within-domain control implicates directional motion degradation rather than blur magnitude; a minimal two-dimensional model illustrates how matched-margin samples can persist under late directional updates. We formalize this through an adaptation tolerance , which predicts co-adapted deployment-head errors (AUC vs. for a single-normal baseline). Degradation-stratified validation exposes these high-risk configurations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
CLIP, Vision-Language Models, Adaptation Depth, Learning-Rate Sensitivity, Representation Drift, Degradation-Selective Sensitivity, Adaptation Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FTszOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates how fine-tuning depth (unfrozen blocks) and learning rate impact CLIP ViT-B/32 robustness to image blur. On a 20-class iNaturalist bird subset stratified by blur level, full fine-tuning with lr=10−4\text{lr} = 10^{-4} achieves 98.5% accuracy on sharp images but drops to 78.8% under slight blur, whereas partial fine-tuning remains robust. While full fine-tuning at high learning rates performs worse than partial fine-tuning on the benchmarks with synthetic degradation (Aircraft, Cars, Imagenette, and ImageNet-100), the gap between sharp and blurred images appears mainly on the Bird dataset. Tracking feature drift from the pretrained model via a fixed linear probe yields an "adaptation tolerance" metric that predicts fine-tuned classifier errors (AUC = 0.96). Finally, blur-stratified validation avoids these failure modes.

Critical strengths

Unfortunately, I cannot find a critical strength that supports acceptance. The paper is very hard to read, and several main experiments are incomplete, as described in the weaknesses. Since ICLR allows authors to revise the paper during the discussion period, I will carefully read the revised version.

Critical weaknesses
  1. The paper is very hard to read.

1.1 All citation uses \citet{} instead of \citep{} which makes it very hard to read.

1.2 There is a lot of jargon and rarely used terms. For example, high-dose full adaptation, dose sweep, and stratum are not used in this domain. Also, degraded data can be replaced with corrupted data (e.g., ImageNet-C). Furthermore, the authors conducted fine-tuning but refer to it as adaptation (L160), due to established fields (domain adaptation and test-time adaptation), readers will be confused.

1.3 Figures 1 and 4 are too small.

  1. The paper mainly focuses on blur. It would be better if the authors moved other cases mentioned in L422-429 to the main paper instead of reporting performance with individual random seeds.

  2. The authors incompletely conducted their main experiments (Table 1), and most experimental results are based on a single seed, which raises another concern about whether the paper's claims hold with other random seeds. This makes the entire paper incomplete.

  3. The observation in Section 3.2 that higher learning, freezing fewer layers, makes more catastrophic forgetting (higher GG) is trivial.

Questions for authors
  1. In L321, what are cosine drift and full jump drift?

  2. What is zz in Eq. (2)?

Reviewer_q5eXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how fine-tuning on degraded data can lead to lower robustness or degradation-specific sensitivity, depending on the number of layers fine-tuned and the learning rates. The authors conduct parameter sweeps across multiple datasets and representation-level analyses to understand these phenomena, and further introduce an adaptation tolerance metric to characterize robustness during fine-tuning.

Critical strengths
  • The paper studies an important problem for fine-tuning — to increase robustness during fine-tuning, existing studies have been integrating certain degraded data during the fine-tuning process, which, however, can distort the feature space and is sensitive to distributional shifts.
  • There are many experimental sweeps across studied parameters with detailed results listed in the paper. The authors also provide focused and representation-level analysis for certain cases (figure 4 and 5).
Critical weaknesses
  • Lack of fine-tuning strategies studied. The paper focuses on the phenomena where different fine-tuning recipes can lead to different robustness sensitivity. The authors also claim that fine-tuning the entire model may lead to lower robustness. However, some widely-used fine-tuning strategies (especially parameter-efficient fine-tuning strategies) are not tested. It would also be interesting to see how some adversarial fine-tuning strategies work under the tested scenarios. It is not convincing or grounded enough to only focus on the number of layers fine-tuned and the learning rate, given how fine-tuning strategies have been evolving already and how much robustness-aware methods have been studied and adopted.
  • The writing in the paper is a little hard to follow with many numerical listings in all the sections (it is understandable that this is a heavily empirical paper), but showing the numbers in the tables and figures is enough, the paragraphs should include more explanation, hypothesis, and analysis.
  • The insight of the resulting selection protocol is limited. In section 7, the degradation-aware rules usually select the same recipe, which also achieves the highest clean accuracy. This general result would provide less guidance on how to further improve robustness in different scenarios.
Questions for authors

Thanks for submitting this work to ICLR 2027! The problem studied is interesting and important, however, the depth and contribution of the work can be improved. The following questions are based on the comments in the previous section.

  • How do different categories of parameter-efficient fine-tuning strategies, such as LoRA and Adapter, work in the examined empirical framework? Some of the robustness-aware fine-tuning strategies are mentioned in the paper but not examined, the two studied parameters (depth and learning rate) are important but may be an over-simplification here.
  • Since the paper points out that the aggregated accuracy/robustness can hide degradation-specific drops, can the designed framework identify different recipes under higher-granularity, more different degradation distributions?
Reviewer_n6egOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how fine-tuning depth and learning rate affect CLIP robustness across different levels of image degradation. On the Bird dataset, full-network fine-tuning at a learning rate of 10⁻⁴ retains 98.5% accuracy on clear images but drops to 78.8% on lightly blurred images. The authors call this pattern degradation-selective sensitivity. They examine adaptation configurations across five classification datasets and analyze feature drift, effective rank, representational similarity, image–text alignment, and displacement relative to a frozen decision boundary. The paper also introduces an adaptation-tolerance metric, τ, and evaluates degradation-stratified validation and several mitigation settings.

The experiments establish a notable performance disparity across degradation levels, while the broader mechanism, predictive interpretation of τ, and practical advantage of stratified model selection remain less firmly supported.

Critical strengths
  1. The paper identifies a robustness failure that can be missed by aggregate or clear-image evaluation. On Bird, high-learning-rate full fine-tuning preserves clear-image accuracy while substantially reducing slight-blur accuracy. The result is examined across multiple random seeds, and the comparison with partial fine-tuning makes the practical relevance of adaptation choices clear.

  2. The experimental analysis is relatively extensive. Cross-dataset results help separate the broader penalty of aggressive full fine-tuning from the more specific degradation-selective pattern. Feature-drift, effective-rank, CKA, and frozen-boundary analyses provide complementary evidence suggesting that the direction of feature displacement, rather than its magnitude alone, is relevant to classification outcomes. The paired same-image corruption experiments are particularly useful because they reduce content variation and isolate the sensitivity to directional motion blur.

Critical weaknesses
  1. Although the paper explicitly distinguishes the broadly replicated high-learning-rate full-fine-tuning penalty from the stronger Bird-specific degradation-selective pattern, the explanation of the latter remains difficult to isolate from optimization and dataset factors. At the same nominal learning rate, full and partial fine-tuning differ in trainable parameter count and may have substantially different update magnitudes and training dynamics; per-layer update statistics or matched-optimization comparisons are not provided. In addition, the naturally occurring sharp and slightly degraded groups may differ in image content, category composition, or source. The same-image synthetic controls are valuable and demonstrate operator-specific sensitivity, but do not fully identify the cause of the gap in natural images. For me, this leaves the direction-boundary explanation suggestive rather than fully identified.

  2. The additional value and generality of the proposed adaptation-tolerance metric and validation strategy remain incompletely established. The paper reports an AUC of 0.96 for τ on 94 initially probe-correct slight-blur Bird examples in the key Full configuration, and a pooled AUC of 0.89 across eight configurations (Appendix H). These are meaningful results, but τ requires feature displacement observed after fine-tuning, so they support post-training error assessment rather than advance prediction of risky training configurations. Its stability across independent data, random seeds, and other tasks also remains unclear. For model selection, Appendix L, Table 26 shows that degradation-aware criteria and overall Top-1 choose the same Bird configuration (L8 at 2 × 10⁻⁵); all four tested selection rules also agree on Aircraft and Cars. I therefore see clearer evidence for degradation-aware validation over sharp-only selection than over standard overall-accuracy validation.

Questions for authors
  1. For key full and partial fine-tuning settings, especially Full (10⁻⁴) and L6/L8, can the authors report training/validation loss curves and per-layer relative parameter updates? Does the full-fine-tuning penalty persist when actual update magnitudes or training progress are made comparable?
  2. For the natural iNaturalist images, how comparable are the sharp and slight groups in class distribution, resolution, content, and source?
  3. Appendix H reports an AUC of 0.96 for Full (10⁻⁴) and 0.89 pooled across eight Bird configurations. How stable is this relationship on held-out images, additional random seeds, or other tasks, and what are its confidence intervals? Given that τ requires the trained model’s feature displacement, is its intended role post-training error assessment rather than advance configuration selection?
  4. Table 26 shows that degradation-aware and overall Top-1 selection choose the same Bird configuration, although degradation-aware selection does improve over sharp-only selection. What additional benefit, if any, is expected over overall-accuracy validation? Can the authors identify a case where the choice differs, or clarify that the main demonstrated benefit is protection against sharp-only evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.