WHEN MORE ADAPTATION HURTS: DEGRADATION- SELECTIVE SENSITIVITY IN CLIP FINE-TUNING

Fine-tuning on degraded data should improve robustness, but we find the opposite can occur. We uncover degradation-selective sensitivity in CLIP fine-tuning: robustness is not uniform, but depends jointly on adaptation depth $L$ and learning rate $\eta$, an interaction concealed by aggregate accuracy. On a fine-grained bird benchmark with naturally occurring degradation, full-network adaptation at $\eta =10^{-4}$ retains $98.5$% accuracy on sharp images but drops to $78.8$% on slight blur, while partial adaptation stays stable; across three domains and three seeds, high-dose full adaptation consistently enters a high-risk regime. Mechanistically, failure reflects how adaptation directions interact with decision geometry, not how far representations move: drift magnitude is matched across strata, and norm-matched counterfactuals show learned drift is protective yet less so on degraded inputs. A dose sweep shows that adaptation continues after error recovery saturates, while a paired within-domain control implicates directional motion degradation rather than blur magnitude; a minimal two-dimensional model illustrates how matched-margin samples can persist under late directional updates. We formalize this through an adaptation tolerance $\tau$, which predicts co-adapted deployment-head errors (AUC $0.96$ vs. $0.76$ for a single-normal baseline). Degradation-stratified validation exposes these high-risk configurations.