acceptodds
Withdrawn from ICLR 2027

Semantic Capability Acquisition and Specialization During Vision-Language Model Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-tuning vision-language models (VLMs) is typically evaluated at a single downstream checkpoint, obscuring whether a semantic capability was never acquired or emerged earlier and later declined during specialization. We ask how semantic capabilities are acquired, when they peak, how well they transfer, and what remains at deployment. We study these dynamics as a semantic capability trajectory, tracking identity- and attribute-based capabilities over training. We formulate a trajectory-based framework that separates capability acquisition, capability-specific optima, and later specialization, and introduce Structured Semantic Routing (SSR) to study how the representation of supervision shapes what is acquired. Across six pretrained backbones spanning DFN, MetaCLIP, and OpenAI CLIP, we show that fine-tuning can acquire semantic capability beyond the pretrained state, including gains observed on held-out evaluations. Unstructured name-and-attribute supervision produces strong name-and-attribute retrieval with comparatively weak name-free attribute-profile retrieval, whereas SSR yields substantially stronger name-free attribute-profile retrieval and is further strengthened by stochastic name-branch dropout. Different capabilities can peak at different stages, so a checkpoint selected by target class-name retrieval need not coincide with a transferable semantic optimum. Continued optimization can therefore preserve strong target class-name retrieval while reducing previously acquired transferable semantic capability. In a representative diagnostic study, this late specialization is consistent with reduced cross-modal semantic accessibility while substantial image-only class structure remains available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vision-language models, representation learning, fine-tuning, transfer learning, semantic representations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gBuZOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates VLM fine-tuning as a process of capability evolution rather than focusing solely on final model performance. Using four evaluation probes across multiple training checkpoints, the authors examine when different capabilities emerge, reach their peak, and how much of their performance is retained at the deployment checkpoint, which is consistently selected based on a class-name retrieval metric. The paper proposes a structured semantic routing mechanism that prevents the attribute branch from accessing class-name information, along with a stochastic dropping strategy for the class-name branch. The main finding is that class-name-free attribute recognition often peaks before the selected deployment checkpoint, and continued training may subsequently degrade this capability.

Critical strengths
  1. The paper provides extensive experimental evaluations, including comprehensive ablation studies and controlled comparisons, which support its main findings.

  2. The experimental setup and dataset splits are clearly documented, facilitating reproducibility.

Critical weaknesses
  1. The writing and organization could be further improved to present the key contributions and findings more clearly.

  2. The paper lacks comparisons with specialized fine-grained vision-language methods, such as FG-CLIP.

  3. The main capability trajectories are based on only a single run per experimental configuration. The observed differences among the six backbones may reflect both model-specific characteristics and training stochasticity, making it difficult to disentangle these effects.

  4. The training configurations are not standardized across backbones. For example, the warmup duration varies from 1,000 to 10,000 steps. These differences may introduce confounding factors into cross-backbone comparisons.

Questions for authors

Why are different training configurations used across backbones?

Reviewer_192MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies semantic capability trajectories during CLIP-style vision–language model fine-tuning, tracking class-name retrieval (P1), name-and-attribute retrieval (P2), name-free attribute-profile retrieval (P3), and Counterfactual Attribute-Profile Preference (CAPP). It introduces Structured Semantic Routing (SSR), which restricts direct class-name access within attribute routes, and SSR+ND, which adds name-summary dropout. Across six pretrained backbones, supervision structure affects the acquired capabilities, and transferable P3 can peak before the target-P1-selected deployment checkpoint. Additional diagnostics examine whether subsequent degradation is consistent with reduced cross-modal accessibility.

Critical strengths

-Separating pretrained, peak, and deployment performance reveals changes obscured by endpoint-only comparisons. The distinct probes also help avoid equating strong name-and-attribute retrieval with effective attribute use.

-Six heterogeneous backbones, name/attribute replacement tests, a name-access ablation, and CAPP provide complementary evidence. Strict class-disjoint transfer results show that some improvements extend beyond the fine-tuning classes. The authors appropriately acknowledge heterogeneous transfer effects and the limited scope of the geometric interpretation.

Critical weaknesses

-P3 uses fixed class-level profiles that may serve as alternative class identifiers. CAPP provides finer-grained evidence, but may still reward matching a recognized class to its typical attributes rather than examining visual evidence. Although the authors exclude localization and part-binding claims, this ambiguity limits the interpretation of attribute-related capability acquisition.

-Although Figures 2a and 4a show a representative strict-transfer case, the broader results remain dispersed across Tables S14/S24 and Figures S7–S9. In Table S24, SSR+ND improves deployment Pairwise CAPP over pretraining in five of six backbones in each strict transfer direction. The descriptive mean gains across the six heterogeneous backbones are approximately 1.16 percentage points for CUB→NAB-348 and 1.19 for NAB→CUB-59, with a 5.67-point decline for DFN5B in the latter direction. A main-paper summary of both strict dataset-class-disjoint transfer directions, with pretrained/U baselines and separate peak/deployment P3 and Pairwise CAPP scores where available, would clarify the extent and variability of transfer.

-The name-access ablation tests restrictions within SSR, and SSR+ND drops the NAME-summary contribution at fusion; neither tests removing or randomly dropping class names from ordinary U training text. These controls therefore do not establish whether structured routing provides additional benefits over simpler name-removal or name-dropout interventions in the ordinary text-encoder pathway.

Questions for authors

-Does description preference change appropriately across images of the same class with different attributes? Another optional control is attribute-relevant versus equal-area unrelated-region occlusion, measuring changes in preference for the correct attribute description rather than overall classification accuracy. Other controls distinguishing visual evidence from class–attribute associations would also be useful.

-Building on the representative results in Figures 2a and 4a, could the authors consolidate both strict transfer directions across all six backbones in the main paper, showing P3 and Pairwise CAPP gains over pretraining and U at peak and deployment where available? Please distinguish already evaluated results from any additional checkpoint evaluations needed. Do the existing diagnostics explain the 5.67-point Pairwise CAPP decline for DFN5B on NAB→CUB-59, or is this failure case outside their scope?

-Could the authors compare SSR/SSR+ND against U trained with class names either always removed (attribute-only text) or randomly dropped? Keeping the ordinary text encoder pathway and other training settings matched, and evaluating all methods with the same P3/CAPP probes, would help isolate the benefit of structured routing.

Reviewer_V8UKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates semantic capability acquisition and degradation during VLM fine-tuning. It introduces Structured Semantic Routing (SSR) to separate class identity and attribute supervision and evaluates capability trajectories across training checkpoints. Experiments show that structured supervision improves name-free attribute retrieval and that transferable capabilities can decline during continued fine-tuning.

Critical strengths
  1. Extensive experiments across six pretrained backbones
  2. Evaluating intermediate checkpoints provides a more detailed insight into capability evolution than final-checkpoint evaluation alone.
  3. SSR+ND demonstrates improvements in name-free attribute retrieval over conventional fine-tuning
Critical weaknesses
  1. Limited novelty and significance: Several central findings are closely related to prior work. Class-name reliance (Roth et al., 2023; Shahgir et al., 2026), performance degradation during task-specific fine-tuning (Andreassen et al., 2021; Wortsman et al., 2022), and the presence of visual information that VLMs fail to exploit (Fu et al., 2025) have been explored in existing research. The paper does not sufficiently establish what fundamentally new insights these observations provide.

  2. Unclear practical motivation: Although different capabilities peak at different checkpoints, the paper does not explain why this is a significant problem when fine-tuning optimizes a specific target objective, or how practitioners can benefit from this observation.

  3. Insufficient methodological details: The main paper provides only a high-level description of SSR, without sufficiently explaining its attention routing, feature fusion, and training procedure. Moreover, the task itself is not clearly formalized.

  4. Unclear presentation: The paper introduces several non-standard terminologies (e.g., capability acquisition and capability-specific optima), abbreviations (e.g., SSR+ND), and evaluation metrics (e.g., P1, P2, P3, and CAPP, Line 142) without sufficient explanation or prior introduction, making the technical narrative difficult to follow.

Questions for authors
  1. What concrete research problem is solved by identifying capability-specific peak checkpoints?
  2. What distinguishes the observed semantic specialization from established phenomena such as catastrophic forgetting and loss of out-of-distribution generalization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.