acceptodds
Under review as a conference paper at ICLR 2027

When Target Accuracy Hides Semantic Interface Drift in Vision–Language Adaptation

Abstract

A target-adapted vision–language model can improve fine-grained accuracy while moving away from the pretrained image–text interface that supports zero-shot transfer. We define semantic interface drift through changes in direct prompt compatibility, prototype relations, local neighborhoods, and held-out zero-shot behavior. We study same-image frozen-feature anchoring in an identity-initialized visual adapter and test the role of the reference with a controlled intervention. The central experiment closely matches adaptation-validation accuracy while replacing the same-image reference with a within-minibatch shuffled feature. Across five-seed CLIP experiments, the same-image reference preserves the measured geometry and held-out transfer axes at comparable adaptation accuracy. The shuffled-reference control yields a 27.49-point mean transfer drop, compared with 3.78 points for the same-image reference. Three-seed replications recover the retention pattern with SigLIP2 and when EuroSAT replaces CIFAR-100 as the adaptation dataset. These results identify reference correspondence as a design axis that target accuracy alone does not measure.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Vision–language adaptation, representation learning, semantic interface preservation, feature anchoring, zero-shot transfer, representation drift, model selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yBQtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the semantic interface drift in target-adapted VLMs. By defining semantic interface drift as the changes in direct prompt compatibility, prototype relations, local neighborhoods, and held-out zero-shot behavior, the paper shows that fine-tuning a VLM can improve target-task accuracy while degrading zero-shot generalization. The proposed same-image anchoring can address the semantic interface drift while preserving the adaptation accuracy.

Critical strengths

1.The paper studies and proposes semantic interface drift, which is useful and worthing researching.

2.The proposed same-image anchoring is simple yet effectively reduces semantic interface drift.

Critical weaknesses

1.The novelty of same-image anchoring is somewhat limited. As stated in the related works, many previous methods had utilized similar strategies to avoid drifting from the original feature, which weakens the contribution of the proposed method. L_anchor is very similar to the fine-tuning loss in LDIFS.

2.The definition of semantic interface drift is not enough comprehensive. The definition of semantic drift, including changes in prompt compatibility, prototype relations, local neighborhoods, are mostly geometric-wise, not essentially describing semantical capabilities.

Questions for authors

Could you restate the main contribution of the paper and explain each evaluation and how does it support your claims?

Reviewer_ESkLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates how adapting a vision–language model (VLM) can improve target-task accuracy while degrading its compatibility with the pretrained text interface and its zero-shot transfer capability. They propose using a residual visual adapter trained with classification loss and a feature-preservation objective, and the central experiment changes the reference used in this objective, comparing each image’s own frozen feature with shuffled, class-related, and aggregate references at similar adaptation accuracy. Thus, it measures retention through direct prompt accuracy, prototype relations, neighborhood overlap, and held-out transfer, where its main matched-reference experiment reports a 3.78-point mean transfer drop with same-image anchoring, compared with 27.49 points with shuffled references. This suggests that same-image references retain more transfer than other modality's references, and preserving the pretrained image–coordinate assignment therefore helps maintain a reusable interface after adaptation.

Critical strengths

For me, the critical strengths of the paper can be summarized as follows:

  1. The paper addresses a critical problem in adapting vision-language models where the target accuracy hinders the ability to retain zero-shot performance, providing an incomplete assessment of the adapted models. Thus, evaluating prompt compatibility and retained transfer capability alongside adaptation performance is useful, particularly when the adapted model is intended for subsequent reuse.
  2. Some results help distinguish feature displacement from transfer degradation; for example, in the EuroSAT experiment, shuffled-reference anchoring produces smaller feature displacement than unregularized adaptation while suffering greater transfer degradation. This helps show that displacement magnitude alone is insufficient to characterize retained model behavior.
Critical weaknesses

Although the paper has addressed a valid problem in adapting Vision-Language models and proposed some extensive experiments to verify it, I find that this paper has several critical weaknesses, listed below:

  1. The incremental insight over existing feature-preservation work is insufficiently established, and the paper contains limited novelty. The main objective uses standard same-image feature matching, and methods such as LDIFS [1] and Lipsum-FT [2] have already studied the broader observation that adaptation can damage pretrained transfer capability. Since the authors already acknowledge this gap, I assumed the paper needs to explain more clearly what substantial new conclusion follows from studying reference assignment.
  2. Table 4, which compares with established preservation baselines, reports LDIFS and WiSE-FT on six transfer datasets but omits the paper’s own same-image adapter in that evaluation group and the baselines’ adaptation-task accuracies, which makes the table incomplete and hard to interpret. The table cannot establish a matched-setting advantage or decide whether the observed differences arise from the preservation strategy, adaptation scope, or operating-point selection. Thus, a same protocol comparison with relevant feature- and image–text-preservation methods would strengthen the conclusions.
  3. The conclusion is not matched with the experiments' result, as the main experiments train an adapter over cached features while keeping the pretrained encoder frozen and report the correlations between geometry changes and transfer degradation. However, this do not establish their additional explanatory value beyond displacement or reference type, as overclaimed by the conclusuon about semantic interface drift. Follow up experiments involving encoder updates, together with analyses at matched displacement, would better support broader conclusions in this case.

[1] Mukhoti, J., Gal, Y., Torr, P. H., & Dokania, P. K. (2023). Fine-tuning can cripple your foundation model; preserving features may be the solution. arXiv preprint arXiv:2308.13320.

[2] Nam, G., Heo, B., & Lee, J. (2024, May). Lipsum-ft: Robust fine-tuning of zero-shot models using random text guidance. In International Conference on Learning Representations (Vol. 2024, pp. 20654-20687).

Questions for authors

N/A

2 comments

Thank you for the discussion. We have completed an additional three-seed comparison under a common visual LoRA protocol, covering task-only adaptation, three reference assignments, and the LDIFS and Lipsum-FT objectives.

The CIFAR-100 validation accuracies are: Method Validation accuracy (%) Task-only 83.96 ± 0.10 Same-image L2 84.53 ± 0.02 Global-shuffle L2 84.22 ± 0.03 Within-class L2 84.18 ± 0.04 LDIFS objective 84.17 ± 0.02 Lipsum-FT objective 83.57 ± 0.12

Values are means and sample standard deviations across three seeds, using validation-selected checkpoints. These additional results show that same-image anchoring maintains strong target-task utility within the shared adaptation setting. They complement the previously reported held-out comparisons, which examine retained geometry, pretrained prediction agreement, and transfer.

Together, the experiments address two separate questions: whether preservation remains compatible with target adaptation, and what different reference relationships preserve. The latter remains our principal contribution.

We also completed a three-seed Lipsum-FT text-budget sweep with M ∈ {1, 5, 10, 20}. The corresponding best validation accuracies are 83.30 ± 0.02%, 83.41 ± 0.03%, 83.36 ± 0.09%, and 83.45 ± 0.10%. This evaluates the baseline across multiple text budgets rather than relying on a single setting.

Reviewer_TVV5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates semantic interface drift during the adaptation of vision-language models. This phenomenon occurs when a model achieves high accuracy on a target task while the image-text alignment interface learned during pretraining is compromised, resulting in significant degradation of zero-shot transfer capabilities. Such latent degradation cannot be detected by evaluating target-task accuracy alone. Within a visual adapter framework with a frozen text encoder, the paper controls variables and aligns target-task accuracy while changing the reference features used by the anchoring regularizer. This isolates the role of instance-level reference correspondence and uses multiple geometric metrics to quantify the resulting drift.

Critical strengths
  1. The paper identifies the problem of semantic interface drift in foundation models and distinguishes it from conventional forgetting. This problem has practical significance.
  2. The experimental evaluation is relatively comprehensive. The paper conducts experiments across multiple random seeds, adaptation datasets, and model architectures, including CLIP ViT-B/32, ViT-B/16, and SigLIP2. The EuroSAT experiments further support the importance of reference feature correspondence.
Critical weaknesses
  1. The main experiments are conducted in a setting where both the visual encoder and text encoder are frozen, and only an identity-initialized visual adapter is trained. Although this design facilitates isolating and analyzing the effect of reference feature correspondence, the paper does not verify whether its conclusions generalize to other common parameter-efficient fine-tuning paradigms, such as LoRA, partial fine-tuning of the visual encoder, or joint updates of the visual and text encoders.
  2. The original-feature L2 anchoring regularizer used in this work is an existing method. The paper does not propose a new algorithm or an improved strategy, limiting its contribution in terms of algorithmic innovation.
  3. All experimental evaluations focus on zero-shot image classification. However, semantic interface drift is fundamentally a problem at the image-text interface level. The paper lacks evaluations on other vision-language tasks, such as image-text retrieval and visual question answering (VQA).
  4. The baseline comparisons suffer from inconsistencies in experimental settings. The compared baselines directly use publicly available pretrained weights and are not reproduced under a unified training setup, hyperparameter configuration, and validation-based model selection protocol. Consequently, it is difficult to rule out the possibility that performance differences between baselines arise from differences in training settings rather than from the effects of the regularization strategies themselves.
  5. The figures throughout the paper are overly simplistic and have relatively poor readability.
Questions for authors

No

1 comment

We have completed an additional three-seed visual LoRA experiment covering six adaptation objectives. Same-image anchoring achieves 84.53 ± 0.02% CIFAR-100 validation accuracy, compared with 83.96 ± 0.10% for task-only LoRA, 84.22 ± 0.03% for global-shuffle anchoring, and 84.18 ± 0.04% for within-class anchoring.

These validation-selected results provide additional evidence that same-image preservation is compatible with effective adaptation inside the visual encoder. Combined with the held-out geometry and prediction-agreement results reported in our previous response, they strengthen the evaluation beyond the original frozen-feature adapter setting.

Comments on the paper

1 comment
Area_Chair_JhuLOpenReview ↗

Discussion period ending soon

Dear Reviewers,

Thank you for your reviews of this submission. The authors have posted responses addressing the concerns raised.

With limited time remaining in the discussion period, please take a moment to review the authors' responses and post a brief follow-up indicating whether your concerns have been addressed or what remains outstanding. If your assessment changes as a result, please update your score accordingly.

Active engagement at this stage is essential for a fair and well-substantiated final decision. Thank you for your time and effort in reviewing this submission.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.