acceptodds
Withdrawn from ICLR 2027

Editing Visual Capabilities of Large Vision-Language Models through Task Arithmetic

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models (LVLMs) are typically adapted through end-to-end multimodal training, even when the desired modification primarily concerns visual capabilities. In this work, we study whether such capabilities can be upgraded independently from the rest of the model, as is common in human-engineered modular systems. We show that LVLMs support localized interventions on the vision tower: task updates can be constructed from independently specialized CLIP vision encoders and applied directly to the LVLM vision tower, while keeping the multimodal adapter and language model fixed. This provides an efficient means of transferring visual expertise into LVLMs without additional multimodal training and, when available, enables the reuse of fine-tuned or domain-specialized CLIP checkpoints. We demonstrate that this intervention supports multiple editing goals, including improving a single downstream task, composing improvements across multiple tasks, and removing targeted visual capabilities while preserving model behavior. Our results show that vision-encoder editing improves target-task performance while maintaining the original multimodal capabilities of the LVLM, at a fraction of the cost of end-to-end training.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
task arithmetic, model merging, model editing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pWpSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose task vectors (the cumulative update on parameters during downstream training on a specific task) as a potential mechanism to update vision encoders attached to Large Visual Language Models. The authors not only show that these tensors are effective in improving LVLMs' capabilities in those specific downstream tasks while maintaining performance on general-purpose benchmarks, they also show these tensors can be used effectively for task unlearning as well. The authors include main and control experiments that are easy to follow and do a decent job demonstrating the simple premise of the paper, that extracting update tensors on CLIP can improve capabilities in Llava.

Critical strengths
  • The method itself is easy to follow, easy to implement and replicate, making it an approach that is very likely to be of interest to the ICLR community. The simplicity of the method is a major strength.
  • The paper is well-written, enabling proper understanding from the first pass.
  • The method is likely to allow targeted, modular-like insertions of capabilities in fully trained models without degrading general capabilities, an excellent property to have.
Critical weaknesses
  • The paper currently provides evidence for a weaker conclusion than the one the commentary seems to suggest, since the improvement is only evident in the downstream task itself, not in "visual capabilities". Since the authors only provide changes in performance for the specific task only, then it does not provide any evidence that the improvement in MNIST performance leads to better "digit-reading" performance overall. More related points follow.
  • Related to the above, why is the VQAv2 intact with unlearning? Shouldn't the model be unable to, for example, perform OCR on numbers after unlearning MNIST? Could the authors offer more intuition or targeted VQAv2 subset evaluations (in the same MNIST example, analyze examples where numbers should be read from the image)? This would increase confidence on the generalizability of the improvements to a general "digit-reading ability" rather than MNIST-specific improvement.
  • From my reading of the paper, a straightforward comparison that is missing is finetuning the vision encoder itself directly in the Llava framework (freezing the language model and the adapter). Why is this baseline missing? It is informative of and grounds the improvement that the addition of the CLIP update provides.
  • The study has been performed on a single model, decreasing the reader's confidence in the generalizability of the findings, in addition to the limitations that are acknowledged by the authors on the divergence of LVLM backbones and the original encoder.
Questions for authors
  • Could the authors clarify which claim the paper is trying to provide evidence for, and have that reflected more clearly in the paper, or provide the additional evidence required for the greater claim that I perceive the paper makes? This is the most important factor for my current and future rating.
  • On the qualitative effect of visual editing: if performance slightly degrades, the why does the paper show cherry-picked examples of improvement? A more thorough analysis, as suggested in my "Critical Weakness" section, would be much more informative.
Reviewer_iasWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper essentially asks whether an LVLM's visual capabilities can be edited solely by modifying its vision encoder (via model-merging style methods). The authors fine-tune CLIP models separately across various datasets/tasks, while keeping the text encoder frozen. The resulting task vectors from each fine-tune of CLIP are used to modify the vision encoder of LLaVA1.5, with the goal of tuning LLaVA's performance cheaply on those tasks (while keeping the language backbone and adapter frozen). The authors claim that the resulting improvements in single-task and multi-task performance show that vision-encoder editing improves target-task performance while preserving multimodal capabilities, at a fraction of the cost of full fine-tuning. Authors also provide some supporting analyses showing correlation between CLIP fine-tuning gains and LVLM model-editing gains, to support their claims.

Critical strengths
  • Findings useful as an existence proof:

    • Efficient adaptation of the LVLM using model editing appears well motivated as a training-free alternative for task-specific performance gains in LVLMs. Although this makes the strong assumption that the task-specific training costs of each CLIP fine-tune are disregarded here. Authors argue that the use of third-party fine-tuned CLIP weights qualifies in such scenarios. While I dispute the validity of this claim in the weaknesses below, it is nevertheless an arguable benefit of the proposed work.

    • The experimental findings themselves may be considered positive given that it is a training-free intervention. Though, in my view, this may also be seen at best as an existence proof that model-merging methods can successfully edit visual encoders in LVLMs.

Critical weaknesses

I have a number of concerns around the novelty in this work and the reliability of the analysis. The primary load-bearing concerns are detailed below, which I believe significantly call into question the usefulness and validity of this work's contributions.

  • Equivalence to simply substituting the visual encoder: Since LLaVA typically keeps its CLIP visual encoder frozen, the paper's editing of the visual encoder is effectively no different from a fine-tuned CLIP plugged into LLaVA, using model editing/merging methods. Thus, the resulting gains of LLaVA on the fine-tuned task don't appear noteworthy at all. In fact, it is also entirely unclear what the argument behind using model merging is, and why this is better than simply substituting the fine-tuned CLIP model into LLaVA instead of doing model merging? See baselines point below.

Effectively, the paper's contributions here appear to sum to showing that visual encoder substitution in LLaVA using fine-tuned CLIP models provides gains towards the fine-tuned task. The model merging methods leveraged are also the vanilla methods applied to CLIP models. It is therefore unclear what the significance or novelty of this contribution is.

  • Unclear Benefit of Proposed Visual Encoder Editing:

    • Baseline Comparisons are entirely missing: Given the above, the paper's primary argument appears to require a comparison between: A) a fine-tuned CLIP model directly copied into LLaVA; B) a fine-tuned CLIP model edited into LLaVA using model merging; and C) CLIP fine-tuned with LLaVA. However, neither comparisons of A or C are provided in the paper. The central motivation the authors have argued is reusing third-party Hugging Face CLIP checkpoints, such as for safety filtering or character recognition. But not only is this is never evaluated, but the authors also do not provide evidence that the proposed use of model merging is preferable to simply copying the third-party checkpoints into LLaVA (baseline A). Finally, without the upper-bound of CLIP fine-tuning with LLaVA (baseline C), it is unclear to what degree fine-tuning the CLIP visual encoder separately is useful to fine-tuning it with LLaVA directly. Especially since the cost of fine-tuning has to be paid in fine-tuning the CLIP models either way. Overall, the most fundamental baselines are missing here, and without them, it is difficult to evaluate the benefit of the proposed visual encoder editing.

    • Gains in LLaVA from edited CLIP model are weak: While fine-tuning dramatically improves every CLIP model, the transfer of these gains to LLaVA by the proposed application of model editing appears quite weak as well. For instance, on the Cars dataset (Table 1), fine-tuning CLIP significantly improves CLIP’s accuracy on Cars, yet the edited LLaVA does not improve at all. I believe this indicates visual encoder fine-tuning in isolation, may be entirely insufficient for LLaVA. This is precisely also why the missing baseline concern above is glaring. Overall, it is therefore unclear if the proposed use of visual encoder editing is even useful, compared to a joint fine-tuning of the visual encoder with LLaVA.

Overall, given the above concerns around the novelty of the contributions and the reliability of the proposed use of model editing, it is difficult to recommend acceptance. I encourage the authors to kindly address the above load-bearing concerns on novelty and primary baselines for the paper's core argument.

Questions for authors
  1. Can the authors please clarify the core novelty and motivation for applying model-merging methods to edit LLaVA's visual encoder, rather than simply substituting it with the fine-tuned CLIP encoder directly?

  2. Why should the proposed approach be preferred over fine-tuning the visual encoder jointly with LLaVA? Particularly given the limited transfer of CLIP improvements to LLaVA on certain tasks (see the point on the Cars task in the Weaknesses). A comparison against joint fine-tuning (LoRA adaptation of LLaVA with full CLIP fine-tuning should be quite cheap), including performance and computational costs is critical.

I am happy to engage in any discussion or clarifications of my concerns, if the authors would benefit from it.

Reviewer_1cwTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors propose that only editing the vision encoder can meaningfully improve the visual performance of the LVLM while maintaining multimodal performance across datasets whilst avoiding additional, costly training. Authors show this framework as their key contribution across multiple tasks: single, multi-task model merging and targeted visual learning and ablate experiments on fine-tuning and LORA (e.g. task arithmetic, tsp-merge and tsp-core space approaches).

Authors validate their approach on very classic vision datasets (Cars, DTD, MNIST showing large improvements over zero-shot CLIP, Llava across tasks.

Critical strengths

Idea is simple and straightforward on showcasing the improvements by improving the encoder without having to do expensive multimodal fine-tuning. Results are notable over zero-shot on classic datasets.

Critical weaknesses

The novelty of the work seems limited and incremental compared to the TA (task arithmetic), TSV-Merge, TSV-Core, which all have been previously applied to the entire model. In this work, the reader’s takeaway are mostly empirical findings and key observations, that the same work can be applied to a subset of the architecture - namely the encoder/vision tower, with the rest of the architecture unchanged w/out performance loss. Furthermore, the gains of the work are more or less on older, more saturated datasets, and more difficult datasets could be evaluated to strengthen the claims/findings (e.g. textVQA, Imagenet, CheXpert).

Questions for authors

Any additional examples or explanation on why/where the approach fails or regresses? It is unclear why the Cars dataset for example drops in accuracy on Table 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.