- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose task vectors (the cumulative update on parameters during downstream training on a specific task) as a potential mechanism to update vision encoders attached to Large Visual Language Models. The authors not only show that these tensors are effective in improving LVLMs' capabilities in those specific downstream tasks while maintaining performance on general-purpose benchmarks, they also show these tensors can be used effectively for task unlearning as well. The authors include main and control experiments that are easy to follow and do a decent job demonstrating the simple premise of the paper, that extracting update tensors on CLIP can improve capabilities in Llava.
Critical strengths
- The method itself is easy to follow, easy to implement and replicate, making it an approach that is very likely to be of interest to the ICLR community. The simplicity of the method is a major strength.
- The paper is well-written, enabling proper understanding from the first pass.
- The method is likely to allow targeted, modular-like insertions of capabilities in fully trained models without degrading general capabilities, an excellent property to have.
Critical weaknesses
- The paper currently provides evidence for a weaker conclusion than the one the commentary seems to suggest, since the improvement is only evident in the downstream task itself, not in "visual capabilities". Since the authors only provide changes in performance for the specific task only, then it does not provide any evidence that the improvement in MNIST performance leads to better "digit-reading" performance overall. More related points follow.
- Related to the above, why is the VQAv2 intact with unlearning? Shouldn't the model be unable to, for example, perform OCR on numbers after unlearning MNIST? Could the authors offer more intuition or targeted VQAv2 subset evaluations (in the same MNIST example, analyze examples where numbers should be read from the image)? This would increase confidence on the generalizability of the improvements to a general "digit-reading ability" rather than MNIST-specific improvement.
- From my reading of the paper, a straightforward comparison that is missing is finetuning the vision encoder itself directly in the Llava framework (freezing the language model and the adapter). Why is this baseline missing? It is informative of and grounds the improvement that the addition of the CLIP update provides.
- The study has been performed on a single model, decreasing the reader's confidence in the generalizability of the findings, in addition to the limitations that are acknowledged by the authors on the divergence of LVLM backbones and the original encoder.
Questions for authors
- Could the authors clarify which claim the paper is trying to provide evidence for, and have that reflected more clearly in the paper, or provide the additional evidence required for the greater claim that I perceive the paper makes? This is the most important factor for my current and future rating.
- On the qualitative effect of visual editing: if performance slightly degrades, the why does the paper show cherry-picked examples of improvement? A more thorough analysis, as suggested in my "Critical Weakness" section, would be much more informative.