acceptodds
Under review as a conference paper at ICLR 2027

ConceptTune: Tuning Concept Vectors for Generalizable Video Understanding

Abstract

Despite their impressive capabilities, multimodal large language models (MLLMs) still face challenges in video understanding, which current approaches mainly address through parameter fine-tuning, represented by LoRA and LOFIT, or activation steering, represented by RepE. The former enables accurate task-specific adjustment by updating model weights or parameter offsets, but may alter pretrained capabilities. The latter offers flexible generalization through direct representation-space intervention, but their positive–negative activation contrasts may not accurately represent the target concept. We propose ConceptTune, a hybrid method combining the accuracy of fine-tuning with the generalization ability of activation steering. ConceptTune fine-tunes a learnable concept offset directly in model activations rather than model parameters, treating this offset as the concept. Each concept offset is learned from many examples through concept-targeted generative completion. This training procedure encourages the concept offset to capture semantic information shared across training examples of the same concept type, yielding a reusable concept representation that can be applied as a lightweight, plug-and-play activation intervention at inference time. Experiments show that ConceptTune improves transfer across video-grounding benchmarks and compares favorably with parameter-efficient fine-tuning and contrastive activation steering, supporting concept-targeted learning in activation space.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Video Understanding, Activation Steering, Parameter-Efficient Fine-Tuning, Cross-Task Transfer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YgASOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ConceptTune, a method that bridges gradient-based fine-tuning and activation steering by directly optimizing learnable concept vectors in model activations rather than updating model parameters.

The method introduces concept-targeted generative supervision, where a shared concept vector is optimized across multiple examples by reconstructing expressions of the corresponding concept type.

Experiments demonstrate consistent improvements across three backbone models and favorable performance compared with existing methods, including LoRA and activation steering.

Critical strengths

Novel activation steering approach. The paper proposes a gradient-based approach to learning concept vectors for activation steering, combining the advantages of fine-tuning and activation-level intervention.

Generalizability across backbone models. The proposed method demonstrates consistent performance improvements across three different backbone models, suggesting its applicability to diverse MLLM architectures.

Critical weaknesses

1.Questionable motivation and trade-offs. Although ConceptTune bridges gradient-based fine-tuning and activation steering, it also sacrifices some advantages of both approaches: LoRA introduces no additional inference overhead when merged into the backbone, while conventional activation steering can be training-free. Furthermore, the introduction suggests that LoRA updates are inherently task-specific, overlooking prior work demonstrating the composability of model updates, such as:

Ilharco, Gabriel, et al. “Editing Models with Task Arithmetic.” In The Eleventh International Conference on Learning Representations (ICLR), 2023.
Zhang, Jinghan, et al."Composing parameter-efficient modules with arithmetic operation." Advances in Neural Information Processing Systems 36 (2023): 12589-12610.

The motivation and practical advantages of the proposed method therefore require stronger justification.

2.Questionable LoRA baseline performance. In Table 3, LoRA underperforms the frozen backbone in two of the four settings and provides no improvement in another. These results raise concerns about whether the LoRA baseline is sufficiently optimized, potentially undermining the fairness of the comparison with ConceptTune.

Questions for authors
  1. LoRA baseline performance. Could the authors explain why LoRA underperforms the frozen backbone in Table 3? Does this suggest overfitting to the task-specific masked-completion training data?
  2. Semantic specificity of concept vectors. How do the authors verify that the learned concept vectors capture distinct semantic concepts rather than general task-specific biases? In particular, the action vector also substantially improves object-completion likelihood (Figure 3 and Table 8). Could the authors provide additional experiments to validate the semantic specificity of the learned vectors?
3 comments
AuthorsOpenReview ↗

Revised Manuscript Uploaded

Thank you again for your constructive suggestions. We have revised the manuscript accordingly and uploaded the updated version, incorporating the additional baseline comparisons and analyses, and clarifying the motivation, practical trade-offs, and concept-related behavior discussed in our responses.

Reviewer_YgASOpenReview ↗

Response to Author

Thank the author for the rebuttal. I think the rebuttal addresses my concerns. I raised the score accordingly.

Thank you for your careful consideration and constructive feedback throughout the discussion. We are glad that our rebuttal has addressed your concerns. Your suggestions have helped us improve the manuscript, and we appreciate your time and thoughtful evaluation.

Reviewer_XYJTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ConceptTune, a approach that learns concept offsets directly in model activation space through concept-targeted generative completion. The learned offsets can be applied as lightweight, plug-and-play interventions at inference time. Experiments on video-grounding benchmarks suggest that ConceptTune offers competitive transfer performance compared with parameter-efficient fine-tuning and contrastive activation steering.

Critical strengths
  • ConceptTune learns concept representations directly in activation space, avoiding changes to model parameters while enabling lightweight interventions at inference time.

  • The method is evaluated against parameter-efficient fine-tuning and activation-steering baselines, with results suggesting potential benefits for cross-benchmark and cross-task transfer.

Critical weaknesses
  • The LoRA baseline in Table 3 uses the same concept-completion objective as ConceptTune, which helps isolate the adaptation mechanism but does not establish its advantage over conventional task-specific fine-tuning. A LoRA baseline trained directly on temporal grounding using the same source data would provide a more informative comparison.

  • The video-permutation experiments provide useful evidence, but they evaluate the probability of generating a target description rather than whether the model correctly distinguishes different visual content. The remaining gains under video permutation also suggest that the learned vectors may influence language-generation preferences. Tests on unseen actions and action-object combinations would help establish whether the vectors capture transferable visual concepts.

  • In Table 4, the object concept vector improves the three reported object-related categories on VideoMME, but overall accuracy slightly decreases from 61.52 to 61.48. The paper does not provide a sufficiently complete analysis of whether gains in targeted categories come at the expense of other categories, leaving the overall transfer benefits unclear.

Questions for authors
  1. How does ConceptTune compare with LoRA trained directly on temporal grounding using the same Charades-STA training data, in terms of both source-task performance and transfer to unseen datasets or tasks?

  2. Can the authors evaluate whether concept vectors correctly distinguish visually different actions under otherwise similar conditions, and test generalization to unseen actions or action-object combinations?

  3. Can the authors report the full category-wise VideoMME results to clarify whether improvements in object-related categories are accompanied by regressions in other categories?

4 comments
AuthorsOpenReview ↗

Revised Manuscript Uploaded

Thank you again for your helpful suggestions. We have revised the manuscript accordingly and uploaded the updated version, incorporating the additional grounding baseline, visual-evidence analysis, and complete VideoMME category breakdown discussed in our responses.

Thank you for the detailed response and additional analyses. Overall, these supplementary results strengthen the empirical support for the paper.

However, I still have some concerns about the practical applicability of the proposed method. The category-wise VideoMME results show that, although ConceptTune–Object improves performance in all three object-related categories, it also leads to varying degrees of performance degradation in other categories. This suggests that ConceptTune may primarily redistribute model capabilities across different tasks rather than achieve an overall improvement in visual understanding. Given that real-world applications typically involve diverse types of visual queries, how to enhance targeted concepts while preserving the model's performance on non-target capabilities remains an important open question.

AuthorsOpenReview ↗

Practical Applicability of Plug-and-Play Concept Vectors

Thank you for recognizing the strengthened empirical support and for raising this point about practical applicability.

A key advantage of ConceptTune's design is that concept enhancement is modular, selectable, and reversible. The learned vectors act as plug-and-play interventions on a frozen backbone. Depending on the task requirements or known query type, a user can activate an Action vector for action-related questions, an Object vector for object-related questions, or leave the intervention disabled when it is unnecessary. Switching between learned vectors requires no additional training, and disabling the intervention recovers the original frozen model's behavior.

In our VideoMME evaluation, the same Object vector is applied across all question categories using a fixed intervention configuration, including questions outside the targeted concept. The category-wise results therefore characterize the trade-offs of continuously applying one concept intervention across heterogeneous queries.

For applications with known task requirements or query types, ConceptTune supports selective use of the relevant concept enhancement, while other queries can be handled by the original frozen backbone. This gives the concept-specific gains practical value: applications retain access to the original model while choosing when and how to apply the learned interventions. Such controllable, on-demand adaptation is an intended advantage of our method.

Thank you again for your thoughtful feedback. As the discussion period draws to a close, we would appreciate your feedback on our clarification above regarding ConceptTune's selective, plug-and-play use. Could you let us know whether this addresses your concern about practical applicability, or whether any further clarification would be helpful?

Reviewer_7d5tOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper improves video understanding by learning small vectors added to a frozen model’s internal features. It trains these vectors by hiding action or object phrases in video descriptions and asking the model to predict the missing phrases. The same vector will be trained on various samples on similar topics such as action prediction. The hope is, they can learn general skill / concept on solving specific tasks into these vectos and transfer these capabilities to relevant tasks in inference time. Results show improvements across several video benchmarks and three backbones, with smaller overall gains on video question answering.

Critical strengths
  1. The approach is simple and needs very little storage for the learned vectors. Reusing them across different video tasks without further training is useful.
  2. The paper includes useful controls. Training on full descriptions does not give the same improvement, and replacing the correct video with another video reduces the benefit. These results support the choice of training examples and suggest that the model uses visual information.
Critical weaknesses
  1. The novelty of the method seems limited. Existing methods such as RED and ReFT already learn changes to a frozen model’s internal features. The main difference seems to be training the vectors to recover action or object phrases. Comparing these methods with the same training examples would help show what ConceptTune adds beyond the training setup.
  2. The claimed transfer advantage is not fully tested. LoRA and LoFiT are trained on the same examples, but their comparison is only reported on the source dataset. They are missing from the other dataset and video question answering comparisons. Testing these already trained baselines on the same new tasks would show whether ConceptTune actually transfers better. The overall question answering gains are also small, so uncertainty estimates would help assess them.
Questions for authors

How do the already trained LoRA and LoFiT baselines perform on the target datasets and video question answering tasks, without further training?

3 comments
Reviewer_7d5tOpenReview ↗

Review update

Thank you for the additional experiments. My main concerns have been addressed, and I will raise my score.

Thank you for your careful consideration and constructive feedback throughout the discussion. We are glad that the additional experiments have addressed your main concerns. We will incorporate the new results and clarifications into the revised manuscript.

AuthorsOpenReview ↗

Revised Manuscript Uploaded

Thank you again for your constructive feedback. We are glad that the additional experiments have addressed your main concerns. We have now uploaded a revised manuscript incorporating the new results and clarifications discussed in our responses.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Reviewer Discussion

Dear Area Chair,

As the discussion period concludes, we would like to briefly summarize the main concerns addressed and the subsequent reviewer feedback. The additional results and clarifications have been incorporated into the revised manuscript.

  • Reviewer 7d5t (initial rating: 2; confidence: 1): We clarified the contribution relative to RED/ReFT, added baseline transfer comparisons, and reported uncertainty estimates for VideoQA. The reviewer subsequently stated: “My main concerns have been addressed, and I will raise my score.”

  • Reviewer XYJT (initial rating: 2; confidence: 2): We added the direct-grounding baseline, further analyses of visual evidence and transfer, and the complete VideoMME category breakdown. The reviewer acknowledged that these results strengthen the empirical support, while retaining a concern about preserving non-target capabilities when enhancing targeted concepts. In our follow-up, we explained the method's selectable, reversible, plug-and-play intervention design and its intended use according to task requirements.

  • Reviewer YgAS (initial rating: 2; confidence: 3): We addressed the motivation and practical trade-offs, LoRA baseline performance, and concept-related behavior. The reviewer confirmed: “I think the rebuttal addresses my concerns. I raised the score accordingly.”

We thank all three reviewers for their constructive feedback and engagement throughout the discussion, and the Area Chair for their time and consideration of our revised manuscript and the full discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.