- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether the improvements obtained by supervised fine-tuning (SFT) of VideoLLMs reflect better video understanding or increased reliance on textual shortcuts. The authors provide evidence that conventional fine-tuning can improve benchmark performance while amplifying text-only signals that do not require the video input. To address this issue, they propose VETO, which identifies a text-only activation subspace and constrains LoRA updates to suppress learning directions associated with these shortcuts.
The method is evaluated across multiple VideoLLM architectThis paper investigates whether the improvements obtained by supervised fine-tuning (SFT) of VideoLLMs reflect better video understanding or increased reliance on textual shortcuts. The authors provide evidence that conventional fine-tuning can improve benchmark performance while amplifying text-only signals that do not require the video input. To address this issue, they propose VETO, which identifies a text-only activation subspace and constrains LoRA updates to suppress learning directions associated with these shortcuts.
The method is evaluated across multiple VideoLLM architectures and out-of-distribution video understanding benchmarks. The results suggest that VETO can improve generalization while reducing gains attributable to text-only shortcuts. Overall, the paper contributes both a diagnostic perspective on VideoLLM fine-tuning and a practical approach to mitigating shortcut learning.ures and out-of-distribution video understanding benchmarks. The results suggest that VETO can improve generalization while reducing gains attributable to text-only shortcuts. Overall, the paper contributes both a diagnostic perspective on VideoLLM fine-tuning and a practical approach to mitigating shortcut learning.
Critical strengths
-
Important problem formulation supported by diagnostic evidence. The paper challenges the assumption that improvements from VideoLLM fine-tuning necessarily indicate better visual-temporal understanding. By explicitly examining text-only behavior, it identifies a potentially misleading source of benchmark gains. This distinction is consequential for both model development and evaluation, particularly when assessing out-of-distribution generalization.
-
Mechanistically motivated intervention with broad empirical validation. VETO goes beyond identifying shortcut behavior by introducing a concrete intervention at the LoRA-update level. The method is relatively lightweight and is evaluated across multiple VideoLLM families and benchmarks. The reported improvements in out-of-distribution performance, together with reduced text-only gains, provide complementary evidence supporting the proposed mechanism.
Critical weaknesses
-
The specificity of the identified shortcut subspace and the causal mechanism behind VETO's improvements could be further validated. Although VETO demonstrates encouraging improvements in out-of-distribution generalization, it remains unclear to what extent the identified text-only activation subspace specifically captures undesirable shortcuts rather than useful linguistic or semantic information. Furthermore, the observed improvements may partially arise from the regularization effect of constraining LoRA updates, rather than from selectively suppressing shortcut-related directions. More targeted control experiments, such as matched-dimensional random orthogonal projections or projections onto alternative non-text-only subspaces, would help distinguish the proposed shortcut-suppression mechanism from generic regularization effects. These comparisons would strengthen the causal interpretation of VETO's effectiveness.
-
Generalization beyond the evaluated task and fine-tuning settings remains uncertain. While the evaluation spans multiple VideoLLM backbones and out-of-distribution benchmarks, it remains uncertain whether the same benefits extend to open-ended video reasoning, free-form generation, or substantially different fine-tuning distributions. Since VETO explicitly restricts the model's adaptation directions, additional evidence would help clarify whether this constraint may suppress useful multimodal learning in certain settings.
Questions for authors
-
Can the authors provide additional evidence distinguishing shortcut-related directions from legitimate semantic directions in the identified text-only subspace? For example, does suppressing this subspace reduce performance on tasks where textual priors are genuinely useful?
-
The shortcut subspace is constructed from the dominant directions of text-only activations in the base model, rather than from an explicit decomposition of shortcut-related and legitimate semantic representations. What evidence supports the assumption that these directions primarily encode undesirable shortcuts? Could the observed improvements instead result from restricting adaptation along high-variance activation directions, regardless of their modality or semantic content?
3 comments
W2. We believe this concern is valid and would like to address it. As for other fine-tuning distributions, Tab. M of Appx. F reports fine-tuning on NExT-QA and STAR, whose questions and options are written freely rather than from templates. On both datasets, the text-only gain of SFT largely disappears under VETO, while the accuracy on the full input stays nearly the same as that of SFT, just as on SSv2 in the main experiments. The effect of VETO thus carries over to other fine-tuning distributions.
Beyond multiple-choice QA, we would like to provide additional experiments conducted with the models trained for the main results of the manuscript, where the model generates its response freely instead of choosing among options. For 300 videos randomly sampled from the validation set of ActivityNet Captions, we prompted the base model, SFT, and VETO with "Describe what happens in the video in detail." Claude Sonnet 5.5 then scored each description from 0 to 5 on its consistency with the human-written captions (Correctness), its coverage of them (Detail), and the order of events (Temporal Order), seeing neither the video nor which model wrote the description. The scores are from a single seed of each model.
| Method | Correctness | Detail | Temporal Order |
|---|---|---|---|
| Base | 2.59 | 2.24 | 2.72 |
| SFT | 2.44 | 2.02 | 2.57 |
| VETO | 2.55 | 2.23 | 2.75 |
VETO stays close to the base model on all three criteria. The abilities of the base model are thus largely preserved under the constraint beyond multiple-choice QA as well.
Dear Reviewers,
Please take a moment to review the authors' rebuttal and provide your feedback on whether your concerns have been adequately addressed. Thank you for your time and effort.
W1 & Q2. We thank the reviewer for these careful questions regarding what the subspace actually captures. The reviewer raises a fair alternative explanation, and the controls suggested in W1 allow us to test it directly. If restricting adaptation along high-variance directions were what matters, regardless of modality or semantic content, the suggested alternatives would bring the same gains under shift. We therefore ran a matched-dimensional random orthogonal projection and the dominant directions of another modality, from the visual tokens of noise videos, and of unrelated text from WikiText, the latter two being orthogonal to the VETO subspace. Each matches the per-module rank of VETO, 250 on average, and is trained with the identical recipe and a single seed. The values for SFT and VETO are those reported in the paper. Each cell reports accuracy (%) / Text Δ (%p).
All the alternative subspaces fall below VETO on every benchmark. None of them reduces the reliance on the text, as their Text Δ stays at the level of SFT or above, while VETO lowers it substantially. Compared with SFT, constraining the LoRA update along them may bring some regularization effect, yet its effect on accuracy under shift is inconsistent and varies by benchmark, whereas VETO improves on every benchmark. The consistent gains thus come with reducing the text-only gain rather than with the regularization effect itself, which supports identifying the shortcut with the text-only subspace. This does not mean that the subspace carries nothing meaningful, as it can also hold linguistic or semantic information. What VETO does is steer the gain from fine-tuning away from the regularities of the text alone, so that the gain comes more from interpreting the video and the text jointly, which is what a VideoLLM is meant to do.
Q1. As noted above, VETO only keeps fine-tuning from adding to the text-only subspace and leaves its legitimate semantic directions in the base model. To verify this, we evaluated SFT and VETO on the items that our blind filter removed from the SSv2 evaluation split, 961 items in total, and report the mean over the three seeds used in our main tables. These are the items that the base model answered correctly from the text alone in all three option orders during filtering, so they are the cases where the textual priors are genuinely useful.
On these items, the text alone leads the base model to the correct answer, so the text-only accuracy shows whether this ability is still in place after fine-tuning. VETO largely retains it at 90.8%, whereas it falls to 33.7% under SFT, and the accuracy of VETO on the full input is slightly higher as well. Suppressing the text-only subspace therefore does not hurt the performance where textual priors are genuinely useful.