acceptodds
Under review as a conference paper at ICLR 2027

Suppressing the Shortcut Hidden Behind the Gains of VideoLLM Fine-Tuning

Abstract

Supervised fine-tuning (SFT) is the stage in which Video Large Language Models (VideoLLMs) advance their capability to reason over video. However, the VideoLLM also learns to answer from textual regularities of its training data, a shortcut that bypasses the video. The regularities behind the shortcut are hard to define at the data level, and the shortcut forms regardless of training epoch, data size, or backbone. The shortcut is hidden behind a rise in accuracy within the training distribution, yet it is not required for the rise and even costs SFT the gain the video offers under distribution shift. We therefore propose Video-Evidence Training via Orthogonal projection (VETO), which intervenes on the fine-tuning update where the shortcut is written rather than on the data. VETO calibrates the subspace that the text alone occupies in the activations of the base model and projects the update onto its orthogonal complement. The VideoLLM is thus left to ground its answer in the full input, with the shortcut suppressed as it forms and never defined. Across seven VideoLLMs VETO preserves the gain of SFT on the training distribution and suppresses the shortcut. On six benchmarks under shift it gains more than SFT and answers from the full input rather than the shortcut.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video large language models, supervised fine-tuning, shortcut, distribution shift, video question answering

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_kZijOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the improvements obtained by supervised fine-tuning (SFT) of VideoLLMs reflect better video understanding or increased reliance on textual shortcuts. The authors provide evidence that conventional fine-tuning can improve benchmark performance while amplifying text-only signals that do not require the video input. To address this issue, they propose VETO, which identifies a text-only activation subspace and constrains LoRA updates to suppress learning directions associated with these shortcuts.

The method is evaluated across multiple VideoLLM architectThis paper investigates whether the improvements obtained by supervised fine-tuning (SFT) of VideoLLMs reflect better video understanding or increased reliance on textual shortcuts. The authors provide evidence that conventional fine-tuning can improve benchmark performance while amplifying text-only signals that do not require the video input. To address this issue, they propose VETO, which identifies a text-only activation subspace and constrains LoRA updates to suppress learning directions associated with these shortcuts.

The method is evaluated across multiple VideoLLM architectures and out-of-distribution video understanding benchmarks. The results suggest that VETO can improve generalization while reducing gains attributable to text-only shortcuts. Overall, the paper contributes both a diagnostic perspective on VideoLLM fine-tuning and a practical approach to mitigating shortcut learning.ures and out-of-distribution video understanding benchmarks. The results suggest that VETO can improve generalization while reducing gains attributable to text-only shortcuts. Overall, the paper contributes both a diagnostic perspective on VideoLLM fine-tuning and a practical approach to mitigating shortcut learning.

Critical strengths
  1. Important problem formulation supported by diagnostic evidence. The paper challenges the assumption that improvements from VideoLLM fine-tuning necessarily indicate better visual-temporal understanding. By explicitly examining text-only behavior, it identifies a potentially misleading source of benchmark gains. This distinction is consequential for both model development and evaluation, particularly when assessing out-of-distribution generalization.

  2. Mechanistically motivated intervention with broad empirical validation. VETO goes beyond identifying shortcut behavior by introducing a concrete intervention at the LoRA-update level. The method is relatively lightweight and is evaluated across multiple VideoLLM families and benchmarks. The reported improvements in out-of-distribution performance, together with reduced text-only gains, provide complementary evidence supporting the proposed mechanism.

Critical weaknesses
  1. The specificity of the identified shortcut subspace and the causal mechanism behind VETO's improvements could be further validated. Although VETO demonstrates encouraging improvements in out-of-distribution generalization, it remains unclear to what extent the identified text-only activation subspace specifically captures undesirable shortcuts rather than useful linguistic or semantic information. Furthermore, the observed improvements may partially arise from the regularization effect of constraining LoRA updates, rather than from selectively suppressing shortcut-related directions. More targeted control experiments, such as matched-dimensional random orthogonal projections or projections onto alternative non-text-only subspaces, would help distinguish the proposed shortcut-suppression mechanism from generic regularization effects. These comparisons would strengthen the causal interpretation of VETO's effectiveness.

  2. Generalization beyond the evaluated task and fine-tuning settings remains uncertain. While the evaluation spans multiple VideoLLM backbones and out-of-distribution benchmarks, it remains uncertain whether the same benefits extend to open-ended video reasoning, free-form generation, or substantially different fine-tuning distributions. Since VETO explicitly restricts the model's adaptation directions, additional evidence would help clarify whether this constraint may suppress useful multimodal learning in certain settings.

Questions for authors
  1. Can the authors provide additional evidence distinguishing shortcut-related directions from legitimate semantic directions in the identified text-only subspace? For example, does suppressing this subspace reduce performance on tasks where textual priors are genuinely useful?

  2. The shortcut subspace is constructed from the dominant directions of text-only activations in the base model, rather than from an explicit decomposition of shortcut-related and legitimate semantic representations. What evidence supports the assumption that these directions primarily encode undesirable shortcuts? Could the observed improvements instead result from restricting adaptation along high-variance activation directions, regardless of their modality or semantic content?

3 comments

W1 & Q2. We thank the reviewer for these careful questions regarding what the subspace actually captures. The reviewer raises a fair alternative explanation, and the controls suggested in W1 allow us to test it directly. If restricting adaptation along high-variance directions were what matters, regardless of modality or semantic content, the suggested alternatives would bring the same gains under shift. We therefore ran a matched-dimensional random orthogonal projection and the dominant directions of another modality, from the visual tokens of noise videos, and of unrelated text from WikiText, the latter two being orthogonal to the VETO subspace. Each matches the per-module rank of VETO, 250 on average, and is trained with the identical recipe and a single seed. The values for SFT and VETO are those reported in the paper. Each cell reports accuracy (%) / Text Δ (%p).

Projected subspace (rank matched to VETO)Perception TestVinogroundAoTBenchMVBenchTVBenchTempCompass
None (SFT)61.1 / +17.063.5 / +13.553.1 / +10.956.5 / +7.844.3 / +7.665.8 / +4.0
Random62.4 / +14.165.3 / +12.852.3 / +9.356.4 / +8.046.0 / +7.265.4 / +4.6
Other modality (noise video)60.2 / +18.363.3 / +17.754.4 / +10.856.5 / +9.143.1 / +7.267.5 / +5.2
Other semantics (WikiText)63.3 / +18.965.6 / +19.750.2 / +12.757.1 / +9.645.3 / +9.665.5 / +5.9
Text-only (VETO)64.3 / +4.966.9 / +7.755.2 / +5.658.0 / +3.546.4 / +3.067.6 / +2.4

All the alternative subspaces fall below VETO on every benchmark. None of them reduces the reliance on the text, as their Text Δ stays at the level of SFT or above, while VETO lowers it substantially. Compared with SFT, constraining the LoRA update along them may bring some regularization effect, yet its effect on accuracy under shift is inconsistent and varies by benchmark, whereas VETO improves on every benchmark. The consistent gains thus come with reducing the text-only gain rather than with the regularization effect itself, which supports identifying the shortcut with the text-only subspace. This does not mean that the subspace carries nothing meaningful, as it can also hold linguistic or semantic information. What VETO does is steer the gain from fine-tuning away from the regularities of the text alone, so that the gain comes more from interpreting the video and the text jointly, which is what a VideoLLM is meant to do.

Q1. As noted above, VETO only keeps fine-tuning from adding to the text-only subspace and leaves its legitimate semantic directions in the base model. To verify this, we evaluated SFT and VETO on the items that our blind filter removed from the SSv2 evaluation split, 961 items in total, and report the mean over the three seeds used in our main tables. These are the items that the base model answered correctly from the text alone in all three option orders during filtering, so they are the cases where the textual priors are genuinely useful.

MethodFull input (%)Text-only (%)
SFT84.2 ± 1.733.7 ± 2.4
VETO85.5 ± 0.590.8 ± 2.2

On these items, the text alone leads the base model to the correct answer, so the text-only accuracy shows whether this ability is still in place after fine-tuning. VETO largely retains it at 90.8%, whereas it falls to 33.7% under SFT, and the accuracy of VETO on the full input is slightly higher as well. Suppressing the text-only subspace therefore does not hurt the performance where textual priors are genuinely useful.

W2. We believe this concern is valid and would like to address it. As for other fine-tuning distributions, Tab. M of Appx. F reports fine-tuning on NExT-QA and STAR, whose questions and options are written freely rather than from templates. On both datasets, the text-only gain of SFT largely disappears under VETO, while the accuracy on the full input stays nearly the same as that of SFT, just as on SSv2 in the main experiments. The effect of VETO thus carries over to other fine-tuning distributions.

Beyond multiple-choice QA, we would like to provide additional experiments conducted with the models trained for the main results of the manuscript, where the model generates its response freely instead of choosing among options. For 300 videos randomly sampled from the validation set of ActivityNet Captions, we prompted the base model, SFT, and VETO with "Describe what happens in the video in detail." Claude Sonnet 5.5 then scored each description from 0 to 5 on its consistency with the human-written captions (Correctness), its coverage of them (Detail), and the order of events (Temporal Order), seeing neither the video nor which model wrote the description. The scores are from a single seed of each model.

MethodCorrectnessDetailTemporal Order
Base2.592.242.72
SFT2.442.022.57
VETO2.552.232.75

VETO stays close to the base model on all three criteria. The abilities of the base model are thus largely preserved under the constraint beyond multiple-choice QA as well.

Dear Reviewers,

Please take a moment to review the authors' rebuttal and provide your feedback on whether your concerns have been adequately addressed. Thank you for your time and effort.

Reviewer_bLTnOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the issue of 'shortcut learning' during the supervised fine-tuning (SFT) of Video Large Language Models (VideoLLMs), which is formally defined and supported by exploratory experiments. Based on the exploration, the authors propose Video-Evidence Training via Orthogonal projection (VETO), a method for suppressing the text-only shortcut while preserving the SFT gain for VideoLLMs. The proposed VETO is extensively evaluated across seven VideoLLMs and six benchmarks, with its original motivation for mitigating the text-only shortcut being supported by additional experiments.

Critical strengths
  • The work is based on a good motivation, well-defines the problem, and addresses it empirically and theoretically.
  • Extensive evaluation of the proposed method VETO is performed across seven VideoLLMs and six benchmarks, with promising experimental results.
  • Provides new insight on the shortcut learning problem of VideoLLM SFT, and the authors experimentally support the claim.
Critical weaknesses
  • The practical cost (additional compute and time consumption) of the proposed method is not evaluated or discussed.
Questions for authors
  • Please provide the details on additional compute and time consumption of VETO for fine-tuning SFT over vanilla SFT.
2 comments

W1 & Q1. We thank the reviewer for raising the practical cost of VETO, as it gives us a chance to elaborate on its practicality. VETO adds significantly less training time than the other baselines and nothing to memory or inference. In the paper, the Training cost paragraph and Tab. G of Appx. D report the time against SFT and the other interventions of Tab. 5, and we extend them with peak memory to exhaustively evaluate any cost caused by the VETO implementation compared to that of vanilla SFT.

MethodOne-off (GPU-h)Per update (s)One epoch (GPU-h)×SFTPeak memory (GiB)
SFT–2.24 ± 0.014.40 ± 0.021.0028.3 ± 0.0
VETO0.03 ± 0.002.36 ± 0.024.67 ± 0.051.0628.3 ± 0.0
LMH17.77 ± 0.332.26 ± 0.0322.21 ± 0.345.0530.5 ± 0.0
mDPO5.28 ± 0.384.44 ± 0.0113.98 ± 0.393.1829.9 ± 0.0

All costs are measured on one B200 under the main recipe presented in the paper. The one-off cost is what a method computes once before training: the calibration for VETO, the bias model and its cached logits for LMH, and the cached reference log-probabilities for mDPO. The one-epoch cost adds it to one pass over the 57k training items. VETO thus adds an overhead to SFT per epoch at the same peak memory, but it is largely insignificant compared to that of the other baselines. Since the projection runs only during training and leaves a standard LoRA, inference is unchanged.

I appreciate the authors' response. I have no further questions and will keep my original positive rating.

Reviewer_Z3FFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies suppressing shortcut learning in Video LLM finetuning. The authors propose VETO, which identifies a subspace associated with text-only activations and projects LoRA updates onto its orthogonal complement. Experiments across seven VideoLLMs and multiple video understanding benchmarks show the effectiveness of VETO.

Critical strengths
  1. The main idea of the paper is clear and well-motivated. And the proposed method is simple yet effective.
  2. Experiments across seven VideoLLMs and multiple video understanding benchmarks show that VETO substantially reduces text-only accuracy gains while largely preserving in-distribution performance and improving results under distribution shift.
Critical weaknesses
  1. Insufficient comparisons with existing methods. The experimental comparisons are insufficient to establish the superiority of VETO over existing shortcut mitigation and subspace-constrained fine-tuning approaches. In particular, Table 5 only considers relatively limited alternatives, including option reshuffling, LMH, mDPO, and random subspace projection. However, several closely related methods for constraining LoRA updates in orthogonal or null spaces are missing, such as LoRA-Null, OPLoRA, and Null-LoRA. These methods provide much more relevant comparisons for evaluating the effectiveness of the proposed projection mechanism. The out-of-distribution experiments in Tables 2 and 3 only compare VETO with standard SFT and the pretrained model. Since improved generalization is one of the paper's central claims, the absence of competitive baselines on these benchmarks weakens the experimental conclusions.
  2. Insufficient Justification for Identifying Shortcut Directions Using PCA. VETO assumes that the leading principal components of text-only activations correspond to shortcut-related directions, but this assumption lacks sufficient justification. These high-energy directions may also encode general linguistic or task-relevant information rather than shortcuts. Although Appendix E (Table I) demonstrates that the selected subspace captures substantial text-only activation energy, it does not establish that these directions specifically encode shortcuts. The random-subspace baseline is also insufficient to validate this assumption. Additional experiments using unrelated text, video-conditioned activations, or alternative spectral components would help clarify whether the identified subspace is truly shortcut-specific.
Questions for authors

Please see weakness part.

3 comments

W1. We thank the reviewer for pointing to these constrained LoRA variants, which are indeed the closest relatives of VETO as they intervene on the LoRA update just like VETO. As the reviewer already mentioned, improved generalization under shift is one of our central claims. We therefore trained LoRA-Null, OPLoRA, and Null-LoRA under the same recipe as SFT and VETO with a single seed, and evaluated them not only on SSv2 but also on the six shifted benchmarks of Tab. 2 and 3. The values for SFT and VETO are those reported in the paper. Each cell reports accuracy (%) / Text Δ (%p).

MethodSSv2PerceptionVinogroundAoTBenchMVBenchTVBenchTempCompass
SFT89.6 / +30.561.1 / +17.063.5 / +13.553.1 / +10.956.5 / +7.844.3 / +7.665.8 / +4.0
LoRA-Null89.2 / +30.559.9 / +17.265.0 / +16.252.0 / +11.355.7 / +8.544.1 / +6.866.0 / +4.9
OPLoRA89.2 / +29.661.0 / +16.163.9 / +14.453.2 / +11.055.1 / +6.942.7 / +5.766.5 / +5.1
Null-LoRA87.0 / +25.762.8 / +12.263.9 / +14.752.7 / +11.257.0 / +5.143.9 / +3.766.5 / +4.3
VETO89.2 / +6.064.3 / +4.966.9 / +7.755.2 / +5.658.0 / +3.546.4 / +3.067.6 / +2.4

None of the variants suppresses the shortcut as much as VETO on any of these benchmarks, and VETO also achieves the best accuracy compared to the three alternatives under shift while largely preserving the in-distribution accuracy. These variants protect what the pretrained model already knows: OPLoRA keeps the update orthogonal to the top singular directions of the weights, Null-LoRA restricts it to their null space, and LoRA-Null initializes it in the null space of activations that represent the pretrained knowledge. VETO instead constrains what fine-tuning adds, keeping the update away from the directions the model uses when it reads the question without the video. The point of our paper is that the gain from standard SFT partly comes from leveraging only the text part of the input rather than the video, which is precisely why protecting what the model already knows does not prevent the shortcut, while VETO does. We hope this view helps future fine-tuning of VideoLLMs, and of MLLMs more broadly, keep the answers grounded in the input modality, and we intend to pursue this direction ourselves.

We would also like to point out some Implementation Details, as some parts of these experiments depend on our own implementations. LoRA-Null provides official code, but we reimplemented its initialization within our pipeline and trained it under our recipe. OPLoRA and Null-LoRA have no official code to utilize, so we implemented them from the papers, and details may differ from the authors' implementations. For example, where a weight of Qwen2.5-VL has no exact null space, Null-LoRA uses its eight smallest singular directions.

W2. The leading principal components of text-only activations carry linguistic ability, and this is exactly why they matter. When fine-tuning adds ability along the pathway that is active without the video, the model learns to answer from the question alone. This part of the gain rests on the textual regularities behind the shortcut and leaves the model less room to improve by actually using the video, as the empirical results under shift illustrate. VETO does not suppress the linguistic ability of the base model but only keeps what fine-tuning adds away from the text-only pathway, so that it is learned from the video and the text together. To validate that the effect is specific to these components, we projected subspaces from unrelated text, video-conditioned activations, and alternative spectral components, as the reviewer suggested. Each has the same per-module rank r as VETO under the identical recipe, where r averages 250. These subspaces are each trained with a single seed, and the values for SFT and VETO are those reported in the paper.

Projected subspaceSSv2 Acc. (%)SSv2 Text Δ (%p)
SFT89.6+30.5
Unrelated text (WikiText), top directions89.5+30.4
Video-conditioned activations at visual tokens, top directions87.4+30.8
Text-only activations, next spectral band (r+1 to 2r)89.4+29.4
Text-only activations, far spectral band (1001 to 1000+r)89.3+30.4
Text-only activations, top directions (VETO)89.2+6.0

Only the top text-only directions of VETO suppress the shortcut, while every other subspace leaves it at the level of SFT. The effect thus does not come from high-energy directions in general, whether of visual tokens or of unrelated text, which supports identifying the shortcut with the leading principal components of text-only activations.

Dear Reviewers,

Please take a moment to review the authors' rebuttal and provide your feedback on whether your concerns have been adequately addressed. Thank you for your time and effort.

Reviewer_Z3FFOpenReview ↗

Keep my original positive rating

I have read the authors' response. I have no further questions and will keep my original positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.