acceptodds
Withdrawn from ICLR 2027

Most of the Gain Is Not Video: Taking Apart a Fine-Tuned Video Question-Answering Score

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video question-answering benchmarks rank video-language models, and fine-tuning on a benchmark's own training questions can lift a model's score by tens of points, a lift naturally read as better use of the video. A multiple-choice score can also rise for reasons unrelated to the video: a fine-tuned model learns the scorer's answer format, the answer statistics of the question set, and even answer letters the prompt never offers. Each effect is known in isolation, but their shares of a fine-tuned gain are rarely measured, and a leaderboard number cannot separate them. We measure them on UrbanVideo-Bench, where a LoRA adapter on Qwen2.5-VL-7B raises accuracy on a clip-disjoint development split from 29 to 71. With prompt, scorer and split fixed and every comparison paired on the same predictions, a fifth to a quarter of the gain is the benchmark's released scoring rule misreading the zero-shot model's replies, and a seventh comes from questions whose answer letter the prompt never offers. An adapter trained on the question text alone, never seeing a frame, recovers 89 to 91 of the gain and scores within three points of each of three video-trained seeds; permuting the answer options costs the video-trained reference six points. Two further readings of the score shrink under controls: an apparent 20-point simulated-versus-real gap at least halves once both simulators are pooled, and three re-trained seeds place the 71 at the top of its recipe's range. These checks are cheap; we collect them in a reporting checklist for fine-tuned video-QA scores and provide the code and aggregate data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
video question answering, vision-language models, parameter-efficient fine-tuning, evaluation methodology, multiple-choice evaluation, answer extraction, text-only baselines

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_R9g8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper analyzes why fine-tuning improves video question-answering accuracy on UrbanVideo-Bench. Using scoring corrections, text-only training, video removal, and option permutation, the authors find that much of the improvement can be obtained without video. They argue that output formatting and question-and-option biases contribute substantially to the reported gain, and propose a checklist for evaluating fine-tuned video-QA models.

Critical strengths

The paper clearly describes the training details and adapter configurations, making the experiments and comparisons easy to follow.

Critical weaknesses
  1. Limited evidence beyond a single benchmark: All experiments are conducted on UrbanVideo-Bench. The results therefore leave open whether the observed dominance of text-only fine-tuning is common across video-QA benchmarks or primarily reflects this dataset’s particular construction and biases.

  2. A per-question analysis of overlapping and model-specific gains, broken down by question type, is needed to clarify this interpretation.

  3. Limited generalizable insight: The results primarily demonstrate that fine-tuning can exploit question-and-answer priors in UrbanVideo-Bench, a phenomenon already recognized in a lot of prior works. The paper attributes much of the observed behavior to dataset-specific regularities, but does not sufficiently establish the underlying mechanisms or examine whether the findings extend to other benchmarks and domains. Consequently, its contribution remains a benchmark-specific diagnosis, with limited new insight into how VLMs learn and rely on such priors more generally.

  4. Novelty limited: The paper combines established diagnostics to examine one benchmark-specific fine-tuning result. The study does not establish a substantial conceptual advance or a generalizable empirical finding. Its contribution is closer to a dataset-specific technical report, and the research novelty appears insufficient for ICLR acceptance.

Questions for authors

How does performance change when the correct video is replaced with an unrelated video, preferably from the same footage source? This control would help distinguish reliance on question-relevant visual evidence from effects associated with the presence of visual input.

Reviewer_fWAnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A rigorous forensic case study of one fine-tuning gain on UrbanVideo-Bench: a LoRA adapter (80.7M trainable parameters) on Qwen2.5-VL-7B lifts a clip-disjoint dev split from 29.2% (zero-shot, as scored) to 70.9%. With a single fixed evaluation setup and every comparison paired (exact McNemar plus 10,000-sample paired bootstrap), the authors decompose the 41.64-point gain exactly: 9.86 points are the released scorer misreading "Answer: C." as "A" (format compliance), 5.98 points come from 65 questions whose gold letter (F/G) the A–E prompt never offers, and 25.80 points are the genuine A–E remainder. A text-only adapter (never sees a frame) reaches 67.2%, within about 2.4 points of three video-trained seeds; permuting options costs the video-trained reference 6.1 points; an apparent 20-point sim-vs-real gap halves (9.7) once both simulators are pooled; and 70.9% is the top of the recipe's seed range (mean 69.66, s.d. 0.40). A seven-item reporting checklist distills the lessons.

Critical strengths

1.Exemplary rigor and transparency. One evaluation setup for all experiments, all comparisons paired with discordant counts, a re-evaluation to measure runtime noise (2 of 903 flips), explicit disclosure of dev-split selection, and an appendix tracing every number to per-question records with hashes and released prediction files. 2.Exact, additive decomposition of the gain into scorer-artifact, out-of-range-answer, and genuine components (9.86 + 5.98 + 25.80 = 41.64) — clean and unusually interpretable. 3.High-value, actionable output. The reporting checklist (split by flight not clip; lenient re-read of baselines; report option/letter distributions; no-video plus text-only plus permuted-option controls; per-category with n and seed spread; per-source; paired tests with disclosure) generalizes beyond this benchmark. 4.Intellectual honesty: the authors repeatedly avoid over-claiming (e.g., "we do not claim fine-tuning hurts any category," noting small categories swing up to 19 points between near-identical checkpoints).

Critical weaknesses

1.Generality is limited by design. One benchmark, one model family, one split. Several headline findings depend on UrbanVideo-Bench-specific defects (an A–E header with F/G answers, already reported in the benchmark's issue tracker; a released parser that defaults to "A"). The transferable artifact is the checklist; the numeric shares are idiosyncratic. A second benchmark/model would help show the decomposition method — not just this instance — travels. 2.No held-out test set. All development decisions (reference, fusion, router) were made on the dev split, and 70.9% is explicitly the seed-range maximum. This is disclosed but means the headline figure is optimistically selected; it should be restated prominently in the abstract, not only later. 3.Clip-disjoint is not flight-disjoint. 215 of 244 dev clips come from flights that also contribute training clips (149 of 396 flights span both). The authors argue the flight-disjoint subset is easier due to question mix rather than leakage, and the text-only result largely moots visual leakage — but shared flights still permit scene/simulator/template similarity, and the 175-question subset is small with wide intervals. 4.Some findings rest on single runs without per-question predictions (majority vote, 32B, fusion, preference optimization). The related "sampling/voting hurts" and "32B does not move the score" claims therefore lack the paired support the rest of the paper enjoys and should be softened or replicated. 5.The core phenomenon is largely confirmatory. Prior work already showed question-only models reproduce much of a fine-tuned model's accuracy; the novelty here is the precise share attribution on a new benchmark, not the existence of the effect — the framing could acknowledge this more sharply.

Questions for authors

1.Can the decomposition be replicated on a second video-QA benchmark (or a second model family) to show the method — not just this instance — generalizes? 2.For the runs lacking per-question predictions, can those be released so the claims receive the same paired treatment? 3.Given shared flights across the split, can you bound how much of the text-only adapter's 67.2% is attributable to flight-level template memorization?

Reviewer_pAQ2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines how much of videoLLM models finetuned gains from the actual gain of knowledge, rather than the answer parsing, answer-letter distribution. The authors mention how usually VideoLLMs are evaluated or ranked based on the final performance, but that does not consider how much and where the gains actually come from. The authros also propose to report the checklist that helps interpreting the fine-tuned VideoQA scores.

Critical strengths
  1. this paper addresses a meaningful evaluation with controlled experiments: different approaches of re-scoring the same prediction with loosen format, or removing the video, training on text-only do help demonstrate why improved benchmark accuracy should not be understood as the improved understanding.

  2. The analysis is clear, and well defined the scope for the experiments. They were cautious about their claims, and also the authors reported paired comparisons, with different seed configurations, making the experiment itself very clear.

Critical weaknesses
  1. The additional insight beyond this known fine-tuning effects could be clearer?. I find the motivation of this paper interesting, and measuring how much these factors contribute I think is very useful. But, at the same time, effects such as learning the answer format, option-order bias, and textual shortcuts are already well-known, as the authors do acknowledge. I would appreciate a clearer explanation on what we learn or could get from examining these effects together beyond this particular benchmark.

  2. For the no-video experiment, the result may reflect the shortcuts in the benchamrk. I wonder for this experiment whether its strong performance mainly reflects the patterns in the Q,O, A distribution of this particular benchamrk. This raises me the question whether the text-only model still recover most of the gain when the answer-label distribution is balanced in both training and evaluation, beyond like the option-permutation experiment already reported.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.