- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper analyzes why fine-tuning improves video question-answering accuracy on UrbanVideo-Bench. Using scoring corrections, text-only training, video removal, and option permutation, the authors find that much of the improvement can be obtained without video. They argue that output formatting and question-and-option biases contribute substantially to the reported gain, and propose a checklist for evaluating fine-tuned video-QA models.
Critical strengths
The paper clearly describes the training details and adapter configurations, making the experiments and comparisons easy to follow.
Critical weaknesses
-
Limited evidence beyond a single benchmark: All experiments are conducted on UrbanVideo-Bench. The results therefore leave open whether the observed dominance of text-only fine-tuning is common across video-QA benchmarks or primarily reflects this dataset’s particular construction and biases.
-
A per-question analysis of overlapping and model-specific gains, broken down by question type, is needed to clarify this interpretation.
-
Limited generalizable insight: The results primarily demonstrate that fine-tuning can exploit question-and-answer priors in UrbanVideo-Bench, a phenomenon already recognized in a lot of prior works. The paper attributes much of the observed behavior to dataset-specific regularities, but does not sufficiently establish the underlying mechanisms or examine whether the findings extend to other benchmarks and domains. Consequently, its contribution remains a benchmark-specific diagnosis, with limited new insight into how VLMs learn and rely on such priors more generally.
-
Novelty limited: The paper combines established diagnostics to examine one benchmark-specific fine-tuning result. The study does not establish a substantial conceptual advance or a generalizable empirical finding. Its contribution is closer to a dataset-specific technical report, and the research novelty appears insufficient for ICLR acceptance.
Questions for authors
How does performance change when the correct video is replaced with an unrelated video, preferably from the same footage source? This control would help distinguish reliance on question-relevant visual evidence from effects associated with the presence of visual input.