acceptodds
Withdrawn from ICLR 2027

Test-Time Paired Option-Wise Verification to Reduce Prior-Following Errors in VideoLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video large language models (VideoLLMs) lose accuracy when only a few frames are available. We found that this degradation stems not only from missing visual evidence but also from prior-following errors: driven by its language prior, the model gives the same wrong answer with or without the video. Replacing the original video with an unrelated one often preserves the preferred option, indicating weak dependence of the decision on the question-specific video. However, reversing the temporal order changes the relative support for the correct and no-video preferred options, showing that sampled frames retain discriminative evidence. This mismatch between retained video evidence and prior-aligned decisions motivates Paired Option-Wise Verification (POV), a training-free correction method that independently verifies each answer option with and without the video. To remove the language prior, POV subtracts a scaled no-video score from each option's video-conditioned score. The scale is estimated from unlabeled target-task data and is reduced only when the video evidence is consistent with the prior, that is, when the verifier supports the prior-preferred option. Across several frozen backbones, POV outperforms competing training-free baselines on temporal multiple-choice and binary QA, and even surpasses dense-frame VideoLLMs while using the sparse-frame budget.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video large language models, Video reasoning, Temporal reasoning, Sparse-frame reasoning, Multimodal reasoning, Test-time correction, Language priors

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vDJwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary
  • this paper claims that degradation in video llm performance is not only attributed to the availability of few frames in the model context but also attributed to its language prior.
  • to address that, this paper introduces Paired Option-Wise Verification, a training free correction approach to reduce the bias that may stem from language prior.
  • the proposed method only works in MCQ setup, where it obtains scores for each option with and without the visual evidence and then the final score is an adjusted difference of between score obtained with visual evidence and without visual evidence. this is somewhat simplified description of how their method works.
Critical strengths
  • this paper tackles a well known limitation of video llms that they are often biased towards their language prior compared to the visual evidence
  • the proposed method is validated on 4 popular video llms and benchmarked on a subset of 4 video understanding benchmarks
Critical weaknesses
  • this method is limited to only MCQ setups and does not work in open-ended QA tasks -- which severely limits the practical utility of this method. how does this method scale if we increase the number of options to 10 or 20, because MCQ setup makes the task quite easy for the model since they models can find the linguistic shortcuts to find the correct answer without truly understanding the question or the video. even if this method is only valid for MCQ setup, can we do a controlled test on a curated subset to see the model behavior when the MCQ options are quite high?

  • we do know that llms in general suffer from yes bias, how can we guarantee that the proposed verification is not impacted by such bias as well? there also exists position bias in option letters? did you consider that error margin as well?

  • the evaluated models are quite older, LLaVA variants and Qwen2-VL are from 2024. It would be interesting to see the impact of this method on newer models such as qwen 3.5/3.8, kimi k3, molmo2, gemma 4.

  • the results in table 2a are not computed on the entire eval set of the corresponding benchmarks, hence they cannot be trusted or compared with the other published results from the literature. the results are presented in a somewhat confusing and cherry picked manner. i also suggest adding reported numbers from newer open-source and closed-source models to provide a point of reference for whether the suggested gain is significant.

  • the evaluation suite is very weak. it would be interesting to see the impact of the proposed method on more diverse benchmarks covering video reasoning and long video understanding tasks such as EgoMemReason, Egoschema, VCGBench-Diverse, Vinoground, EgoTaskQA, LongVideoBench, MMVU, MLVU.

  • re. tab 3 and 4, why do we need to hand pick the number of frames (k, as mentioned in table 1) for individual benchmarks? is this method only effective for a specific k with respect to that benchmark? since, latency is a factor of number of frames in the input and the token length of the response, can we see a more detailed analyses on latency, in comparison to SOTA training free methods?

  • minor comment: l.192: replacing the video with another random video that belongs to the same subtask may not be ideal, imagine the sub-task is action recognition related, the original video is of lifting and the replaced video is dancing, completely different from the original one, that would not nudge the model confidence much -- i think what we would be beneficial is if the replaced video is a hard negative of the original video. a more controlled replacement could be explored for this experiment.

Questions for authors

please see above under Critical Weaknesses.

Reviewer_od2bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes POV, a training-free and label-free test-time correction for language-prior bias in VideoLLMs on multiple-choice and binary QA. The authors identify "prior-following errors", where the wrong video-conditioned answer is the same as the answer given without the video, and show with video replacement, temporal-order reversal, and layer-wise probing that the sampled frames still carry useful evidence in these cases. POV turns each option into a YES/NO claim, scores it with and without the video, and subtracts a scaled no-video score (s = v − λp), where λ is estimated from unlabeled questions in each model–sub-task cell, with a one-sided estimate by default and a signed one when the prior-preferred option is still supported by the video.

Critical strengths
  1. The diagnosis of prior-following errors is interesting and supported by several analyses.
  2. The method is training-free and label-free, simple to apply, and the gains are fairly consistent across backbones.
Critical weaknesses
  1. The framing around sparse frames does not match the evidence. The first sentence of the abstract stresses that VideoLLMs lose accuracy when only a few frames are available, but the paper's own results suggest that both the problem and the method are largely independent of the frame budget.
  2. The paper does not establish a relation between the language prior and the number of frames. What is proposed is essentially a training-free, label-free language-prior correction for VQA, so the more appropriate comparisons would be language-prior correction and multiple-choice calibration methods (e.g., contextual calibration (Zhao et al., 2021), noisy-channel or PMI-style scoring, BOLD), which are cited but not compared.
Questions for authors
  1. Since the gain does not depend on the frame budget, why is the paper framed around sparse frames? Is there evidence that prior-following errors are related to the number of frames?
  2. How does POV compare with contextual calibration, noisy-channel/PMI scoring and BOLD under the same setting?
  3. How much of the gain comes from option-wise verification alone versus the prior correction on each benchmark, and how sensitive are the results to the specific estimator rules?
Reviewer_xX4EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the tendency of video-language models (VLMs) to over-rely on textual priors when answering questions. It proposes a training-free inference method that computes option verification scores under both video-conditioned and video-free settings, then applies a correction based on an estimated coefficient. Experiments show that the method improves performance across multiple video understanding benchmarks.

Critical strengths

The method is easy to understand and follows a simple yet effective design. It does not require modifying model parameters and has potential practical value. The authors evaluate the method across multiple models and datasets, and provide ablation studies and case analyses.

Critical weaknesses

1.The core explanation is insufficiently supported. The output scores obtained without video can reflect a model's textual preferences, but this alone is insufficient to establish that they capture the prior component responsible for errors in video-conditioned reasoning. Similarly, removing the component associated with these scores may also suppress useful video evidence. The observed accuracy improvements alone do not adequately validate the proposed mechanistic explanation.

2.The model coverage is outdated, limiting the scope of the conclusions. The experiments focus on four relatively early 7B models, with no validation on newer architectures or larger-scale models. It remains unclear whether the identified problem is still significant in stronger video-language models and whether the proposed method can continue to provide benefits.

3.There are inconsistencies in the experimental reporting, and some results are difficult to interpret. The coefficient implied by the second example in Table 17 appears inconsistent with the selection rule described in the main text. In addition, some ablation results may have been incorrectly used or reported, as exemplified by Table 11. The authors should verify the consistency between the method description and the experimental results, and clarify the actual contributions of individual design choices.

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.