- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
- this paper claims that degradation in video llm performance is not only attributed to the availability of few frames in the model context but also attributed to its language prior.
- to address that, this paper introduces Paired Option-Wise Verification, a training free correction approach to reduce the bias that may stem from language prior.
- the proposed method only works in MCQ setup, where it obtains scores for each option with and without the visual evidence and then the final score is an adjusted difference of between score obtained with visual evidence and without visual evidence. this is somewhat simplified description of how their method works.
Critical strengths
- this paper tackles a well known limitation of video llms that they are often biased towards their language prior compared to the visual evidence
- the proposed method is validated on 4 popular video llms and benchmarked on a subset of 4 video understanding benchmarks
Critical weaknesses
-
this method is limited to only MCQ setups and does not work in open-ended QA tasks -- which severely limits the practical utility of this method. how does this method scale if we increase the number of options to 10 or 20, because MCQ setup makes the task quite easy for the model since they models can find the linguistic shortcuts to find the correct answer without truly understanding the question or the video. even if this method is only valid for MCQ setup, can we do a controlled test on a curated subset to see the model behavior when the MCQ options are quite high?
-
we do know that llms in general suffer from yes bias, how can we guarantee that the proposed verification is not impacted by such bias as well? there also exists position bias in option letters? did you consider that error margin as well?
-
the evaluated models are quite older, LLaVA variants and Qwen2-VL are from 2024. It would be interesting to see the impact of this method on newer models such as qwen 3.5/3.8, kimi k3, molmo2, gemma 4.
-
the results in table 2a are not computed on the entire eval set of the corresponding benchmarks, hence they cannot be trusted or compared with the other published results from the literature. the results are presented in a somewhat confusing and cherry picked manner. i also suggest adding reported numbers from newer open-source and closed-source models to provide a point of reference for whether the suggested gain is significant.
-
the evaluation suite is very weak. it would be interesting to see the impact of the proposed method on more diverse benchmarks covering video reasoning and long video understanding tasks such as EgoMemReason, Egoschema, VCGBench-Diverse, Vinoground, EgoTaskQA, LongVideoBench, MMVU, MLVU.
-
re. tab 3 and 4, why do we need to hand pick the number of frames (k, as mentioned in table 1) for individual benchmarks? is this method only effective for a specific k with respect to that benchmark? since, latency is a factor of number of frames in the input and the token length of the response, can we see a more detailed analyses on latency, in comparison to SOTA training free methods?
-
minor comment: l.192: replacing the video with another random video that belongs to the same subtask may not be ideal, imagine the sub-task is action recognition related, the original video is of
liftingand the replaced video isdancing, completely different from the original one, that would not nudge the model confidence much -- i think what we would be beneficial is if the replaced video is a hard negative of the original video. a more controlled replacement could be explored for this experiment.
Questions for authors
please see above under Critical Weaknesses.