acceptodds
Under review as a conference paper at ICLR 2027

WHEN VIDEO VERIFIER SCORES MISLEAD: CAPABILITY ASSESSMENT AND VERIFIER SELECTION

Abstract

We audit video verifiers, models that judge whether a candidate answer or trajectory agrees with a video, and show that their scores can mislead both conclusions about model ability and verifier selection. We vary two evaluation choices—negative type and judgment interface—while holding the tested models fixed. On controlled tasks, direct judgment yields win rates of 0.78–0.94 against visible-start negatives but only 0.46–0.58 against endpoint-matched negatives that require checking the internal trajectory. Yet on paired videos where the correct candidate reverses with the internal evidence, CoT judgment raises paired success from 0–6% to 75–88% for three of four models, showing that poor direct-judgment performance need not imply an absence of evidence-responsive discrimination. This CoT gain does not transfer to natural video: across 400 videos, CoT judgment yields no significant accuracy gain for any model and significantly reduces accuracy against event-denial negatives for three of four. Negative type also affects verifier selection: when content-matched negatives are the target, selecting on event-denial performance reduces held-out accuracy by 8.9 percentage points. These results separate success on the negative type tested, ability that can be elicited from the model, and suitability for the intended verification target.

Primary area:
datasets and benchmarks
Keywords:
video verifiers, vision-language models, MLLM-as-a-judge, evaluation methodology, measurement validity, hard negatives, chain-of-thought reasoning, state tracking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kWQrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how negative construction and judgment interfaces affect video-verifier evaluation. It introduces endpoint-matched trajectory negatives and paired truth-switch videos, then examines direct and CoT judgments on controlled tasks and natural videos. The results show that verifier performance and configuration selection depend on the negative type and interface used.

Critical strengths

The evaluation problem is relevant, and the motivation, contributions, and experiments are clearly connected.

The truth-switch design fixes candidate text while changing the correct answer through video evidence.

Candidate-order controls, evidence ablations, and statistical uncertainty analyses strengthen the reported observations.

Critical weaknesses

Internal-trajectory judgments combine spatial grounding, object correspondence, temporal ordering, and state updating; their contributions are insufficiently disentangled.

Direct–CoT comparisons conflate reasoning with prompting, output budgets, parsing, and response biases.

Natural-video negatives lack sufficiently documented independent validation of ambiguity and sampled-evidence visibility.

The broad capability framing exceeds the diagnostic depth; verifier-selection implications remain limited to offline comparisons.

The presentation is conceptually dense, while some related-work distinctions and operational definitions could be sharper.

Endpoint matching removes endpoint shortcuts but does not isolate temporal verification. Internal-trajectory judgments jointly require spatial localization, object correspondence, event ordering, state updating, and text–video alignment. Without controls separating these demands, the performance drop cannot be attributed to a specific verification deficit, limiting the interpretation of the capability analysis.

Questions for authors

Could you add an independent human validation study for the natural-video comparisons? Annotators should verify candidate truth values, identify the contradictory evidence, and assess whether it is visible in the sampled frames. Agreement and ambiguity rates would help establish evaluation validity, while controlled spatial, temporal, and semantic probes are still needed to localize model failures.

Could you introduce staged controls using verified object locations, event sequences, and state trajectories to distinguish perceptual, spatial, temporal, and reasoning failures? The text-record test bypasses several intermediate stages.

1 comment
AuthorsOpenReview ↗

The revised pdf is uploaded; promised results included.

The revised PDF is up, and the tables promised above are in it: Section 5.1 and Appendix H.1 for the validation study, Section 4.3 and Appendix H.4 for the staged controls (Table 51 takes each demand you listed in turn), and Section 2 and Table 70 for the definitions. Two statements in our reply need a qualifier.

  • "Every natural-video result holds" on the validated videos. We re-ran three results there: the interface contrasts and the selection cost on the videos that both annotators validate and see the evidence in (343–351 videos), and the reranking gap on the 385 whose references both confirm. All three hold (Appendix H.1, H.10).
  • "One extraction rule serves all models." The rule is supplemented by an LLM check of the outputs it flags, with the correct answer withheld (Section 2.2; Appendix H.3, H.7, H.8).
Reviewer_rtiCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how negative construction and judgment interfaces affect video-verifier evaluation. It introduces legal endpoint-matched negatives and a paired truth-switch test, then examines interface effects and verifier selection on natural videos. The central finding is that measured success, elicitable capability, and suitability for a target task can differ substantially.

Critical strengths
  1. The controlled constructions and fixed-candidate truth-switch design provide a useful way to test evidence-responsive judgments.
  2. The experiments include meaningful evidence controls, uncertainty estimates, and held-out selection analyses. The paper also candidly discusses position bias, output failures, and limitations.
Critical weaknesses
  1. Limited incremental contribution. The general lessons about hard negatives, interface-dependent performance, and ranking changes are closely related to established contrastive-evaluation and judge-reliability findings. The executable constructions are useful, but the paper should more clearly establish what new scientific insight they provide beyond a controlled demonstration of these known concerns.

  2. Interface effects are not sufficiently disentangled. Direct and CoT judgment differ in prompts, reasoning modes, computational budgets, and answer extraction. The score-level and de-biasing analyses are valuable, but do not fully isolate which changes produce the large gains. Likewise, GLM’s natural-video decline mixes judgment errors with completion failures. A generated-answer baseline without CoT and broader prompt/budget sensitivity tests would strengthen the interpretation and robustness of these findings.

  3. The connection to video-verification capability remains limited. The synthetic inputs explicitly label events and their positions, so success need not demonstrate temporal understanding of unannotated video. In the natural-video study, references are verified against full videos, while models receive sampled frames. The Qwen reappearance check usefully confirms a decline on an evidence-sufficient subset, but does not establish input-level answerability across the whole panel and all models. Broader evidence-sufficiency checks would help separate verifier errors from missing visual information.

  4. Practical selection conclusions need broader validation. The held-out analyses support a mismatch between selection and target negative types on this panel, but involve only eight configurations and one declared target. Repeated partitions do not provide an independent replication, and the all-configuration cost is not significant on the separate 200/200 split. Confirmation on a fresh panel or a realistic mixture of deployment negatives would make the practical implications more convincing.

Questions for authors

Please refer to weaknesses.

1 comment
AuthorsOpenReview ↗

The revised pdf is uploaded; promised results included.

The revised PDF is up, and the tables promised above are in it: Section 5.1 and Appendix H.1 for the validation study, Section 4.3 and Appendix H.4 for the staged controls (Table 51 takes each demand you listed in turn), and Section 2 and Table 70 for the definitions. Two statements in our reply need a qualifier.

  • "Every natural-video result holds" on the validated videos. We re-ran three results there: the interface contrasts and the selection cost on the videos that both annotators validate and see the evidence in (343–351 videos), and the reranking gap on the 385 whose references both confirm. All three hold (Appendix H.1, H.10).
  • "One extraction rule serves all models." The rule is supplemented by an LLM check of the outputs it flags, with the correct answer withheld (Section 2.2; Appendix H.3, H.7, H.8).
Reviewer_WK6sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how negative sample construction and judgment interfaces affect the evaluation of video verifiers. It introduces endpoint-matched negatives and a truth-switch test to examine whether models genuinely rely on video evidence, and compares direct judgment with chain-of-thought reasoning. Experiments suggest that verifier performance and model selection are sensitive to evaluation settings, while improvements observed on synthetic videos do not consistently transfer to natural videos.

Critical strengths
  1. The paper highlights how negative sample construction can influence video verifier evaluation, raising useful concerns about the reliability of existing evaluation protocols.

  2. The truth-switch test and evidence controls provide a systematic way to examine whether verifier decisions depend on visual evidence rather than superficial textual cues.

Critical weaknesses
  1. The paper largely revisits existed ideas, including hard negative construction, contrastive evaluation, chain-of-thought prompting, and evidence perturbation. The proposed endpoint-matched negatives and truth-switch tests are relatively straightforward extensions of existing evaluation methods. The paper does not introduce a new verifier architecture, training algorithm, or fundamentally new evaluation methodology, making its technical contribution rather limited.

  2. The substantial CoT improvements observed on synthetic, highly controlled videos do not transfer to natural videos, where CoT provides no significant accuracy gains and sometimes substantially degrades performance. This discrepancy raises concerns about whether the synthetic experiments capture meaningful challenges in real-world video verification. Moreover, the natural-video evaluation is limited to 400 videos and four models, restricting the generalizability of the conclusions.

  3. Although the paper reports that selecting verifiers using different negative types can change performance, the selection experiments are restricted to a small set of models and offline binary judgments. The reported selection penalty is specific to the chosen evaluation setting and is not consistently significant across alternative splits. Without experiments on actual verifier applications, such as trajectory reranking, reward modeling, or verifier-guided training, it remains unclear whether these observations translate into meaningful practical improvements.

Questions for authors

Can the authors validate the verifier selection findings across more models, datasets, and evaluation?

Can the authors demonstrate how the proposed evaluation insights improve actual verifier selection or downstream applications?

1 comment
AuthorsOpenReview ↗

The revised pdf is uploaded; promised results included.

The revised PDF is up, and the tables promised above are in it: Section 5.5, Table 4 and Appendix H.10 for reranking, Section 5.2 and Appendix H.8 for the second panel and the added models, and Appendix H.9 for selection robustness. The general response maps each point to the main text.

Comments on the paper

2 comments
AuthorsOpenReview ↗

what the paper demonstrates, and the new results by reviewer point

To the AC and all reviewers: thank you for the careful reviews. The revised PDF is uploaded. All three reviews credit the truth-switch design and the evidence controls. The shared reservations are novelty, the source of the gain from reasoning, the validity of the natural-video panel, and practical consequence. This note gives the result and location for each point. Part 2 lists what changed in the text and the numbers, and what did not.

What the paper establishes beyond known concerns (WK6s W1, rtiC W1). Hard negatives, interface sensitivity and ranking changes are known (Section 6, Table 31). The paper answers three questions they leave open.

  1. What does a low score mean? Direct judgment reads the answer from the model's A/B probabilities, with no written reasoning. Against endpoint-matched negatives (trajectories wrong only in the middle) it fails at one step: working out the states from the events. Given the events as text, it still gets both videos of a truth-switch pair right for only 10–13% of pairs (guessing gives 25%); given the states, for 100% (Section 4.3, Figure 3a).
  2. Which way does reasoning move the score? When the frames show each event, written reasoning lifts three of four models from under 10% to 70–100% on that test. It helps all four even against event-denial negatives, which state that the event never occurs (+7 to +20 points). On natural video, where the event must first be found, it brings no significant gain, and against event-denial negatives it costs two of four models about 23 points (Sections 4.2 and 5.2, Figure 4).
  3. What does a mismatched test cost? A verifier chosen on event-denial negatives is 8.9 points less accurate against content-matched negatives, which keep the event and change a detail. That is the kind of error a generator mostly makes: 73 of 100 hand-labeled wrong answers, against 3 that deny the event. When it reranks that generator's answers, it gains 4–5 points where the verifier chosen on content-matched negatives gains 9–10 (Section 5.5, Tables 3 and 4).

What is new, by reviewer point (main text first, then Appendix H).

  • Reasoning, or prompt, budget and answer extraction? (rtiC W2, kWQr W2) Reasoning. Under one protocol for all models, an answer generated without a rationale stays at the direct floor (2–16%), prompt wording moves results by at most 7.5 points, and at most 1 of 240 rationales hits the token cap. A fixed rule extracts each answer, and an LLM assistant checks the outputs it flags, with the correct answer withheld. In the common-protocol runs on natural video this check moves no accuracy by more than 0.50 points. Sections 2.2 and 4.3, Table 2, Figure 3b; Appendix H.3, H.7, H.8.
  • Which step fails? (kWQr W1 and Q2) State updating (finding 1). Table 51 takes each demand kWQr lists in turn. Section 4.3, Figure 3a; Appendix H.4.
  • Do the printed labels do the work? (rtiC W3) In part. With arrows as the only cue, paired success with reasoning is 85%, 48% and 40% for Qwen, GLM and InternVL (100%, 90% and 70% with labels), against at most 4% under direct judgment. Section 4.3, Figure 3c; Appendix H.5.
  • Why does the gain not carry to natural video? (WK6s W2) There the event must first be found, and reasoning does not make that reliable. Asked without candidates, Qwen confirms a reappearance visible in its own frames on only 55% of clips, even with a rationale. A missed event then favors the candidate that denies it. Section 5.2, Figure 4; Appendix H.6, H.7.
  • Human validation, and visibility in the sampled frames (kWQr W3 and Q1, rtiC W3). Two non-author annotators agree on 98.2% of truth values (κ = 0.96), and the contradicting evidence is visible in each model's frames for 90–92% of propositions. The interface contrasts and the selection cost hold on the videos both annotators validate and see the evidence in (343–351 videos). The reranking gap holds on the 385 whose references both confirm. Section 5.1; Appendix H.1, H.10.
  • A second panel and two more models (WK6s W2 and Q1, rtiC W4). On 400 Vinoground pairs, reasoning lowers event-denial accuracy for all four models (3.8 to 10.3 points) and moves content-matched accuracy by at most 2.3. With Kimi-VL-A3B and Aria added, reasoning hurts event-denial accuracy significantly more than content-matched accuracy for four of six models on TOC-Bench and three of six on Vinoground. Section 5.2; Appendix H.8.
  • Selection: robustness and a realistic target (WK6s W3, Q1 and Q2; rtiC W4; kWQr W4). Beyond finding 3, the selection cost is positive in all 21 splits (not significant on the 200/200 split, whose interval is about 13 points wide) and in both cross-panel directions (8.9 and 3.6 points). Selecting on a generator's error mixture instead of event denials gains 7.5 points. Section 5.5, Tables 3 and 4; Appendix H.9, H.10.

We are glad to run further checks.

AuthorsOpenReview ↗

Succinct roadmap about what changed in the text and the numbers, and what did not

What did not change. The title, the two questions, the four findings, and the two constructions (endpoint-matched negatives and the truth-switch test). Every addition answers a point raised in a review and tests a claim the submission already made.

What was rewritten, and what was added.

  • Presentation (scored 2 in all three reviews; kWQr W5). The abstract, the introduction, Section 5 and the discussion are rewritten for clarity, and Sections 4 and 5 have new titles. Figures 3 and 4 of the revision are new. Section 2 and Table 70 define the terms and protocol labels.
  • Prior work (WK6s W1, rtiC W1, kWQr W5). Section 6 is shorter and pairs each line of prior work with what this study adds. Table 31 details the closest comparisons.
  • Scope of the capability claim (kWQr W4, rtiC W3). Section 4 now claims responsiveness to disclosed events in stylized keyframes, not tracking of unannotated motion (end of Section 4.3; Limitations in Section 7).
  • New material in the main text. Section 4.3, "Validation" in 5.1, "Why the effect reverses" and "A second panel" in 5.2, and "Robustness", "A realistic target" and "Reranking" in 5.5. Appendix H is new, and its first paragraph maps each part to the main text.

Numbers that changed, and why. The submission read direct scores at the first token after the prompt, where GLM's two answer labels carry under 1% of the probability. We now read each model at the position where it writes its answer letter; with video, the two labels carry at least 87% of the probability there (Appendix H.2). The main interface comparisons also run all models under one protocol: one set of instructions, an 8,192-token budget, and built-in reasoning modes off. Verifier selection and the rationale analysis keep the submission's per-model protocol, and the second panel its per-model reasoning settings. The consequences:

  • Section 3. Six model-task cells meet the three pre-set criteria instead of seven, and their visible-start win rates are 0.85–0.96 (was 0.78–0.94).
  • Truth-switch test. Paired success goes from 5–9% to 70–100% for three of four models (was 0–6% to 75–88%).
  • Natural video. With reasoning, the decline in accuracy against event-denial negatives is significant for two of four models instead of three. For the third, GLM, it is significant with the model's built-in reasoning on. The difference between the two negative types is significant for three models instead of four, and model gaps change significantly for five of six pairs instead of three.
  • Selection. The cost is unchanged at 8.9 points, because the analysis keeps the submission's inputs. With common-protocol configurations it is 9.4.

Appendix H.11 (Table 69) lists every changed number with its reason. Our reply to rtiC said "no conclusion changes". To be exact, the four findings stand and the counts above changed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.