- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents VID-EVI-QA, a benchmark that measures whether video-language models change their decisions when visual evidence is removed, degraded, or conflicting, rather than relying only on answer accuracy. The benchmark contains 24k condition-level instances across four evidence conditions: Sufficient, Insufficient, Partial, and Conflicting. For the paired conditions, the questions and answer options are kept fixed while the supporting visual evidence is manipulated. The authors also introduce paired metrics, including Evidence-Contingent Success (ECS) and Evidence Withdrawal Rate (EWR), to measure whether models answer correctly when evidence is available and appropriately refuse when the supporting evidence is removed. They evaluate ten open-weight and two closed video-language models and find that high answer accuracy does not necessarily correspond to evidence-aware reasoning, as many models continue answering even after the relevant evidence has been removed. Finally, they propose two training-free inference pipelines that explicitly assess whether each answer option is supported by visual evidence before making a decision. These methods improve evidence-contingent decision making and conflict detection compared with direct prompting, although substantial errors remain.
Critical strengths
The paper looks at a real and important problem. A video model can give a correct answer without actually relying on the video evidence, and accuracy alone cannot show this. This makes it important to know if the model makes different decisions when the evidence is removed.
The benchmark does this by keeping the question and answer options the same, while changing only the evidence. It also covers several evidence states (sufficient, insufficient, partial, conflicting) instead of just "answerable vs. not." This paired design should lets us check how well evidence is used more directly. The results from different models show that accuracy, refusal, and localization are not always aligned. The benckmark in terms of numbers of instances is large enough to get relevant results.
Critical weaknesses
The ECS scores do not explain why a model fails. A model only receives a score if it answers correctly using the evidence and provides an exact reason for rejection when the evidence is removed. Therefore, a low score could mean that the model missed the evidence, guessed, or used different wording. The conflict score also gives credit for saying "several answers fit" without specifying which ones. Ideally, one would like to see scores with and without the reason requirement. For conflict questions, one would also require the model to name the supported answers.
The labels may not match what the model actually sees. Since models only see 32 frames per video, they may miss key moments. Therefore, some "answerable" questions may not have an answer based on the model's input. The authors report human checks, but these are done jointly, so the error rate of the labels is unknown. Annotating a random sample of models for answerability or running experiments with denser sampling would clarify this issue.
It's unclear how helpful the two proposed methods are. Evidence-First learns to reject more, but its accuracy on answerable questions decreases from 39.82% to 19.57% (339–342). Two-pass verification changes several things at once, and it only counts questions for which it produced citations. Testing each part of the two-pass verification method separately and running both methods on at least one more model would help clarify this.
The results are fragile and difficult to reproduce. Small wording changes in the instructions drops the score for 27 points (1701–1708). The prompts are not disclosed, nor are the three models used to filter out 1,492 questions. Reporting scores across several prompt rewordings and disclosing the filtering models would help clarify the results, as would showing results with and without the filter.
Minor: The figures are difficult to read, and their captions are neither explanatory nor self-contained. Captions such as "Two evidence-awareness failures" (line 125) and "Accuracy response" (395), do not indicate what to look for. Figure 3 is crowded with tiny text and tables filled with "…"
Questions for authors
Can you compare a base model with its instruction-tuned or safety-tuned version from the same family, for example Gemma 3 (pretrained vs instruction-tuned), and report ECS and EWR for both? It is unclear whether the scores reflect better evidence use or just how strongly the model was trained to refuse. If the tuned model scores higher mainly because it refuses more, that would show ECS and EWR partly measure refusal training.
See weaknesses for more questions.
12 comments
Input validity and full-set grounded evaluation
Thank you for raising the concerns about whether labels match the sampled inputs and whether evaluating only outputs with valid citations introduces selection bias. We address these separately below.
1. Input-level verification and sensitivity analysis
We added a manual audit of the actual 32-frame input setting and a conservative subset analysis. The main findings persist after excluding examples whose visual support is harder to assess.
We randomly sampled 100 Sufficient–Insufficient pairs, reconstructed the inputs using the evaluation configuration, and examined whether the frames supported the complete question and answer. We then retained 84 pairs with more direct visual support, excluding 16 pairs involving harder-to-assess imagery, additional inference, or answer ambiguity. We recomputed strict ECS/EWR using predictions from the same evaluation versions.
Table 1. Strict ECS/EWR before and after input-level manual screening.
| Method | ECS: original 100 pairs | ECS: retained 84 pairs | EWR: original 100 pairs | EWR: retained 84 pairs |
|---|---|---|---|---|
| Qwen3-VL-8B Direct | 1.00% | 1.19% | 3.13% | 3.33% |
| Matched-image one-pass | 3.00% | 3.57% | 25.00% | 30.00% |
| Two-pass VV | 18.00% | 21.43% | 42.86% | 47.37% |
After screening, Direct still exhibits low strict paired evidence sensitivity, while VV retains its ECS/EWR advantage over both controls. The main findings therefore do not disappear when these harder-to-assess examples are excluded and do not depend on retaining them.
This joint audit is a sensitivity analysis of sample selection, not an independent blinded estimate of the label error rate. We will include the audit protocol, screening criteria, and these results in the revised manuscript.
2. Full-set grounded ECS: missing or invalid citations count as failures
We will clearly distinguish localization statistics conditional on valid citations from full-set paired performance. The former describes localization quality only when the model provides a citation; it does not represent performance over all examples.
To directly test the selection-bias concern, we computed grounded ECS over all 3,787 Sufficient–Insufficient pairs, using the original paper's VV outputs. Success requires a correct Sufficient answer with evidence localization meeting the specified threshold, together with refusal for the insufficient-evidence reason on the paired Insufficient input. Missing or invalid citations count as failures, and every pair remains in the denominator.
Table 2. Grounded ECS over all 3,787 pairs and valid-citation coverage.
| Method | G-ECS at IoP ≥ 0.5 | G-ECS at IoU ≥ 0.5 | Sufficient valid-citation coverage |
|---|---|---|---|
| Qwen3-VL-8B Direct | 1.45% | 1.19% | 97.07% |
| Original two-pass VV | 14.21% | 8.50% | 62.98% |
With all examples included, VV still achieves higher joint success under both localization criteria. The paired gains are 12.75 and 7.31 percentage points, with source-video-clustered 95% confidence intervals of [11.53, 14.02] and [6.39, 8.29] percentage points, respectively.
The improvement on this joint metric therefore does not depend on restricting evaluation to examples with valid citations.
Evidence-First benefits and accuracy trade-offs
Thank you for highlighting the accuracy cost of Evidence-First. We agree that this decrease requires explicit discussion. However, a decrease in Sufficient accuracy alone does not establish whether Evidence-First improves behavior under changes in evidence availability. The task requires both answering correctly when evidence is sufficient and appropriately refusing when it is insufficient, so both conditions must be examined jointly.
On the same 3,787 Sufficient–Insufficient pairs, we obtain:
Table. Answer accuracy and strict/adjusted relaxed paired metrics.
| Metric | Qwen3-VL-8B Direct | Evidence-First |
|---|---|---|
| Sufficient accuracy | 39.82% | 19.57% |
| Strict ECS | 1.45% | 11.49% |
| Strict EWR | 3.65% | 58.70% |
| Adjusted relaxed ECS | 6.26% | 11.49% |
| Adjusted relaxed EWR | 15.72% | 58.70% |
The adjusted relaxed rule is the same as in our response on ECS interpretation: it does not require the exact refusal reason, but does not count Conflict refusals explicitly claiming that at least two options are supported as successful. All examples are retained. ECS uses all 3,787 pairs as its denominator; EWR uses the questions answered correctly in the Sufficient condition by each method.
Evidence-First improves paired success, not merely the number of refusals. Strict ECS requires a correct Sufficient answer and refusal for the correct reason on the corresponding Insufficient input. Its increase of 10.04 percentage points uses the same full denominator, so this improvement is not merely an effect of EWR's smaller conditional denominator. An always-refuse strategy cannot score any ECS successes.
This gain also does not depend entirely on requiring the exact refusal reason. Under the adjusted relaxed rule, ECS still increases from 6.26% to 11.49%, a gain of 5.23 percentage points. Thus, the paired-success advantage remains when the reason requirement is relaxed without rewarding explicit claims that multiple options are supported.
We do not interpret these results as showing that Evidence-First is uniformly better than direct inference. The decrease in Sufficient accuracy demonstrates a substantial cost from conservative behavior. The conclusion is that Evidence-First improves the evaluated paired evidence behavior, but does not yet resolve the trade-off between correct answering and appropriate refusal.
This also illustrates the contribution of our evaluation: controlled evidence interventions and decomposed metrics distinguish answer accuracy, refusal behavior, and option-level evidence-support claims, revealing benefits and costs that neither accuracy nor refusal rate alone can capture. We will report these metrics together in the revised manuscript, clarify the scope of Evidence-First's benefits, and narrow any wording that could imply across-the-board performance improvements.
ECS interpretation and conflict scoring
Thank you for these suggestions. We address both the interpretation of ECS and the requirement to identify supported options in conflict cases.
1. ECS decomposition: refusal behavior versus evidence-support judgments
We agree that ECS is a composite metric: its aggregate value alone cannot identify the source of failure. To distinguish evidence-support judgments from refusal decisions, we analyzed Qwen3-VL-8B Direct under three scoring rules. All metrics below retain the full denominator of 3,787 pairs.
ECS additionally requires a correct answer on the paired Sufficient input. The adjustment changes only which outputs count as successes; no examples are removed and the denominator is unchanged.
Our semantic examination identified 1,998 Conflict refusals explicitly claiming that at least two distinct options were supported. Outputs that merely mentioned options while denying their support were not included in this category. Thus, removing the reason requirement entirely counts two different judgments—"evidence is insufficient" and "multiple answers are supported"—as equally successful. We therefore compare strict ECS with the adjusted relaxed measure.
This analysis shows that ECS involves both grounding-level evidence-support judgments and decision-level answer, refusal, and reason selection. The differences cannot all be attributed to alternative wording of a refusal. We will report strict and adjusted relaxed ECS together in the paper, reducing the influence of the final refusal-reason selection and expression requirements while helping distinguish support judgments from decision factors. This examination concerns support claims in model outputs; it does not establish that those claims are visually grounded or independently quantify the contribution of each factor.
2. Conflict scoring already includes option-level verification
Our evaluation distinguishes detecting a conflict from identifying the supported options; not all conflict metrics reward merely stating that multiple answers fit.
Conflict Detect measures whether a model refuses and correctly identifies conflicting evidence. The original strict Conflict metric additionally requires it to explicitly identify at least two supported options, without including any option outside the ground-truth support set. The reviewer's proposed option-level requirement is therefore already part of our strict evaluation. We acknowledge that the naming and distinction between these metrics were insufficiently clear and will clarify them in the revision.
We additionally evaluated exact support-set matching: a model must correctly detect the conflict and recover the entire ground-truth support set. On the same 1,635 conflict questions:
VV's gains are not limited to producing "conflict" more frequently. Its performance remains higher than Direct and matched-image one-pass when explicit identification of supported options is required. Even under exact support-set matching, VV exceeds matched-image one-pass by 17.43 percentage points.
These results support the contribution of option-level evidence assessment and verification to conflict recognition and support-set identification, beyond increasing refusal. We will report all three metrics side by side to clearly distinguish conflict detection from supported-option identification.