acceptodds
Under review as a conference paper at ICLR 2027

Vid-Evi-QA: A Controlled Benchmark for Evidence-Adaptive Video Reasoning

Abstract

Existing video question answering (VideoQA) benchmarks assess whether models answer or refuse correctly, but not whether decisions adapt to changes in relevant visual evidence. We introduce Vid-Evi-QA, a controlled evidence-intervention benchmark spanning Sufficient, Insufficient, Partial, and Conflicting evidence, with 24,357 condition-level instances from 5,422 questions and 1,739 videos. It intervenes on temporal evidence while holding questions fixed and separately constructs cases with multiple visually supported options. Same-question pairs and matched non-evidence controls distinguish answering ability, refusal tendency, and evidence sensitivity. Evaluations of ten open-weight video-language models across all four evidence conditions reveal a clear separation between these capabilities: the strongest answerer attains 70.53% Sufficient accuracy but only 1.87% strict Evidence Withdrawal Rate (EWR; correct-reason withdrawal conditional on initial correctness) and 1.32% joint Evidence-Contingent Success (ECS); the highest Evidence-Specific Success in matched-control experiments is just 12.68%. Unstable decisions under graded evidence and failures to recognize and appropriately reject multiple supported options reveal further limitations. We also introduce two training-free inference methods that explicitly assess option-level evidence support. Across all 9,209 non-Partial instances, two-pass visual verification improves the reason-correct paired success of the backbone from 1.45% to 18.01% and conflict detection from 5.44% to 50.95%. These results demonstrate the potential of explicit support assessment to improve evidence-aware perception and decision making, and show that video reasoning evaluation must go beyond static accuracy and refusal rates to directly test whether decisions adapt appropriately to changes in supporting evidence.

Primary area:
datasets and benchmarks
Keywords:
video question answering, video-language models, evidence-contingent reasoning, counterfactual benchmark, temporal grounding, evidence-aware abstention, training-free inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uH12OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents VID-EVI-QA, a benchmark that measures whether video-language models change their decisions when visual evidence is removed, degraded, or conflicting, rather than relying only on answer accuracy. The benchmark contains 24k condition-level instances across four evidence conditions: Sufficient, Insufficient, Partial, and Conflicting. For the paired conditions, the questions and answer options are kept fixed while the supporting visual evidence is manipulated. The authors also introduce paired metrics, including Evidence-Contingent Success (ECS) and Evidence Withdrawal Rate (EWR), to measure whether models answer correctly when evidence is available and appropriately refuse when the supporting evidence is removed. They evaluate ten open-weight and two closed video-language models and find that high answer accuracy does not necessarily correspond to evidence-aware reasoning, as many models continue answering even after the relevant evidence has been removed. Finally, they propose two training-free inference pipelines that explicitly assess whether each answer option is supported by visual evidence before making a decision. These methods improve evidence-contingent decision making and conflict detection compared with direct prompting, although substantial errors remain.

Critical strengths

The paper looks at a real and important problem. A video model can give a correct answer without actually relying on the video evidence, and accuracy alone cannot show this. This makes it important to know if the model makes different decisions when the evidence is removed.

The benchmark does this by keeping the question and answer options the same, while changing only the evidence. It also covers several evidence states (sufficient, insufficient, partial, conflicting) instead of just "answerable vs. not." This paired design should lets us check how well evidence is used more directly. The results from different models show that accuracy, refusal, and localization are not always aligned. The benckmark in terms of numbers of instances is large enough to get relevant results.

Critical weaknesses

The ECS scores do not explain why a model fails. A model only receives a score if it answers correctly using the evidence and provides an exact reason for rejection when the evidence is removed. Therefore, a low score could mean that the model missed the evidence, guessed, or used different wording. The conflict score also gives credit for saying "several answers fit" without specifying which ones. Ideally, one would like to see scores with and without the reason requirement. For conflict questions, one would also require the model to name the supported answers.

The labels may not match what the model actually sees. Since models only see 32 frames per video, they may miss key moments. Therefore, some "answerable" questions may not have an answer based on the model's input. The authors report human checks, but these are done jointly, so the error rate of the labels is unknown. Annotating a random sample of models for answerability or running experiments with denser sampling would clarify this issue.

It's unclear how helpful the two proposed methods are. Evidence-First learns to reject more, but its accuracy on answerable questions decreases from 39.82% to 19.57% (339–342). Two-pass verification changes several things at once, and it only counts questions for which it produced citations. Testing each part of the two-pass verification method separately and running both methods on at least one more model would help clarify this.

The results are fragile and difficult to reproduce. Small wording changes in the instructions drops the score for 27 points (1701–1708). The prompts are not disclosed, nor are the three models used to filter out 1,492 questions. Reporting scores across several prompt rewordings and disclosing the filtering models would help clarify the results, as would showing results with and without the filter.

Minor: The figures are difficult to read, and their captions are neither explanatory nor self-contained. Captions such as "Two evidence-awareness failures" (line 125) and "Accuracy response" (395), do not indicate what to look for. Figure 3 is crowded with tiny text and tables filled with "…"

Questions for authors

Can you compare a base model with its instruction-tuned or safety-tuned version from the same family, for example Gemma 3 (pretrained vs instruction-tuned), and report ECS and EWR for both? It is unclear whether the scores reflect better evidence use or just how strongly the model was trained to refuse. If the tuned model scores higher mainly because it refuses more, that would show ECS and EWR partly measure refusal training.

See weaknesses for more questions.

12 comments
AuthorsOpenReview ↗

ECS interpretation and conflict scoring

Thank you for these suggestions. We address both the interpretation of ECS and the requirement to identify supported options in conflict cases.

1. ECS decomposition: refusal behavior versus evidence-support judgments

We agree that ECS is a composite metric: its aggregate value alone cannot identify the source of failure. To distinguish evidence-support judgments from refusal decisions, we analyzed Qwen3-VL-8B Direct under three scoring rules. All metrics below retain the full denominator of 3,787 pairs.

Scoring ruleInsufficient refusals counted as successfulECS
Strict: refusal with the insufficient-evidence reason108/3,787 (2.85%)55/3,787 (1.45%)
Relaxed: refusal for any reason2,750/3,787 (72.62%)739/3,787 (19.51%)
Adjusted relaxed: exclude Conflict refusals explicitly claiming support for at least two options752/3,787 (19.86%)237/3,787 (6.26%)

ECS additionally requires a correct answer on the paired Sufficient input. The adjustment changes only which outputs count as successes; no examples are removed and the denominator is unchanged.

Our semantic examination identified 1,998 Conflict refusals explicitly claiming that at least two distinct options were supported. Outputs that merely mentioned options while denying their support were not included in this category. Thus, removing the reason requirement entirely counts two different judgments—"evidence is insufficient" and "multiple answers are supported"—as equally successful. We therefore compare strict ECS with the adjusted relaxed measure.

This analysis shows that ECS involves both grounding-level evidence-support judgments and decision-level answer, refusal, and reason selection. The differences cannot all be attributed to alternative wording of a refusal. We will report strict and adjusted relaxed ECS together in the paper, reducing the influence of the final refusal-reason selection and expression requirements while helping distinguish support judgments from decision factors. This examination concerns support claims in model outputs; it does not establish that those claims are visually grounded or independently quantify the contribution of each factor.

2. Conflict scoring already includes option-level verification

Our evaluation distinguishes detecting a conflict from identifying the supported options; not all conflict metrics reward merely stating that multiple answers fit.

Conflict Detect measures whether a model refuses and correctly identifies conflicting evidence. The original strict Conflict metric additionally requires it to explicitly identify at least two supported options, without including any option outside the ground-truth support set. The reviewer's proposed option-level requirement is therefore already part of our strict evaluation. We acknowledge that the naming and distinction between these metrics were insufficiently clear and will clarify them in the revision.

We additionally evaluated exact support-set matching: a model must correctly detect the conflict and recover the entire ground-truth support set. On the same 1,635 conflict questions:

MetricQwen3-VL-8B DirectMatched-image one-passTwo-pass VV
Conflict Detect5.44%29.48%50.95%
Original strict Conflict3.06%14.68%41.90%
Exact support-set match with correct conflict detection1.77%10.70%28.13%

VV's gains are not limited to producing "conflict" more frequently. Its performance remains higher than Direct and matched-image one-pass when explicit identification of supported options is required. Even under exact support-set matching, VV exceeds matched-image one-pass by 17.43 percentage points.

These results support the contribution of option-level evidence assessment and verification to conflict recognition and support-set identification, beyond increasing refusal. We will report all three metrics side by side to clearly distinguish conflict detection from supported-option identification.

AuthorsOpenReview ↗

Input validity and full-set grounded evaluation

Thank you for raising the concerns about whether labels match the sampled inputs and whether evaluating only outputs with valid citations introduces selection bias. We address these separately below.

1. Input-level verification and sensitivity analysis

We added a manual audit of the actual 32-frame input setting and a conservative subset analysis. The main findings persist after excluding examples whose visual support is harder to assess.

We randomly sampled 100 Sufficient–Insufficient pairs, reconstructed the inputs using the evaluation configuration, and examined whether the frames supported the complete question and answer. We then retained 84 pairs with more direct visual support, excluding 16 pairs involving harder-to-assess imagery, additional inference, or answer ambiguity. We recomputed strict ECS/EWR using predictions from the same evaluation versions.

Table 1. Strict ECS/EWR before and after input-level manual screening.

MethodECS: original 100 pairsECS: retained 84 pairsEWR: original 100 pairsEWR: retained 84 pairs
Qwen3-VL-8B Direct1.00%1.19%3.13%3.33%
Matched-image one-pass3.00%3.57%25.00%30.00%
Two-pass VV18.00%21.43%42.86%47.37%

After screening, Direct still exhibits low strict paired evidence sensitivity, while VV retains its ECS/EWR advantage over both controls. The main findings therefore do not disappear when these harder-to-assess examples are excluded and do not depend on retaining them.

This joint audit is a sensitivity analysis of sample selection, not an independent blinded estimate of the label error rate. We will include the audit protocol, screening criteria, and these results in the revised manuscript.

2. Full-set grounded ECS: missing or invalid citations count as failures

We will clearly distinguish localization statistics conditional on valid citations from full-set paired performance. The former describes localization quality only when the model provides a citation; it does not represent performance over all examples.

To directly test the selection-bias concern, we computed grounded ECS over all 3,787 Sufficient–Insufficient pairs, using the original paper's VV outputs. Success requires a correct Sufficient answer with evidence localization meeting the specified threshold, together with refusal for the insufficient-evidence reason on the paired Insufficient input. Missing or invalid citations count as failures, and every pair remains in the denominator.

Table 2. Grounded ECS over all 3,787 pairs and valid-citation coverage.

MethodG-ECS at IoP ≥ 0.5G-ECS at IoU ≥ 0.5Sufficient valid-citation coverage
Qwen3-VL-8B Direct1.45%1.19%97.07%
Original two-pass VV14.21%8.50%62.98%

With all examples included, VV still achieves higher joint success under both localization criteria. The paired gains are 12.75 and 7.31 percentage points, with source-video-clustered 95% confidence intervals of [11.53, 14.02] and [6.39, 8.29] percentage points, respectively.

The improvement on this joint metric therefore does not depend on restricting evaluation to examples with valid citations.

AuthorsOpenReview ↗

Evidence-First benefits and accuracy trade-offs

Thank you for highlighting the accuracy cost of Evidence-First. We agree that this decrease requires explicit discussion. However, a decrease in Sufficient accuracy alone does not establish whether Evidence-First improves behavior under changes in evidence availability. The task requires both answering correctly when evidence is sufficient and appropriately refusing when it is insufficient, so both conditions must be examined jointly.

On the same 3,787 Sufficient–Insufficient pairs, we obtain:

Table. Answer accuracy and strict/adjusted relaxed paired metrics.

MetricQwen3-VL-8B DirectEvidence-First
Sufficient accuracy39.82%19.57%
Strict ECS1.45%11.49%
Strict EWR3.65%58.70%
Adjusted relaxed ECS6.26%11.49%
Adjusted relaxed EWR15.72%58.70%

The adjusted relaxed rule is the same as in our response on ECS interpretation: it does not require the exact refusal reason, but does not count Conflict refusals explicitly claiming that at least two options are supported as successful. All examples are retained. ECS uses all 3,787 pairs as its denominator; EWR uses the questions answered correctly in the Sufficient condition by each method.

Evidence-First improves paired success, not merely the number of refusals. Strict ECS requires a correct Sufficient answer and refusal for the correct reason on the corresponding Insufficient input. Its increase of 10.04 percentage points uses the same full denominator, so this improvement is not merely an effect of EWR's smaller conditional denominator. An always-refuse strategy cannot score any ECS successes.

This gain also does not depend entirely on requiring the exact refusal reason. Under the adjusted relaxed rule, ECS still increases from 6.26% to 11.49%, a gain of 5.23 percentage points. Thus, the paired-success advantage remains when the reason requirement is relaxed without rewarding explicit claims that multiple options are supported.

We do not interpret these results as showing that Evidence-First is uniformly better than direct inference. The decrease in Sufficient accuracy demonstrates a substantial cost from conservative behavior. The conclusion is that Evidence-First improves the evaluated paired evidence behavior, but does not yet resolve the trade-off between correct answering and appropriate refusal.

This also illustrates the contribution of our evaluation: controlled evidence interventions and decomposed metrics distinguish answer accuracy, refusal behavior, and option-level evidence-support claims, revealing benefits and costs that neither accuracy nor refusal rate alone can capture. We will report these metrics together in the revised manuscript, clarify the scope of Evidence-First's benefits, and narrow any wording that could imply across-the-board performance improvements.

Reviewer_eZDnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether VideoQA models change their answers and refusals when relevant visual evidence changes. VID-EVI-QA contains 5,422 questions from 1,739 videos and 24,357 condition-level instances. It combines Sufficient–Insufficient pairs and four Partial variants for 3,787 questions with a separate set of 1,635 Conflicting questions that have multiple supported options. Strict ECS measures correct answering followed by refusal for the missing-evidence reason; strict EWR measures this withdrawal among initially correct answers. Ten open-weight models are evaluated across all four conditions. The results show that strong answer accuracy can coexist with poor evidence-contingent decisions. The paper also proposes two training-free pipelines using Qwen3-VL-8B. Two-pass visual verification raises strict ECS from 1.45% to 18.01% and Conflict Detect from 5.44% to 50.95% (Table 2).

Critical strengths
  1. The benchmark separates correct answers from appropriate responses to evidence changes. Questions and options stay fixed within the paired track, while matched non-evidence perturbations help distinguish evidence-specific responses from general sensitivity to corruption. ECS and EWR also separate joint success from withdrawal conditional on an initially correct answer. This makes the benchmark useful for evaluating behavior that answer accuracy or refusal frequency alone would miss (Sections 2–3; Appendix E.7).

  2. The experiments provide a concrete diagnosis of the gap between answering and evidence sensitivity. For example, LLaVA-OneVision-7B reaches 70.53% Sufficient accuracy but only 1.32% strict ECS and 1.87% strict EWR. The separate multiple-support track further tests whether models recognize that a question has no unique supported answer. These results make the evaluation problem clear and relevant beyond a single model (Table 2; Sections 5.2–5.3).

Critical weaknesses

W1. Confidence in the final Insufficient labels remains my main concern. The paper includes substantial residual-evidence checks, and the final review of 100 Sufficient and 100 randomly sampled Insufficient questions found no issues. However, Appendix C.2 states that calibration recall on known positive controls does not estimate recall for unknown residual evidence. Appendix C.7 also explains that the two reviewers worked jointly and that release-wide ensemble-check coverage is not established. These are limitations of the available validation, rather than evidence that the labels are wrong. If relevant support remains, a model could use it and still be penalized for failing to withdraw. A clearer estimate of the remaining label uncertainty, and its effect on paired scores, would increase my confidence in the benchmark’s central conclusions (Appendices C.2 and C.7).

W2. The inference results establish a benefit for the full pipeline, but leave the contribution of the verification design unclear. Two-pass VV changes the number of calls, context, frame-location hints, prompting, and support-to-decision procedure relative to direct inference. The matched-image one-pass control is useful: its strict ECS is 2.30%, compared with 18.01% for VV. However, this comparison still changes several elements together, as the authors acknowledge. Appendix G studies support decomposition separately and does not provide component ablations of the full pipelines. I therefore take the gains as evidence for the combined inference procedure, with less evidence about which design choices are necessary. This limits the methodological contribution I can infer from the improvements (Section 5.4; Appendices G.1, G.8, and H.2–H.4).

Questions for authors
  1. Reliability of the Insufficient labels (clarification). Appendix C.7 reports no issues in the final random sample of 100 Insufficient questions, but the reviewers worked jointly and release-wide ensemble-check coverage is not established. How many of the 3,787 final Insufficient questions completed that ensemble check, and how were any unresolved cases handled before inclusion? Please distinguish final-release checks from development or targeted inspections. If a subset with independently verified labels is available, do the strict ECS/EWR findings remain consistent on it? This would help assess whether failed withdrawal could reflect remaining visual support.

  2. Contribution of frame-location hints (suggested limited validation). Appendix H.4 reports 2.30% strict ECS for the one-pass support-set control, compared with 18.01% for two-pass VV. To separate the value of transferred locations from an additional verification pass, could you compare VV with the same two-pass procedure but omit the frame hints? The 32 frames, prompts, output budget, retry policy, and final decision rule should stay fixed. Reporting strict ECS and its paired difference from VV on the same question pairs would show whether the first pass provides useful guidance beyond a fresh second assessment (Appendices H.2–H.4).

  3. Definition of the conflict-control result (clarification). Section 5.4 calls the one-pass control’s 14.68% result “Conflict Detect,” whereas Appendix H.4 calls it “strict Conflict.” These criteria differ: VV obtains 50.95% Conflict Detect in Table 2 and 41.90% Strict Conflict in Table 23. Which criterion was used for the control’s 14.68%? Please report the control and VV under the same criterion on the same Conflicting questions. This would make the size of the conflict-recognition gain interpretable.

  4. Evidence availability in the sampled input (clarification). Appendix E.4 samples 32 frames uniformly, and Appendix I.1 checks the actual sampled frames for the error audits. Across the paired set, how often do these 32-frame Sufficient inputs retain enough evidence to verify the complete question and answer? If existing audits identify a verified subset, could you report whether the main ECS/EWR findings hold on it? Please distinguish overlap with an annotated interval from sufficient visual information. This would help separate missing input evidence from failure to use evidence that the model actually received.

  5. Evidence-specificity of the VV improvement (suggested limited validation). Tables 3 and 12 give matched-control results for three direct models, but not VV. Could you compare VV with direct Qwen3-VL-8B on the existing 800-question control set using the same inputs, corruption seeds, and weights, and report ESS, cESS, and their paired differences? This would test whether VV still answers correctly after non-evidence corruption while withdrawing after evidence corruption. Since this control uses noise rather than the main Insufficient replacement, the result should be interpreted within that control protocol (Section 3.2; Appendix E.7).

  6. Grounded success on complete question pairs (request from existing predictions). Appendix E.6 gives 62.98% valid-evidence coverage for VV and correctly distinguishes conditional localization means from full-population scores. Could you report grounded ECS from Equation (7), at IoP and IoU thresholds of 0.5, for direct Qwen3-VL-8B and VV over the same 3,787 pairs, with absent or invalid evidence receiving zero credit? Please retain the stated frame-cell reconstruction and explain its difference from direct interval predictions. This would show how often correct answering, grounding, and correct-reason withdrawal succeed together.

12 comments
AuthorsOpenReview ↗

Ensemble-check coverage and evaluation on manually reviewed samples

Thank you for requesting clarification of the validation coverage. Our leakage-checking procedure uses a routing strategy. We use the construction model’s probability signals to screen candidate windows only when it answers correctly on both the full-video and evidence-only inputs, with a visual gain greater than 10−610^{-6} in both settings. For the 1,755 questions that did not satisfy this criterion, we used dense visual inspection followed by independent visual grounding from VideoLLaMA3-7B, Qwen3-VL-32B, and Qwen2.5-VL-32B. This avoids interpreting the construction model’s failure to use evidence as evidence that no residual information remains.

Of these 1,755 questions, 1,654 are included in the final set of 3,787 questions. Thus, the ensemble check was a targeted verification mechanism for questions that did not pass the construction model’s behavioral check, rather than an additional check applied uniformly to every question. Under the stated protocol, leakage was confirmed when independent observations from at least two different grounding models supported it. Single-model positives, uncertain cases, and incomplete responses underwent further review. The final released set contains no unresolved cases. We will clarify the routing criteria, actual coverage, and final disposition in Appendix C.7.

To further examine whether label uncertainty affects the paired evaluation, we aggregated two random manual-review batches of 100 Sufficient–Insufficient pairs collected during rebuttal. The batches overlap by four pairs, yielding 196 unique pairs from 174 source videos. We examined the 32-frame inputs reconstructed using the evaluation configuration, checking whether the Sufficient input supported the complete answer and whether the Insufficient input retained enough visual evidence to answer. All sampled pairs were retained following final review; none were selected based on model performance.

Using the same prediction versions as the paper’s main table, we obtain:

Table 1. Strict ECS and EWR on the full benchmark and manually reviewed subsets (%).

Evaluation setPairsDirect ECSTwo-pass VV ECSDirect EWRTwo-pass VV EWR
Full benchmark3,7871.4518.013.6541.26
Manual-review batch 11001.0018.003.1342.86
Manual-review batch 21002.0015.005.2641.67
Deduplicated manual-review subset1961.5316.844.4142.86

Direct denotes Qwen3-VL-8B. Strict ECS requires a correct answer on the Sufficient input and abstention for the correct reason—insufficient evidence—on the paired Insufficient input. Strict EWR uses each method’s correctly answered Sufficient questions as its denominator.

The method comparison on the reviewed subset is consistent with the full-benchmark results. On the 196 unique pairs, VV improves strict ECS over Direct by 15.31 percentage points, with a 95% confidence interval of [10.24, 20.71], and strict EWR by 38.45 percentage points, with a 95% confidence interval of [27.27, 49.75]. Intervals use 10,000 paired bootstrap resamples clustered by source video.

These results support the persistence of VV’s paired-performance advantage on manually reviewed inputs. This review is a robustness check, rather than an independent blinded estimate of the dataset-wide label error rate. We will report its protocol and results explicitly in the revision.

AuthorsOpenReview ↗

Controlled ablation of frame-location hints in two-pass verification

Thank you for suggesting this controlled ablation. We compared the same two-pass VV pipeline with and without frame-location hints on all 3,787 Sufficient–Insufficient pairs, keeping the model, 32-frame inputs, output budget, retry policy, and decision rules fixed.

Table 1. Effect of frame-location hints within the same two-pass pipeline.

SettingStrict ECS
Two-pass VV without frame-location hints16.85%
Two-pass VV with frame-location hints17.69%
Paired difference, with 95% confidence interval+0.84 percentage points [−0.13, +1.83]

The confidence interval was computed using 2,000 paired bootstrap resamples clustered by source video. Failed samples remain in the full denominator. The point estimate favors location hints, but their independent contribution to strict ECS is not statistically established.

To further clarify the role of the second pass, we also compared the original direct prompting, the structured first pass, and the complete two-pass pipeline.

Table 2. Direct prompting, the structured first pass, and complete two-pass VV.

MetricOriginal Qwen3 DirectStructured first passComplete two-pass VV
Sufficient accuracy ↑39.82%40.08%43.20%
Sufficient abstention rate ↓49.54%46.71%40.67%
Insufficient correct-reason abstention rate ↑2.85%61.50%55.72%
Strict ECS ↑1.45%17.61%17.69%
Strict EWR ↑3.65%43.94%40.95%

Compared with direct prompting, the structured first pass substantially improves correct-reason abstention and strict ECS while largely preserving Sufficient accuracy. This supports the combined structured evidence-assessment and decision procedure, without isolating the contribution of any individual prompt component.

The additional benefit of the second pass is primarily the recovery of correct answers under sufficient evidence. Relative to the first pass, it reduces Sufficient abstention by 6.04 percentage points and increases accuracy by 3.12 percentage points. At the question level, 241 Sufficient examples that received an abstention in the first pass were answered correctly in the second pass. After accounting for other corrections and regressions, the net increase is 118 correct answers. Thus, the second pass does not simply increase abstention; it also corrects some overly conservative decisions.

This benefit comes with a 5.78-percentage-point decrease in correct-reason abstention on Insufficient inputs, reflecting a trade-off between recovering answerable questions and maintaining appropriate abstention. Consequently, the net change in strict ECS is small, and strict EWR decreases.

We will include these ablations in the revision, clarify the second pass’s benefit in recovering correct answers, and narrow our claims about the independent contribution of location hints and the second pass’s improvement in paired performance.

AuthorsOpenReview ↗

Clarifying the 14.68% conflict score and comparing methods under identical criteria

Thank you for pointing out the inconsistent metric naming between the main text and the appendix. We checked the original predictions and scoring code and confirmed that the one-pass score of 14.68% (240/1,635) corresponds to strict Conflict, not Conflict Detect. We will correct the name in the main text.

Conflict Detect requires a valid abstention with the correct reason of conflicting evidence. The original strict Conflict metric additionally requires the model to explicitly identify at least two supported options, without including any option outside the ground-truth support set. However, it permits omission of some supported options and therefore differs from exact matching of the complete support set.

Using the same 1,635 conflict questions and identical scoring criteria, we obtain:

Table 1. Conflict evaluation under consistent scoring criteria.

MetricMatched-image one-passTwo-pass VV
Conflict Detect482/1,635 = 29.48%833/1,635 = 50.95%
Original strict Conflict240/1,635 = 14.68%685/1,635 = 41.90%
Exact support-set match with correct conflict detection175/1,635 = 10.70%460/1,635 = 28.13%

VV achieves higher scores under all three consistent criteria, including the stricter requirement of recovering the complete supported-answer set. We will align the metric names, definitions, and comparisons throughout the main text and appendix, avoiding any comparison between one-pass strict Conflict and VV Conflict Detect.

Reviewer_ZD3dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors address the topic of evidence-grounding in video QA - while prior benchmarks (e.g., NExT-GQA) have mainly focused on assessing joint grounding + QA, this benchmark goes a step further and "intervenes" on the evidence to see how the model changes its answer. This is useful for cases when additional or different evidence can be introduced that complicates video reasoning. There are two parts: (1) 3,787 multiple-choice questions from NExT-GQA: "sufficient evidence" is the same original video, while "insufficient" replaces the evidence area with blurred frozen frames from somewhere else (correct answer then is to refuse to answer), and "partial" is a more intermediate state. (2) 1,635 questions built from E.T. Bench, where 2 or more answer options are supported by the video. Overall, the preliminary rating for the paper is WR pending rebuttal.

Critical strengths

The paper addresses an important analysis point which isn't strongly assessed in the current NExT-GQA benchmark.

Critical weaknesses

The paper's findings seem to rest on models that are somewhat out of date, particularly on the higher end. e.g., GPT-4o and Gemini 2.5 Pro in Table 2 are quite old - the state-of-the-art now is substantially better, and it's unclear if some of the core findings (e.g., that overall accuracy doesn't align with refusal rates) will hold. Further, the method for creating a video with insufficient evidence introduces a very specific flavor of artifact, so it's not clear if these models are just detecting that there is this artifact, or if there is truly insufficient evidence. This would need some significant revision to likely address. There also don't seem to be clear simple baselines (e.g., a text-only one, few frame baseline, or human baseline); these are relevant since it's possible that NExT-GQA's annotations don't actually remove all the evidence and a smart model could actually have sufficient information to answer the question (the human baseline would verify this). Prompts across these models also don't seem to be provided, so possible there are some confounders in the analysis.

Questions for authors

Please address the points raised in weaknesses, thanks!

4 comments
AuthorsOpenReview ↗

Evaluation of newer models

Thank you for suggesting evaluation of more recent models. We have added GPT-6.1-sol and Gemini-4-hydrogen, using the same v2.2 prompts and 32-frame input protocol on a shared stratified subset of 150 Sufficient–Insufficient pairs and 100 Conflicting examples.

ModelSufficient accuracyStrict ECSStrict EWRRelaxed ECSRelaxed EWRConflict detection
GPT-6.1-sol58.0013.3322.9916.6728.7479.00
Gemini-4-hydrogen58.6734.6759.0935.3360.2362.00

All values are percentages and are unweighted results on this subset. Strict ECS requires a correct answer in the Sufficient condition and refusal for the correct reason in the paired Insufficient condition. Strict EWR measures this success rate among examples answered correctly in the Sufficient condition. The relaxed versions remove only the refusal-reason requirement. Removing the full output-format validity requirement leaves all metrics in the table unchanged.

These additional experiments support three conclusions:

  1. Similar answer accuracy does not imply similar evidence-dependent behavior. Gemini withdraws answers more appropriately after evidence removal, whereas GPT performs better at conflict detection. Answer accuracy, evidence-based withdrawal, and conflict recognition therefore capture distinct capabilities and must be evaluated separately.

  2. The shortfall in evidence-dependent behavior cannot be attributed solely to the refusal-reason requirement. Even without this requirement, a substantial gap remains between answering correctly when evidence is available and appropriately refusing after it is removed.

  3. The paper's central phenomenon persists in the newly evaluated closed-source models, with substantial differences between models. Both models continue to give definite answers after evidence removal on some questions they originally answered correctly. Stronger model capabilities therefore do not automatically produce reliable evidence-dependent decisions.

AuthorsOpenReview ↗

Intervention artifacts and evidence location

Thank you for raising this concern. We added controls varying both the intervention type and its location. All experiments use the same 759 questions from 522 source videos. Starting from a stratified sample of 800 questions, we excluded 41 before inference because matched non-evidence-region controls could not be constructed. Selection did not depend on model performance.

Consistency across intervention types. As Table 1 shows, refusal rates after evidence-region interventions are generally close across freezing alone, freezing with blur, and freezing with noise. Under VV, the range across these interventions is at most 1.37 percentage points for each of the three models. Direct inference shows model-dependent sensitivity, with a larger variation for Qwen2.5-VL-7B. Thus, the observed refusal response is not confined to the specific freeze-and-blur construction.

Table 1. Actual refusal rates (%) for original inputs and three evidence-region interventions.

ModelMethodOriginalFreezeFreeze+blurFreeze+noise
Qwen3-VL-8BDirect84.6992.4493.3093.38
Qwen3-VL-8BVV44.5659.2360.3259.18
InternVL3.5-8BDirect78.5687.3088.6289.50
InternVL3.5-8BVV48.8959.0460.3559.36
Qwen2.5-VL-7BDirect55.2470.0575.4475.03
Qwen2.5-VL-7BVV15.6218.0919.4618.36

Evidence versus non-evidence regions. Applying the same freeze-and-blur intervention to non-evidence regions produces smaller changes in refusal rates overall, especially under VV, where changes are close to zero. Evidence-region interventions produce larger increases. In every model–method configuration, the difference between intervention locations exceeds the corresponding range across intervention types (Table 2). In this experiment, behavioral differences associated with intervention location are therefore larger than those associated with intervention type, supporting a relationship between model responses and the evidential relevance of the removed content.

Table 2. Refusal-rate changes under freeze-and-blur interventions: ΔR = post-intervention refusal rate minus original refusal rate (percentage points).

ModelMethodNon-evidence ΔREvidence ΔREvidence − non-evidence
Qwen3-VL-8BDirect−0.01+8.61+8.62
Qwen3-VL-8BVV−0.59+15.76+16.35
InternVL3.5-8BDirect+2.21+10.06+7.85
InternVL3.5-8BVV+0.27+11.46+11.19
Qwen2.5-VL-7BDirect+4.32+20.21+15.89
Qwen2.5-VL-7BVV−0.77+3.84+4.61

These results weaken the explanation that refusal is driven solely by a specific intervention artifact. We will include these controls in the revised manuscript and state the scope of the conclusions accordingly.

Table notes. Rates use the experiment's sampling weights; the three noise seeds are averaged within each question. Invalid outputs remain in the denominator and do not count as refusals. Differences and ranges use unrounded values and may differ by 0.01 from calculations using displayed values. Table 2 reports aggregate net changes, not per-question transition rates.

AuthorsOpenReview ↗

Text-only and few-frame baselines, sampling protocol, and manual validation

Thank you for raising the possibility of residual evidence in the Insufficient condition. To distinguish observing the evidence needed to answer from inferring an answer through contextual cues, we added three control conditions on the same 120-question subset: text-only input containing the question and options, and inputs additionally containing two or four frames from outside the annotated evidence intervals.

In all three conditions, models must select one candidate answer; refusal is not allowed. These controls therefore measure forced-choice answer accuracy, rather than the ability to independently judge evidence sufficiency or decide whether to refuse.

ModelText-only2 outside-interval frames4 outside-interval frames
Qwen2.5-VL-32B44.17%55.00%60.00%
InternVL3.5-8B41.67%53.33%59.17%

Both models show the same trend. Relative to text-only input, adding two and four outside-interval frames improves accuracy by 10.83 and 15.83 percentage points for Qwen2.5-VL-32B, and by 11.67 and 17.50 percentage points for InternVL3.5-8B. Because all conditions require an answer, these differences cannot be attributed to changes in refusal rates. At the same time, both models answer a substantial fraction of questions correctly without any visual input, demonstrating that a correct answer does not always depend on observing the key evidence in the video.

These results are consistent with the following explanation: even without observing the key evidence within the annotated intervals, a model forced to answer can still infer the correct option from question–option priors and contextual cues about scenes, people, or objects in a few frames. Consequently, a correct answer under the Insufficient condition does not, by itself, establish that the key evidence needed to support that answer remains available.

Original sampling protocol and additional manual validation. We retain the uniform 32-frame sampling setting used in the original NExT-GQA evaluation. During this rebuttal round, we additionally conducted manual validation of 200 examples, inspecting the sampled visual inputs to assess whether they provide sufficient evidence for answering the questions, including whether sufficient evidence remains after the intervention. This complements the text-only and few-frame controls: the controls quantify answer accuracy under restricted inputs, while manual inspection directly examines the visual support for those answers.

Together, these analyses reinforce the need to distinguish answer correctness from evidence sufficiency. A correct forced-choice answer alone does not establish that the available visual input sufficiently supports it. We therefore combine controlled baselines with input-level manual validation to examine both model behavior and the evidence available to the model.

AuthorsOpenReview ↗

Prompt disclosure and experimental comparability

Thank you for highlighting this issue. Appendix E.4 of the original manuscript describes the task requirements and execution protocol for direct inference, while Appendix H describes the stages of EF and VV. However, the complete prompt templates were originally planned for release with the code and were not fully presented in the manuscript. We recognize that this made it difficult to directly inspect prompt differences and their potential effects.

We will include the complete prompts and output formats actually used in the revised appendix, covering direct inference and every stage of EF and VV. We will also provide a mapping from experiments to prompt versions, input settings, decoding budgets, and retry rules. For cross-model direct inference, the original experiments use the common v2.2 task definition. We will explicitly distinguish these shared task requirements from model-specific implementation differences, including input packaging and structured-output support, rather than equating a common task definition with identical implementation details.

For comparisons between inference methods, the prompts and staged procedures are themselves part of the methods; the comparison therefore measures the effect of each complete pipeline. Together with the existing prompt-sensitivity analysis, we will identify which conclusions depend on particular prompt settings and delimit the corresponding attribution and generalization claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.