acceptodds
Under review as a conference paper at ICLR 2027

Better Scores Are Not Enough: Evidence-Dependence Evaluation for Video Hallucination

Abstract

Video hallucination is commonly associated with weak visual and temporal grounding, over-dependence on language priors, and response bias. Yet benchmark scores alone do not establish whether correct responses depend on video evidence. When a mitigation method improves a video hallucination benchmark score, does that gain actually depend on video evidence? We propose EVIDENCE-DEPENDENCE EVALUATION (EDE), an evaluation framework that uses controlled interventions on visual and temporal evidence to measure how benchmark performance and mitigation gains depend on that evidence, while preserving each benchmark’s original evaluation protocol. EDE combines controlled evidence interventions with complementary diagnostics that assess other evaluation factors relevant to interpreting benchmark scores and mitigation gains, requiring neither additional training nor new benchmark-specific annotations. EDE reveals markedly different evidence-dependence profiles: some benchmarks exhibit substantial performance declines when targeted video evidence is disrupted, whereas others remain stable or even increase after such evidence is removed or disrupted. We evaluate four video LLMs on six video hallucination benchmarks and analyze gains from six mitigation methods on selected benchmarks. Complementary diagnostics show that mitigation gains can coincide with error trade-offs, be partly reproduced by response shifts, and vary substantially across evaluation formats, while predictions can change under meaning-preserving question paraphrases. Together, EDE and these diagnostics reveal dependence on video evidence and sensitivity to other factors that benchmark scores and mitigation gains alone do not capture. We therefore argue that video hallucination evaluation should report benchmark scores and mitigation gains alongside their evidence-dependence profiles.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video hallucination, video LLMs, hallucination evaluation, evidence dependence, benchmark evaluation, hallucination mitigation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RVgiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether video-hallucination benchmark scores and mitigation gains depend on the video evidence required by the task. The authors propose the EDE diagnostic framework, which keeps the questions and scoring rules fixed while applying black-frame, text-only, single-frame repetition, and frame-shuffling interventions. It measures score changes and mitigation gains relative to the corresponding base model under matched conditions. The study covers seven tasks from six benchmarks, four video language models, additional model scales in the appendix, and six mitigation methods. Response-rate matching, paired scoring, multiple-choice conversion, and question paraphrasing show that aggregate scores can conceal limited evidence dependence, trade-offs between error types, and instability in individual predictions. The main contributions are reusable evaluation diagnostics and empirical findings across benchmarks.

Critical strengths
  1. The evidence-dependence diagnostics expose a practically important blind spot in evaluation. In Table 1, Qwen3 improves from 55.92% to 66.67% on MHBench with black frames, but drops from 87.19% to 50.00% on the VidHalluc action task. Appendix B.3 further identifies a video-free structural upper bound of 68.21% for MHBench and notes that its 2,400 entries correspond to only 200 distinct questions. Connecting intervention results to label structure explains why higher scores need not imply stronger visual grounding more effectively than input ablations alone. The experiments at additional model scales and the paired-metric validation in Appendix B.2 strengthen this observation.
  2. The comparison of mitigation gains is well targeted. Hound's preference-optimized model is compared with its corresponding supervised fine-tuned model, while the other methods are compared on the same base model, avoiding the direct inclusion of differences in base-model capability as mitigation gains. The authors explicitly identify MotionCD's zero gains under black frames and single-frame repetition as structural deactivation rather than independent evidence of visual dependence. The FAR/FRR decomposition in Table 3, response-rate matching in Appendix D, and semantic-unit matching in Table 4 show from different angles that gains may depend on response tendencies and scoring protocols. These analyses support reporting evidence-dependence profiles alongside conventional scores.
Critical weaknesses
  1. The adapted implementations do not yet support general conclusions about the original mitigation methods. Appendix A.3 changes TCD's original 16-to-4-frame setting to 8-to-4 frames, maps DTR's intervention layers by relative depth, and replaces VTI's original image-captioning examples with samples from the video benchmark pool. In Table 2, VTI drops by 59.90 percentage points on EventHallusion, and DTR drops by 28.83 points on the temporal task. Such extreme degradation may combine adaptation failures with limitations of the methods themselves. The intervention-strength sweeps in Appendix C.1 mainly cover MHBench and do not adequately establish evidence dependence within these methods' effective operating ranges. Appendix D.3 also reports differences of −0.04 and +0.21 percentage points for MotionCD and DTR, respectively, after the interventions are disabled, suggesting that the execution paths are not fully equivalent. The authors should verify consistency at zero intervention, establish the independence of VTI's examples from the evaluation data, and recheck the main conclusions using reasonable configurations selected on a separate validation set. This concern affects attribution to mitigation methods rather than invalidating the benchmark diagnostics themselves.
  2. Some metrics and controls do not adequately separate changes in evidence use from changes in evaluation conditions. Table 4 compares paired-question scores requiring both answers to be correct with scores for a single multiple-choice question. Matching semantic units does not remove differences in correctness criteria or difficulty. Appendix B.2 already shows that the direction of VidPair's changes under black frames and single-frame repetition depends on whether question accuracy or the native wAcc metric is used; the common accuracy metric in the main text therefore cannot substitute for conclusions under native benchmark metrics. In Table 16, VTI's response-rate matching error is 2.63 percentage points, and the offset grid does not cover the target response rate. Appendix D uses question-level or question-pair-level resampling without explaining how dependencies from shared videos and repeated questions are handled. Appendix F.1 uses the same model to generate and validate paraphrases; separate calls do not rule out shared semantic errors. The paper should highlight exceptions under native metrics, narrow its interpretation of protocol differences, and add controls covering the target response rate, paired intervals respecting the data's dependency structure, and a small blinded human semantic audit.
Questions for authors
  1. Are VTI's 50 direction-estimation examples independent of each evaluation set at the video, question, and answer levels? Please provide the key DTR and VTI results using configurations selected on a separate validation set. Which execution-path or numerical factors cause nonzero differences after interventions are disabled, and do the conclusions hold after those differences are eliminated?
  2. Please provide paired confidence intervals for the gains and intervention-induced changes in gains in Table 2 that materially affect the conclusions, accounting for shared videos and repeated questions. If the response-rate matching in Table 16 is extended to cover VTI's target rate, does its residual gain remain positive?
  3. Which evidence-dependence conclusions hold under native benchmark metrics, and which require exceptions? For Table 4, can you report the proportions of paired questions with exactly zero, one, and two correct answers, to distinguish sensitivity to answer format from the effects of requiring both answers to be correct? In a blinded audit of paraphrases, what proportion preserves both meaning and evidence requirements?
6 comments

We sincerely thank the reviewer for the careful and constructive review, and for recognizing the practical value of EDE and the benchmark-level diagnostics. The detailed comments prompted several additional validation experiments and helped us substantially strengthen the empirical analysis and clarify the scope of our claims. We address each point below.

Weakness 1 (& Question 1)

Our study focuses on assessing mitigation gains and their evidence dependence across diverse mitigation mechanisms, rather than establishing the optimal performance or relative ranking of individual methods. To this end, we adapted the image-based VCD and VTI methods for video hallucination benchmarks and incorporated them into our EDE analysis. We therefore conducted the following additional validation experiments and consistency checks to examine the effects of these adaptations and configurations.

The adapted implementations do not yet support general conclusions about the original mitigation methods. Appendix A.3 changes TCD's original 16-to-4-frame setting to 8-to-4 frames, and replaces VTI's original image-captioning examples with samples from the video benchmark pool.

Are VTI's 50 direction-estimation examples independent of each evaluation set at the video, question, and answer levels?

Adaptation and Evaluation Setup. In the original submission, VTI's visual direction was estimated using 50 videos from VideoHallucer, while its textual direction used 50 question pairs and their ground-truth labels from the same benchmark. Because these samples were used for direction estimation, we excluded VideoHallucer from the VTI evaluation. We additionally checked for overlap with the evaluation benchmarks using video identifiers, filenames, YouTube IDs, and exact or near-duplicate question matches, and found no overlaps under these criteria. Further details of the direction-estimation setup and overlap checks are provided in Appendix A.3.

The original TCD paper used different frame configurations depending on the backbone. In our study, we adopted an 8-to-4-frame configuration to ensure a fair comparison with other mitigation methods, which use eight-frame video inputs. We additionally evaluated the 16-to-4-frame configuration highlighted by the reviewer, using a frame-matched baseline for each setting. Across four evaluation tasks and three evidence interventions, the signs of intervention-induced gain changes (ΔG\Delta G) under the 16-to-4 setting were consistent with those under our original 8-to-4 setting in 11 of 12 comparisons (Rebuttal Table 2; see Appendix C.3, Table 16). The sole disagreement occurred under Frame Shuffle on VidHalluc-ACH, where the 16-to-4 setting yielded a ΔG\Delta G of +0.35 pp with a 95% CI of [−0.52, +1.23]. These results indicate that the main diagnostic trends were largely maintained across frame configurations.

BenchmarkFramesΔG\Delta G BlackΔG\Delta G SingleΔG\Delta G ShuffleSign Match
MHBench8→4−3.00−1.33−2.71—
16→4−2.96−0.58−2.333/3
EventHallusion8→4−1.71−3.18−0.49—
16→4−0.49−2.93−2.693/3
VidHalluc-TSH8→4−3.17−2.50−1.33—
16→4−2.00−4.17−1.173/3
VidHalluc-ACH8→4−0.27−0.53−0.71—
16→4−0.04−0.78+0.352/3
Total11/12

Rebuttal Table 2. TCD frame configuration sensitivity (pp). Each configuration uses a frame-matched baseline.

In Table 2, VTI drops by 59.90 percentage points on EventHallusion, and DTR drops by 28.83 points on the temporal task. Such extreme degradation may combine adaptation failures with limitations of the methods themselves. The intervention-strength sweeps in Appendix C.1 mainly cover MHBench and do not adequately establish evidence dependence within these methods' effective operating ranges.

maps DTR's intervention layers by relative depth

Please provide the key DTR and VTI results using configurations selected on a separate validation set.

Configuration Validation. The initial DTR configuration followed the original paper's strengths (α=0.5\alpha=0.5, β=0.4\beta=0.4), with its layer range mapped to Qwen3-VL layers 20–35 by relative depth, while VTI used the original paper's strengths (α=β=0.9\alpha=\beta=0.9) with a VideoHallucer-based textual direction. To examine the impact of configuration choices, we evaluated 45 DTR configurations spanning seven decoder-layer ranges, axis-wise strength sweeps (α\alpha, β\beta: 0.0–0.6, step 0.1), and additional scaled-strength settings. For VTI, we evaluated 100 strength combinations (10×1010 \times 10 grid) using the COCO-based textual direction from the original authors' implementation, while keeping the visual direction unchanged. Both methods were evaluated on a separate Dr.V-Bench validation set of 400 questions. For DTR, we first evaluated seven decoder-layer ranges and identified 18–31 and 28–35 as promising alternatives. We then performed axis-wise strength sweeps over these two ranges and the initial 20–35 range. Across all 45 configurations, the highest validation accuracy was 70.25%, achieved at layers 20–35 with halved strengths (α=0.25\alpha=0.25, β=0.2\beta=0.2). Based on validation accuracy, the selected DTR configuration retained layers 20–35 while halving both strengths, whereas VTI adopted the COCO-based textual direction and validation-selected strengths (Rebuttal Table 3; see Appendix C.4, Table 17).

MethodSettingLayers / Directionα\alphaβ\betaVal. Acc.
DTRInitial20–350.50.468.50%
Selected20–350.250.270.25%
VTIInitialVideoHallucer0.90.965.25%
Direction controlCOCO0.90.966.50%
SelectedCOCO0.20.772.75%

Rebuttal Table 3. Configuration validation for DTR and VTI. Validation accuracy (%) on 400 Dr.V-Bench questions.

Validation-selected configurations substantially attenuated the initial negative gains on VidHalluc-TSH for DTR (−29.00 to −1.00 pp) and EventHallusion for VTI (−59.90 to −8.56 pp) (Rebuttal Table 4). Interestingly, however, consistent with our original findings, improvements in original-condition gains did not necessarily translate into improvements under evidence interventions, and intervention-induced gain changes (ΔG\Delta G) exhibited different patterns across configurations. While changes in original-condition performance reflect the configuration sensitivity of the mitigation methods themselves, EDE provides complementary diagnostic information by revealing their sensitivity to evidence interventions beyond what original-condition gains alone can capture (Appendix C.4, Table 18).

MethodBenchmarkInitial Gain (pp)Selected Gain (pp)
DTRMHBench−8.08−4.92
EventHallusion+4.16+2.69
VidHalluc-TSH−29.00−1.00
VidHalluc-ACH−1.68−0.44
VTIMHBench+3.50−3.08
EventHallusion−59.90−8.56
VidHalluc-TSH−21.50−5.17
VidHalluc-ACH−18.33+0.64

Rebuttal Table 4. Original-condition gains under initial and validation-selected configurations (pp).

Appendix D.3 also reports differences of −0.04 and +0.21 percentage points for MotionCD and DTR, respectively, after the interventions are disabled, suggesting that the execution paths are not fully equivalent. Which execution-path or numerical factors cause nonzero differences after interventions are disabled, and do the conclusions hold after those differences are eliminated?

Zero-intervention and Baseline Consistency. In response to the reviewer's zero-intervention concern, we audited the execution paths and baselines of all mitigation methods and mechanistic controls, beyond VTI and DTR. Some nonzero gains observed under zero intervention arose from mismatched execution paths (SDPA forward/generate and eager attention), rather than the interventions themselves. With method-matched baselines, MotionCD and DTR achieved exactly zero gains under zero intervention (Appendix D.3, Table 24), and the resulting corrections were limited. On MHBench, the same base model differed on only 1 of 2,400 predictions across the compared execution paths, and gain corrections in Table 2 were at most 1.22 pp. Only one intervention-induced gain change (ΔG\Delta G) reversed sign—DTR under Black Frames on EventHallusion (−0.98 to +0.24 pp)—with its 95% CI including zero. Thus, the directions of the main original-condition gains and key diagnostic findings remained unchanged. We have incorporated these corrections into Table 2 and Appendix A.3, and plan to release the implementation, evaluation code, and documentation for reproducibility.

Reviewer_XFVeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the EDE. This evaluation framework to examine whether the hallucination mitigation methods rely on the video content itself, by disrupting the video (e.g., black or shuffle frames). Experiments on six benchmarks and mitigation methods show that some number gains do not rely on video itself.

Critical strengths
  1. Under the paper’s own implementations, none of the six mitigation methods improves consistently across benchmarks and evaluation formats, showing that their reported gains alone do not prove a real reduction of hallucination.
Critical weaknesses
  1. The technical novelty is very limited. Similar video disruptions are already used in prior work, e.g., reversed, downsampled, or noised videos as negatives in contrastive decoding approaches (Section 2.2), and single-frame or no-video checks in prior evaluation studies (Section 2.1).
  2. The core idea of EDE, comparing the model’s outputs on the original and disrupted videos, follows the same idea as contrastive decoding. The paper does not discuss this relation or what EDE adds beyond it.
Questions for authors

Please to my weakness part.

2 comments

We thank the reviewer for raising the question of technical novelty and for the opportunity to clarify the distinction between EDE and prior input-perturbation or contrastive-decoding methods. We address these concerns below.

Weakness 1

The technical novelty is very limited. Similar video disruptions are already used in prior work, e.g., reversed, downsampled, or noised videos as negatives in contrastive decoding approaches (Section 2.2), and single-frame or no-video checks in prior evaluation studies (Section 2.1).

We agree with the reviewer that similar video interventions have been used in prior work. The reverse, downsample, and noise perturbations used in contrastive decoding, as well as the single-frame and no-video analyses in prior evaluation studies, are also discussed in Sections 2.1–2.2.

Nevertheless, EDE's main contribution lies not in the technical novelty of individual interventions, but in providing an evaluation framework that applies evidence-dependence diagnostics to both benchmark performance and hallucination mitigation gains while preserving the questions and evaluation protocols of existing video hallucination benchmarks. In particular, EDE not only analyzes condition-matched gains by comparing the base model and the hallucination mitigation method under the same evidence condition, but also incorporates complementary diagnostics beyond intervention-based analyses, including FAR/FRR, response-rate matching, and evaluation-protocol sensitivity, to examine evaluation blind spots that conventional scores alone may not reveal.

The other two reviewers (pYrc and RVgi) also recognized this evaluation gap and diagnostic utility as strengths of our work. For example, as shown in Table 2, MotionCD's mitigation gain on VidHalluc-TSH reversed from +7.33 pp under the original condition to −1.83 pp under frame shuffling, whereas Hound-DPO's gain was largely maintained, changing from +12.67 pp to +10.00 pp. This illustrates differences in intervention sensitivity across mitigation methods that are difficult to distinguish from their original-condition gains alone.

Weakness 2

The core idea of EDE, comparing the model’s outputs on the original and disrupted videos, follows the same idea as contrastive decoding. The paper does not discuss this relation or what EDE adds beyond it.

Although both approaches use original and perturbed videos, they differ fundamentally in their objectives and computational procedures. Contrastive decoding is a hallucination mitigation method that adjusts predictions during generation by comparing the token-level prediction distributions under the two inputs. In contrast, EDE is an evaluation framework that measures changes in benchmark performance and condition-matched mitigation gains across evidence conditions constructed through controlled evidence interventions. Thus, while the former produces adjusted predictions, the latter provides evidence-dependence profiles and diagnostic results.

Indeed, our study includes contrastive decoding-based methods, such as MotionCD, TCD, and VCD, as evaluation targets of EDE. Even when accuracy improves under the original condition, whether the corresponding gain is maintained after disrupting video evidence requires separate evaluation, and EDE diagnoses whether such gains are maintained, reduced, or reversed. This relationship is also reflected in the original paper's related work (Section 2.2), definition of EDE (Sections 3.1–3.3), and experimental results (Section 4.3, Table 2). We appreciate the reviewer’s suggestion to further clarify the distinction between the two approaches. To clarify this distinction, we have added a sentence to Section 2.2 of the revised paper directly explaining their different objectives and roles.

Dear Reviewer XFVe,

Thank you for taking the time to review our paper and provide your feedback.

Your comments provided an opportunity for us to clarify the distinction between EDE and prior approaches. In our rebuttal, we explained how contrastive decoding adjusts predictions during inference, whereas EDE evaluates benchmark performance and condition-matched mitigation gains across evidence conditions.

In particular, we emphasize that EDE is not a new hallucination mitigation method, but an evaluation framework for systematically examining benchmark performance and mitigation gains.

We would appreciate your feedback on whether these clarifications have addressed your main concern, or if any further explanation would be helpful.

Best regards,
The Authors

Reviewer_pYrcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that standard video hallucination benchmark scores and mitigation gains do not by themselves establish whether model predictions actually depend on the visual or temporal evidence that a benchmark is intended to evaluate. To address this gap, the authors propose Evidence-Dependence Evaluation (EDE), which applies controlled interventions to visual and temporal inputs and measures how benchmark performance and mitigation gains change while preserving the original evaluation protocol. The paper further studies response tendency, evaluation-protocol sensitivity, and robustness to meaning-preserving question paraphrases. Across multiple benchmarks, models, and mitigation methods, the authors identify substantially different evidence-dependence profiles, including cases where performance or mitigation gains remain stable or even improve after target evidence is disrupted.

Critical strengths
  • The paper identifies an important evaluation gap: a correct answer does not necessarily imply that a model relied on the visual or temporal evidence the benchmark was designed to probe. The black-frame experiments provide a particularly clear illustration of this issue.

  • The proposed method is well aligned with the stated motivation. Rather than introducing another aggregate hallucination score, EDE directly intervenes on the evidence whose role the benchmark is intended to measure.

  • The complementary analyses on response tendency, evaluation format, and paraphrasing are useful because they show that measured mitigation gains can vary for reasons other than improved use of video evidence.

Critical weaknesses
  • My main concern is that the intervention effects may partially conflate removal of the target evidence with distribution shift introduced by the intervention itself. For example, replacing a video with black frames removes almost all visual information and also produces a highly atypical input. This does not invalidate EDE as a diagnostic framework, but it limits stronger interpretations of the resulting profile as direct evidence of causal grounding.

  • The semantics of the interventions differ substantially across tasks. Frame shuffling is relatively targeted when evaluating temporal ordering, while black-frame replacement affects many forms of visual evidence at once. It would be useful to clarify how comparable the resulting evidence-dependence scores are across intervention types.

  • The paper shows that benchmark scores and mitigation gains may be misleading, but it does not yet provide a principled criterion for what constitutes sufficient evidence dependence or how multiple intervention results should be summarized into a single evaluation decision.

  • The paraphrase analysis demonstrates prediction instability but does not isolate its source. Such instability may reflect language sensitivity, response bias, benchmark ambiguity, or more general reasoning instability.

Questions for authors

Overall, I find the empirical findings useful, but I would prefer the paper to more explicitly distinguish intervention sensitivity from stronger claims about causal evidence use.

3 comments

We appreciate the reviewer’s careful and constructive feedback, as well as their recognition of EDE’s motivation and empirical value. We address the concerns below.

Weakness 1

My main concern is that the intervention effects may partially conflate removal of the target evidence with distribution shift introduced by the intervention itself. For example, replacing a video with black frames removes almost all visual information and also produces a highly atypical input. This does not invalidate EDE as a diagnostic framework, but it limits stronger interpretations of the resulting profile as direct evidence of causal grounding.

We agree with the reviewer that controlled interventions may alter the input distribution in addition to removing or disrupting the target evidence. Accordingly, we do not interpret an individual intervention effect as a direct causal estimate of target-evidence use. Instead, EDE characterizes how benchmark scores and mitigation gains respond to specific evidence interventions. We have revised the paper in Section 3.3 to define evidence-dependence profiles as intervention-specific sensitivity profiles and to clarify the scope of their interpretation.

Nevertheless, our central benchmark-level conclusion remains: if benchmark performance remains unchanged or improves after the target visual or temporal evidence is removed or disrupted, the original score alone cannot establish that model responses depended on that evidence. We therefore interpret EDE intervention results as diagnostics of whether benchmark performance is sufficient to support claims of visual or temporal grounding, rather than as direct causal evidence that a model does or does not use a particular source of evidence.

To address the reviewer’s Black Frames example, we designed a targeted control that removes question-relevant visual evidence without replacing the entire video with an atypical input. Specifically, Target Evidence Masking masks only the bounding-box regions of objects relevant to each question while preserving the remainder of the video. We sampled 50 video–question pairs from each of MHBench and VidHalluc-ACH and evaluated Qwen3-VL-4B/8B on the retained samples. As shown in Rebuttal Table 1, all four model–benchmark combinations reproduced the direction observed under Black Frames: accuracy increased on MHBench and decreased on VidHalluc-ACH. Although localized masking does not eliminate all intervention-induced distribution shift, this controlled comparison shows that the opposing benchmark responses are not specific to the all-black input and persist when the perturbation is restricted to question-relevant visual regions. Because this control requires object-level annotations, we present it as a targeted validation, while the core EDE protocol remains directly applicable without additional training or benchmark-specific annotations. Further experimental details are provided in Appendix G.

BenchmarkVideo LLMNΔ\Delta Black FramesΔ\Delta Target Evidence Masking
MHBenchQwen3-VL-4B49+12.2+6.1
MHBenchQwen3-VL-8B49+20.4+4.1
VidHalluc-ACHQwen3-VL-4B43−44.2−20.9
VidHalluc-ACHQwen3-VL-8B43−51.2−25.6

Rebuttal Table 1. Accuracy changes under Black Frames and Target Evidence Masking (pp). N denotes samples retained after bounding-box validation from 50 sampled pairs per benchmark.

Weakness 2

The semantics of the interventions differ substantially across tasks. Frame shuffling is relatively targeted when evaluating temporal ordering, while black-frame replacement affects many forms of visual evidence at once. It would be useful to clarify how comparable the resulting evidence-dependence scores are across intervention types.

We agree with the reviewer that different interventions disrupt different types and amounts of information. Accordingly, changes in accuracy (ΔAccuracy\Delta \mathrm{Accuracy}) across interventions cannot be directly compared as equally calibrated measures of evidence dependence. We have therefore revised Section 3.3 to clarify the comparability of intervention-specific results and the scope of interpreting evidence-dependence profiles.

EDE does not aim to compare changes in benchmark performance across different interventions on a common scale. Instead, EDE characterizes sensitivity to each intervention separately and examines whether the resulting benchmark responses align with the target evidence each benchmark is designed to probe. Specifically, we distinguish the types of information disrupted by each intervention in Section 3.2 and map each benchmark’s target evidence to the relevant interventions based on its original task design in Table 6.

Weakness 3

The paper shows that benchmark scores and mitigation gains may be misleading, but it does not yet provide a principled criterion for what constitutes sufficient evidence dependence or how multiple intervention results should be summarized into a single evaluation decision.

As explained in Weakness 2, EDE does not aim to aggregate results from different interventions into a single score or to provide a universal threshold for determining sufficient evidence dependence. Because each intervention disrupts different types and amounts of target evidence, reducing these results to a single score could obscure important intervention-specific differences.

We appreciate the reviewer’s suggestion to further clarify how results from multiple interventions should be interpreted together. Accordingly, we have clarified the principles for interpreting evidence-dependence profiles in Section 3.3 of the revised paper.

EDE systematically diagnoses benchmark performance and mitigation gains by relating predefined target evidence to the responses observed under each intervention. Specifically, for benchmark performance, EDE examines alignment and mismatch between target evidence requirements and observed responses, while for mitigation gains, it analyzes whether improvements are maintained, reduced, or reversed under different intervention conditions. In this way, EDE provides evidence-dependence profiles that retain intervention-specific diagnostic results and identifies potential mismatches that warrant further investigation.

Weakness 4

The paraphrase analysis demonstrates prediction instability but does not isolate its source. Such instability may reflect language sensitivity, response bias, benchmark ambiguity, or more general reasoning instability.

In addition to the automated validation described in Appendix F.1, we conducted an independent, blinded human semantic audit, reported in Appendix F.3 (Table 32), to assess whether the paraphrases preserve their original meaning and evidence requirements. Four annotators evaluated 210 original–paraphrase pairs sampled from the seven evaluation tasks presented in Table 5. For 98.6% (207/210) of the pairs, at least three annotators agreed that both criteria were preserved, while 89.5% (188/210) received unanimous agreement. These results support the reliability of our paraphrase analysis.

While our paraphrase analysis does not isolate the individual causes of prediction instability, it examines how model predictions change when only the question wording is modified while preserving the video, ground-truth answer, and required evidence. Even when average accuracy changes only slightly, individual predictions can shift between correct and incorrect answers, revealing instability that aggregate accuracy alone may not capture.

We agree that distinguishing among the potential causes suggested by the reviewer, including response bias, benchmark ambiguity, and reasoning instability, is a meaningful direction for future research. As a first step, our study focuses on the more fundamental diagnostic question of whether prediction instability occurs under meaning- and evidence-preserving paraphrases and whether conventional evaluation scores can obscure such instability.

Dear Reviewer pYrc,

We sincerely appreciate your recognition of the important evaluation gap addressed by our work and your positive assessment of the complementary analyses on response tendency, evaluation format, and paraphrasing.

In response to your main concerns, we conducted additional Target Evidence Masking experiments, which showed that the contrasting benchmark trends observed under Black Frames persisted under localized Target Evidence Masking. We also clarified the scope and limitations of EDE's interpretation to avoid overstating claims about causal evidence use.

We would greatly appreciate your feedback on whether these additional experiments and revisions have adequately addressed your main concerns, or if any issues remain unresolved.

Best regards,
The Authors

Comments on the paper

1 comment

We thank all reviewers for their careful evaluation and valuable feedback on our paper.

We highlight the following strengths and contributions of our work, building on the reviewers' positive feedback:

  • An important evaluation gap: We show that high benchmark scores and mitigation gains do not necessarily establish dependence on the target video evidence.
  • A systematic evaluation framework: EDE systematically diagnoses both benchmark performance and condition-matched mitigation gains under controlled evidence interventions.
  • Broad empirical analysis: Across diverse benchmarks, Video LLMs, mitigation methods, and interventions, EDE reveals heterogeneous responses that original-condition scores alone cannot capture.
  • Practicality and applicability to existing benchmarks: EDE requires neither additional training nor benchmark-specific annotations and incorporates complementary evaluation diagnostics.

Our responses can be summarized as follows:

  • Diagnostic validation and interpretation: We clarified both EDE's intervention-specific interpretation and the distinction between intervention sensitivity and causal evidence use, and further examined intervention-induced distribution shift through Target Evidence Masking experiments, which show consistent qualitative patterns under localized perturbations. (Reviewer pYrc: W1–W3; revised Sec. 3.3 and Appendix G)
  • Contribution and novelty: We clarified EDE's distinction from prior perturbation-based evaluations and contrastive decoding, particularly in its role as an evaluation framework for benchmark scores and condition-matched mitigation gains. (Reviewer XFVe: W1–W2; revised Sec. 2.2)
  • Extensive experimental validation: We conducted configuration analyses, zero-intervention consistency checks, cluster-aware paired confidence intervals, response-rate matching, native-metric and protocol analyses, and an independent blinded human paraphrase audit. (Reviewers RVgi: W1–W2 and pYrc: W4; Revised Appendix B.2, C.3–C.5, D.1–D.3, E.1, F.3.)

We have also uploaded a revised manuscript incorporating these clarifications and additional analyses. All revisions are highlighted in blue text for ease of reference, with further details provided in the corresponding sections and appendices.

Please refer to our detailed responses below. We hope they address the reviewers' concerns, and we welcome any additional comments, questions, or requests for clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.