- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks whether video-hallucination benchmark scores and mitigation gains depend on the video evidence required by the task. The authors propose the EDE diagnostic framework, which keeps the questions and scoring rules fixed while applying black-frame, text-only, single-frame repetition, and frame-shuffling interventions. It measures score changes and mitigation gains relative to the corresponding base model under matched conditions. The study covers seven tasks from six benchmarks, four video language models, additional model scales in the appendix, and six mitigation methods. Response-rate matching, paired scoring, multiple-choice conversion, and question paraphrasing show that aggregate scores can conceal limited evidence dependence, trade-offs between error types, and instability in individual predictions. The main contributions are reusable evaluation diagnostics and empirical findings across benchmarks.
Critical strengths
- The evidence-dependence diagnostics expose a practically important blind spot in evaluation. In Table 1, Qwen3 improves from 55.92% to 66.67% on MHBench with black frames, but drops from 87.19% to 50.00% on the VidHalluc action task. Appendix B.3 further identifies a video-free structural upper bound of 68.21% for MHBench and notes that its 2,400 entries correspond to only 200 distinct questions. Connecting intervention results to label structure explains why higher scores need not imply stronger visual grounding more effectively than input ablations alone. The experiments at additional model scales and the paired-metric validation in Appendix B.2 strengthen this observation.
- The comparison of mitigation gains is well targeted. Hound's preference-optimized model is compared with its corresponding supervised fine-tuned model, while the other methods are compared on the same base model, avoiding the direct inclusion of differences in base-model capability as mitigation gains. The authors explicitly identify MotionCD's zero gains under black frames and single-frame repetition as structural deactivation rather than independent evidence of visual dependence. The FAR/FRR decomposition in Table 3, response-rate matching in Appendix D, and semantic-unit matching in Table 4 show from different angles that gains may depend on response tendencies and scoring protocols. These analyses support reporting evidence-dependence profiles alongside conventional scores.
Critical weaknesses
- The adapted implementations do not yet support general conclusions about the original mitigation methods. Appendix A.3 changes TCD's original 16-to-4-frame setting to 8-to-4 frames, maps DTR's intervention layers by relative depth, and replaces VTI's original image-captioning examples with samples from the video benchmark pool. In Table 2, VTI drops by 59.90 percentage points on EventHallusion, and DTR drops by 28.83 points on the temporal task. Such extreme degradation may combine adaptation failures with limitations of the methods themselves. The intervention-strength sweeps in Appendix C.1 mainly cover MHBench and do not adequately establish evidence dependence within these methods' effective operating ranges. Appendix D.3 also reports differences of −0.04 and +0.21 percentage points for MotionCD and DTR, respectively, after the interventions are disabled, suggesting that the execution paths are not fully equivalent. The authors should verify consistency at zero intervention, establish the independence of VTI's examples from the evaluation data, and recheck the main conclusions using reasonable configurations selected on a separate validation set. This concern affects attribution to mitigation methods rather than invalidating the benchmark diagnostics themselves.
- Some metrics and controls do not adequately separate changes in evidence use from changes in evaluation conditions. Table 4 compares paired-question scores requiring both answers to be correct with scores for a single multiple-choice question. Matching semantic units does not remove differences in correctness criteria or difficulty. Appendix B.2 already shows that the direction of VidPair's changes under black frames and single-frame repetition depends on whether question accuracy or the native wAcc metric is used; the common accuracy metric in the main text therefore cannot substitute for conclusions under native benchmark metrics. In Table 16, VTI's response-rate matching error is 2.63 percentage points, and the offset grid does not cover the target response rate. Appendix D uses question-level or question-pair-level resampling without explaining how dependencies from shared videos and repeated questions are handled. Appendix F.1 uses the same model to generate and validate paraphrases; separate calls do not rule out shared semantic errors. The paper should highlight exceptions under native metrics, narrow its interpretation of protocol differences, and add controls covering the target response rate, paired intervals respecting the data's dependency structure, and a small blinded human semantic audit.
Questions for authors
- Are VTI's 50 direction-estimation examples independent of each evaluation set at the video, question, and answer levels? Please provide the key DTR and VTI results using configurations selected on a separate validation set. Which execution-path or numerical factors cause nonzero differences after interventions are disabled, and do the conclusions hold after those differences are eliminated?
- Please provide paired confidence intervals for the gains and intervention-induced changes in gains in Table 2 that materially affect the conclusions, accounting for shared videos and repeated questions. If the response-rate matching in Table 16 is extended to cover VTI's target rate, does its residual gain remain positive?
- Which evidence-dependence conclusions hold under native benchmark metrics, and which require exceptions? For Table 4, can you report the proportions of paired questions with exactly zero, one, and two correct answers, to distinguish sensitivity to answer format from the effects of requiring both answers to be correct? In a blinded audit of paraphrases, what proportion preserves both meaning and evidence requirements?
6 comments
In Table 2, VTI drops by 59.90 percentage points on EventHallusion, and DTR drops by 28.83 points on the temporal task. Such extreme degradation may combine adaptation failures with limitations of the methods themselves. The intervention-strength sweeps in Appendix C.1 mainly cover MHBench and do not adequately establish evidence dependence within these methods' effective operating ranges.
maps DTR's intervention layers by relative depth
Please provide the key DTR and VTI results using configurations selected on a separate validation set.
Configuration Validation. The initial DTR configuration followed the original paper's strengths (, ), with its layer range mapped to Qwen3-VL layers 20–35 by relative depth, while VTI used the original paper's strengths () with a VideoHallucer-based textual direction. To examine the impact of configuration choices, we evaluated 45 DTR configurations spanning seven decoder-layer ranges, axis-wise strength sweeps (, : 0.0–0.6, step 0.1), and additional scaled-strength settings. For VTI, we evaluated 100 strength combinations ( grid) using the COCO-based textual direction from the original authors' implementation, while keeping the visual direction unchanged. Both methods were evaluated on a separate Dr.V-Bench validation set of 400 questions. For DTR, we first evaluated seven decoder-layer ranges and identified 18–31 and 28–35 as promising alternatives. We then performed axis-wise strength sweeps over these two ranges and the initial 20–35 range. Across all 45 configurations, the highest validation accuracy was 70.25%, achieved at layers 20–35 with halved strengths (, ). Based on validation accuracy, the selected DTR configuration retained layers 20–35 while halving both strengths, whereas VTI adopted the COCO-based textual direction and validation-selected strengths (Rebuttal Table 3; see Appendix C.4, Table 17).
| Method | Setting | Layers / Direction | Val. Acc. | ||
|---|---|---|---|---|---|
| DTR | Initial | 20–35 | 0.5 | 0.4 | 68.50% |
| Selected | 20–35 | 0.25 | 0.2 | 70.25% | |
| VTI | Initial | VideoHallucer | 0.9 | 0.9 | 65.25% |
| Direction control | COCO | 0.9 | 0.9 | 66.50% | |
| Selected | COCO | 0.2 | 0.7 | 72.75% |
Rebuttal Table 3. Configuration validation for DTR and VTI. Validation accuracy (%) on 400 Dr.V-Bench questions.
Validation-selected configurations substantially attenuated the initial negative gains on VidHalluc-TSH for DTR (−29.00 to −1.00 pp) and EventHallusion for VTI (−59.90 to −8.56 pp) (Rebuttal Table 4). Interestingly, however, consistent with our original findings, improvements in original-condition gains did not necessarily translate into improvements under evidence interventions, and intervention-induced gain changes () exhibited different patterns across configurations. While changes in original-condition performance reflect the configuration sensitivity of the mitigation methods themselves, EDE provides complementary diagnostic information by revealing their sensitivity to evidence interventions beyond what original-condition gains alone can capture (Appendix C.4, Table 18).
| Method | Benchmark | Initial Gain (pp) | Selected Gain (pp) |
|---|---|---|---|
| DTR | MHBench | −8.08 | −4.92 |
| EventHallusion | +4.16 | +2.69 | |
| VidHalluc-TSH | −29.00 | −1.00 | |
| VidHalluc-ACH | −1.68 | −0.44 | |
| VTI | MHBench | +3.50 | −3.08 |
| EventHallusion | −59.90 | −8.56 | |
| VidHalluc-TSH | −21.50 | −5.17 | |
| VidHalluc-ACH | −18.33 | +0.64 |
Rebuttal Table 4. Original-condition gains under initial and validation-selected configurations (pp).
Appendix D.3 also reports differences of −0.04 and +0.21 percentage points for MotionCD and DTR, respectively, after the interventions are disabled, suggesting that the execution paths are not fully equivalent. Which execution-path or numerical factors cause nonzero differences after interventions are disabled, and do the conclusions hold after those differences are eliminated?
Zero-intervention and Baseline Consistency. In response to the reviewer's zero-intervention concern, we audited the execution paths and baselines of all mitigation methods and mechanistic controls, beyond VTI and DTR. Some nonzero gains observed under zero intervention arose from mismatched execution paths (SDPA forward/generate and eager attention), rather than the interventions themselves. With method-matched baselines, MotionCD and DTR achieved exactly zero gains under zero intervention (Appendix D.3, Table 24), and the resulting corrections were limited. On MHBench, the same base model differed on only 1 of 2,400 predictions across the compared execution paths, and gain corrections in Table 2 were at most 1.22 pp. Only one intervention-induced gain change () reversed sign—DTR under Black Frames on EventHallusion (−0.98 to +0.24 pp)—with its 95% CI including zero. Thus, the directions of the main original-condition gains and key diagnostic findings remained unchanged. We have incorporated these corrections into Table 2 and Appendix A.3, and plan to release the implementation, evaluation code, and documentation for reproducibility.
We sincerely thank the reviewer for the careful and constructive review, and for recognizing the practical value of EDE and the benchmark-level diagnostics. The detailed comments prompted several additional validation experiments and helped us substantially strengthen the empirical analysis and clarify the scope of our claims. We address each point below.
Weakness 1 (& Question 1)
Our study focuses on assessing mitigation gains and their evidence dependence across diverse mitigation mechanisms, rather than establishing the optimal performance or relative ranking of individual methods. To this end, we adapted the image-based VCD and VTI methods for video hallucination benchmarks and incorporated them into our EDE analysis. We therefore conducted the following additional validation experiments and consistency checks to examine the effects of these adaptations and configurations.
Adaptation and Evaluation Setup. In the original submission, VTI's visual direction was estimated using 50 videos from VideoHallucer, while its textual direction used 50 question pairs and their ground-truth labels from the same benchmark. Because these samples were used for direction estimation, we excluded VideoHallucer from the VTI evaluation. We additionally checked for overlap with the evaluation benchmarks using video identifiers, filenames, YouTube IDs, and exact or near-duplicate question matches, and found no overlaps under these criteria. Further details of the direction-estimation setup and overlap checks are provided in Appendix A.3.
The original TCD paper used different frame configurations depending on the backbone. In our study, we adopted an 8-to-4-frame configuration to ensure a fair comparison with other mitigation methods, which use eight-frame video inputs. We additionally evaluated the 16-to-4-frame configuration highlighted by the reviewer, using a frame-matched baseline for each setting. Across four evaluation tasks and three evidence interventions, the signs of intervention-induced gain changes (ΔG) under the 16-to-4 setting were consistent with those under our original 8-to-4 setting in 11 of 12 comparisons (Rebuttal Table 2; see Appendix C.3, Table 16). The sole disagreement occurred under Frame Shuffle on VidHalluc-ACH, where the 16-to-4 setting yielded a ΔG of +0.35 pp with a 95% CI of [−0.52, +1.23]. These results indicate that the main diagnostic trends were largely maintained across frame configurations.
Rebuttal Table 2. TCD frame configuration sensitivity (pp). Each configuration uses a frame-matched baseline.