- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies two kinds of scene-transition errors in Video LLMs: reporting a transition when there is none (hallucination) and missing a real one (omission). Using last-layer attention interventions, the authors argue that over-attending to non-boundary semantic shifts causes hallucination, while under-attending to true boundaries causes omission. Based on this, they propose SSACD, a training-free method: semantic shifts between adjacent frames are detected with CLIP, each shift is labeled OVER or UNDER from its attention ratio in a preliminary decoding pass, noise is added to the corresponding frames to build a "failure-amplified" branch, and the two branches are combined with VCD-style contrastive decoding. With Qwen2.5-VL-7B and LLaVA-OV-7B at 8 uniformly sampled frames, SSACD outperforms DINO-HEAL, TCD and VCD on VidHalluc, the scene-transition subsets of MVBench/TVBench, and on LVBench and MLVU.
Critical strengths
- The problem is decomposed clearly, and the story from diagnosis to method is coherent. Scene-transition errors are split into hallucination and omission, which map to two perturbation modes, and the ablation supports this split (Table 3: removing Context Perturbation raises hallucination from 22.2 to 31.0, and removing Shift Perturbation raises omission from 35.8 to 42.1).
- The gains on VidHalluc are clear and backed by a key control: in Table 3, perturbing all frames uniformly gives 46.6, while shift-aware perturbation reaches 50.7, showing the gain is not just from VCD-style global perturbation. The method is training-free, and the authors report the inference overhead honestly (Table 4).
Critical weaknesses
W1. Effectiveness on long videos has not been demonstrated. The abstract (L025) and §5.2 (p.7, L377 onward) present long-video improvement as a main result, yet all experiments use only 8 frames (§5.1, p.7, L361–362). Following TCD's 8-frame setting is reasonable for the scene-transition benchmarks, but LVBench videos are hour-long, and at 8 frames adjacent samples are minutes apart, so it is hard to say the model still "sees" scene transitions. The base models score only 23.3 (Qwen) and 25.4 (LLaVA) on LVBench, close to the chance level of four-way multiple choice; SSACD's gains there (+9.5 and +6.6) are the largest in the table, larger even than its margins over the strongest baseline on the scene-transition tasks, while VCD and TCD gain only about 1 point. From such a near-chance starting point, these gains are hard to attribute to better discrimination of scene transitions.
W2. The inference-time rule is inconsistent with the diagnostic analysis. The diagnostic experiments separate transition and non-transition samples using ground truth (§3, p.4, L190–191) and conclude that over-attending to non-boundaries and under-attending to true boundaries cause the two error types. At inference, however, a shift is labeled only by whether its attention ratio exceeds γ (§4.2, p.6, L270–274), regardless of whether it is a true boundary. A true boundary that is correctly attended to would then be labeled OVER and suppressed by the contrast, which could itself cause omission. The paper does not show how this labeling corresponds to true boundaries.
W3. The "causally contributes" claim is too strong. In natural conditions, the attention ratio of correct and incorrect samples differs by only about 0.02 around 0.25 (Fig. 1, p.2), whereas in Fig. 2 (p.4) error rates rise noticeably mostly above 0.4, and above 0.7 for LLaVA on TVBench/MVBench. On VidHalluc, the random-region control also shows a sizable effect, and LLaVA's Suppress curve nearly overlaps the random control. Interventions far outside the natural range show that forcing attention this way can cause errors, but not that naturally occurring errors stem from it.
W4. Missing hyperparameters and statistical uncertainty. The default values of α, β, σ, γ and λ are not given anywhere (§5.1, p.7, L363–364 points to App. D, which contains only a sensitivity plot); the sensitivity analysis (Figs. 5–6) is done on VidHalluc, which is also the main test set; and results "averaged over three seeds" come without standard deviations. Given that the margins over the stronges
Questions for authors
- Could you report base and SSACD results on LVBench and MLVU with 32/64 frames, along with answer-parsing failure rates? If not, would you consider softening the long-video claims?
- On a subset with ground truth, could you show the confusion matrix between OVER/UNDER labels and true boundaries, break SSACD's changes into wrong→right and right→wrong, and give an oracle upper bound where the perturbation mode is chosen from ground truth?
- Could you repeat the attention interventions within the natural range (e.g., 0.2–0.35), and report the sample sizes and variance for Fig. 1?
- Could you list all default hyperparameters, state which data were used for tuning, and report the standard deviation over the three seeds and the configurations of the baselines?
- On more recent Video LLMs, are scene-transition hallucination and omission still significant, and how much of SSACD's gain remains?
6 comments
B. [Question 2] Boundary-Grounded Labels, Prediction Changes, and Oracle References
On a subset with ground truth, could you show the confusion matrix between OVER/UNDER labels and true boundaries, break SSACD's changes into wrong→right and right→wrong, and give an oracle upper bound where the perturbation mode is chosen from ground truth?
We conducted a controlled study on a balanced 74-video VidHalluc STH subset (37 transition and 37 non-transition videos), using eight frames and three seeds (1, 2, 3). The supplementary implementation differs from that used for the submitted Table 1; its scores are not directly substituted for those results.
1. Ground-Truth Scope and Boundary Matching
Transition timestamps were reconstructed from video assembly splices and screened through assistant-assisted visual inspection; negative labels came from the benchmark. These are neither independent human annotations nor exhaustive boundary labels.
A candidate matches when its sampled-frame interval, extended by 0.25 seconds at each endpoint, contains an annotated transition. This measures interval containment, not boundary localization within ±0.25 seconds. Since positive transitions are concentrated near video midpoints, this is a controlled evaluation rather than a general boundary-detection benchmark.
2. OVER/UNDER Labels versus Annotated Boundaries
OVER/UNDER are attention-based diagnostic states, not binary boundary predictions. Each model/seed has 134 candidates (32 matched, 102 unmatched).
Table B1. OVER/UNDER states versus annotated boundary matches.
| Model | Seed | OVER Matched | OVER Unmatched | UNDER Matched | UNDER Unmatched |
|---|---|---|---|---|---|
| Qwen2.5-VL-7B | 1 | 8 | 38 | 24 | 64 |
| 2 | 12 | 37 | 20 | 65 | |
| 3 | 10 | 40 | 22 | 62 | |
| LLaVA-OneVision-7B | 1 | 8 | 44 | 24 | 58 |
| 2 | 8 | 54 | 24 | 48 | |
| 3 | 8 | 55 | 24 | 47 |
The results show no direct correspondence between OVER/UNDER and boundary presence. Unmatched candidates are not necessarily false boundaries because annotations are non-exhaustive.
3. Prediction Changes: Wrong→Right and Right→Wrong
We compare Base and SSACD using binary YES/NO scene-change accuracy, counting parsing failures as incorrect. This differs from the free-generation STH Overall metric.
Table B2. Prediction changes from Base to SSACD.
| Model | Seed | Wrong→Right | Right→Wrong | Net Gain (pp) |
|---|---|---|---|---|
| Qwen2.5-VL-7B | 1 | 3 | 2 | +1.35 |
| 2 | 5 | 1 | +5.41 | |
| 3 | 4 | 2 | +2.70 | |
| LLaVA-OneVision-7B | 1 | 12 | 2 | +13.51 |
| 2 | 17 | 0 | +22.97 | |
| 3 | 11 | 3 | +10.81 |
Wrong→right changes consistently outnumber right→wrong changes across both backbones and all seeds, yielding positive net accuracy gains.
4. GT-Conditioned Policy and Restricted Oracle Reference
We evaluate two reference strategies:
- GT boundary rule: Applies shift perturbation to boundary-matched candidates and context perturbation otherwise.
- Two-mode answer oracle: Retrospectively selects a correct answer, if available, from saved Shift-only and Context-only outputs.
Table B3. Binary decision accuracy (mean ± sample SD, %).
| Model | Base | SSACD (Ours) | GT Boundary Rule | Two-Mode Answer Oracle |
|---|---|---|---|---|
| Qwen2.5-VL-7B | 76.13 ± 3.12 | 79.28 ± 2.06 | 77.03 ± 3.58 | 80.63 ± 3.12 |
| LLaVA-OneVision-7B | 56.31 ± 3.90 | 72.07 ± 4.75 | 71.62 ± 4.87 | 75.68 ± 3.58 |
The GT boundary rule does not outperform SSACD, suggesting that boundary presence alone is insufficient for optimal perturbation selection. The GT rule is not an upper bound, while the two-mode oracle provides only a restricted ceiling over the saved fixed-mode outputs, not over arbitrary per-candidate assignments or adaptive policies.
5. Manuscript Clarification
We will include this exploratory analysis with explicit annotation provenance, matching criteria, implementation differences, and oracle limitations. We will clarify that OVER/UNDER are attention diagnostics rather than boundary labels, and avoid claims of general boundary-detection accuracy or globally optimal perturbation selection.
C. [Question 3] Fig. 1 Variability and Natural-Range Interventions
Could you repeat the attention interventions within the natural range (e.g., 0.2–0.35), and report the sample sizes and variance for Fig. 1?
Thank you for this suggestion. We conducted additional analyses on VidHalluc STH to quantify Fig. 1 variability and evaluate interventions within the natural attention range.
1. Fig. 1 Sample Sizes and Variability
Attention values are reported as mean ± SD across unique videos. Paired differences are computed as generated-error attention minus teacher-forced ground-truth attention.
Table C1. Attention statistics for Fig. 1 (mean ± SD).
| Model / Panel | Videos | Teacher-forced GT | Generated Error | Paired Difference [95% Bootstrap CI] |
|---|---|---|---|---|
| Qwen2.5-VL / Hallucination | 93 | 0.2443 ± 0.1086 | 0.2621 ± 0.0938 | +0.0178 [0.0078, 0.0272] |
| LLaVA-OneVision / Omission | 81 | 0.2506 ± 0.0737 | 0.2285 ± 0.0796 | −0.0220 [−0.0345, −0.0101] |
These within-video comparisons reveal attention differences between teacher-forced ground-truth responses and generated errors. However, they do not isolate response-content effects from teacher-forcing versus free-generation differences or establish the causes of naturally occurring errors.
2. Natural-Range Attention Interventions
We repeated the Fig. 2 intervention protocol on VidHalluc STH using LLaVA-OneVision, with target attention ratios of 0.25, 0.30, and 0.35. We evaluated hallucination under attention boosting on non-transition samples and omission under attention suppression on true-transition samples, comparing semantic-shift regions with size-matched random-region controls.
Each target used 150 paired hallucination samples and 248 paired omission samples.
Table C2. Natural-range attention intervention results on VidHalluc STH.
| Target Ratio | Error Type | Paired Samples | Semantic-shift Error (%) | Random-region Error (%) | Difference (pp) |
|---|---|---|---|---|---|
| 0.25 | Hallucination | 150 | 30.00 | 28.00 | +2.00 |
| Omission | 248 | 5.65 | 4.03 | +1.61 | |
| 0.30 | Hallucination | 150 | 28.67 | 28.00 | +0.67 |
| Omission | 248 | 5.24 | 4.03 | +1.21 | |
| 0.35 | Hallucination | 150 | 27.33 | 26.67 | +0.67 |
| Omission | 248 | 5.24 | 3.63 | +1.61 |
Across all three targets, semantic-shift interventions yield higher error rates than random-region controls for both failure modes. Hallucination differences range from +0.67 to +2.00 pp, while omission differences range from +1.21 to +1.61 pp.
Although modest, these consistently positive differences align with Fig. 2, suggesting that semantic-shift-specific failure tendencies remain observable within the natural attention range. The effect magnitude nevertheless varies with intervention settings.
Our motivating analysis aims to investigate whether attention manipulation can amplify scene-transition errors, rather than establish uniformly strong causal effects across all attention ranges. Fig. 2 shows stronger failure amplification under more aggressive interventions, while these additional results provide complementary evidence of smaller, directionally consistent effects within the natural range.
We will refine the Fig. 2 discussion to clarify the intervention-dependent effect magnitude and avoid claims of uniformly strong causal effects. These observations motivate SSACD, which leverages failure-amplified predictions for contrastive decoding.
Thank you for your constructive comments. We address your five questions below, covering (1) longer-video evaluation and claim scope, (2) boundary-grounded analysis, (3) natural-range attention interventions, (4) reproducibility, and (5) recent backbones. We distinguish supplementary controlled studies from the submitted main-result configurations and qualify conclusions where evidence remains limited.
We provide detailed responses and supporting experimental results for each concern below.
A. [Question 1] Longer-Video Evaluation and Claim Scope
To investigate the effectiveness of SSACD under larger frame budgets, we conducted additional evaluations on LVBench with 32 and 64 frames for both backbones, including corresponding answer-parsing failure rates.
1. LVBench Accuracy with 32/64 Frames
The results below report mean ± sample standard deviation across three seeds, with gains expressed in percentage points.
Table A1. LVBench accuracy with 32 and 64 frames (mean ± SD, %).
SSACD consistently improves mean LVBench accuracy over Base at both tested frame budgets for both backbones. These results demonstrate that the performance improvements extend beyond the original eight-frame setting and remain positive under larger frame budgets. However, they do not establish exhaustive scene-boundary detection or scalability to arbitrary video lengths.
2. Answer-Parsing Failure Rates with 32/64 Frames
We additionally report the corresponding LVBench answer-parsing failure rates (%), expressed as mean ± sample standard deviation across three seeds.
Table A2. LVBench answer-parsing failure rates with 32 and 64 frames (mean ± SD, %, lower is better).
SSACD exhibits no observed answer-parsing failures across the evaluated 32/64-frame runs on either backbone, as does VCD. Since the Base parsing failure rates are already low (0.065–0.258%), the observed LVBench accuracy gains are unlikely to be explained solely by reductions in answer-parsing failures. Nevertheless, these downstream improvements do not establish that improved scene-boundary understanding is their sole cause.
3. Manuscript Clarification
Following the reviewer's suggestion, we will include the additional LVBench accuracy and answer-parsing results in Sec. 5.2 and explicitly narrow our longer-video claims to the observed improvements under the evaluated frame budgets. We will also clarify that sparse frame sampling may miss brief scene transitions and that the current results do not establish exhaustive boundary detection or scalability to arbitrary video lengths.