acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Scene Transition Misunderstanding in Video LLMs via Semantic-Shift-Aware Contrastive Decoding

Abstract

Recognizing scene transitions is crucial for long-video understanding, yet Video Large Language Models (Video LLMs) often hallucinate nonexistent transitions or omit real ones. We analyze these failures through semantic-shift regions and find that over-attending to non-boundary shifts promotes hallucination, while under-attending to boundary-relevant shifts promotes omission. Controlled interventions provide evidence that such mis-calibrated attention causally contributes to scene-transition misunderstanding. Based on this finding, we propose SSACD (Semantic Shift-Aware Contrastive Decoding), a training-free method that preserves the original prediction in a clean branch while amplifying the identified attention tendency in an auxiliary branch and contrastively suppressing the resulting failure-prone predictions. SSACD realizes this principle through attention-level sharpening and input-level perturbation. Across multiple Video LLMs and benchmarks, SSACD outperforms existing training-free decoding methods on scene-transition understanding and improves multi-scene long-video performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video-LLMs, Multimodal LLMs, Hallucination, Contrastive Decoding, Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gxXqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies two kinds of scene-transition errors in Video LLMs: reporting a transition when there is none (hallucination) and missing a real one (omission). Using last-layer attention interventions, the authors argue that over-attending to non-boundary semantic shifts causes hallucination, while under-attending to true boundaries causes omission. Based on this, they propose SSACD, a training-free method: semantic shifts between adjacent frames are detected with CLIP, each shift is labeled OVER or UNDER from its attention ratio in a preliminary decoding pass, noise is added to the corresponding frames to build a "failure-amplified" branch, and the two branches are combined with VCD-style contrastive decoding. With Qwen2.5-VL-7B and LLaVA-OV-7B at 8 uniformly sampled frames, SSACD outperforms DINO-HEAL, TCD and VCD on VidHalluc, the scene-transition subsets of MVBench/TVBench, and on LVBench and MLVU.

Critical strengths
  1. The problem is decomposed clearly, and the story from diagnosis to method is coherent. Scene-transition errors are split into hallucination and omission, which map to two perturbation modes, and the ablation supports this split (Table 3: removing Context Perturbation raises hallucination from 22.2 to 31.0, and removing Shift Perturbation raises omission from 35.8 to 42.1).
  2. The gains on VidHalluc are clear and backed by a key control: in Table 3, perturbing all frames uniformly gives 46.6, while shift-aware perturbation reaches 50.7, showing the gain is not just from VCD-style global perturbation. The method is training-free, and the authors report the inference overhead honestly (Table 4).
Critical weaknesses

W1. Effectiveness on long videos has not been demonstrated. The abstract (L025) and §5.2 (p.7, L377 onward) present long-video improvement as a main result, yet all experiments use only 8 frames (§5.1, p.7, L361–362). Following TCD's 8-frame setting is reasonable for the scene-transition benchmarks, but LVBench videos are hour-long, and at 8 frames adjacent samples are minutes apart, so it is hard to say the model still "sees" scene transitions. The base models score only 23.3 (Qwen) and 25.4 (LLaVA) on LVBench, close to the chance level of four-way multiple choice; SSACD's gains there (+9.5 and +6.6) are the largest in the table, larger even than its margins over the strongest baseline on the scene-transition tasks, while VCD and TCD gain only about 1 point. From such a near-chance starting point, these gains are hard to attribute to better discrimination of scene transitions.

W2. The inference-time rule is inconsistent with the diagnostic analysis. The diagnostic experiments separate transition and non-transition samples using ground truth (§3, p.4, L190–191) and conclude that over-attending to non-boundaries and under-attending to true boundaries cause the two error types. At inference, however, a shift is labeled only by whether its attention ratio exceeds γ (§4.2, p.6, L270–274), regardless of whether it is a true boundary. A true boundary that is correctly attended to would then be labeled OVER and suppressed by the contrast, which could itself cause omission. The paper does not show how this labeling corresponds to true boundaries.

W3. The "causally contributes" claim is too strong. In natural conditions, the attention ratio of correct and incorrect samples differs by only about 0.02 around 0.25 (Fig. 1, p.2), whereas in Fig. 2 (p.4) error rates rise noticeably mostly above 0.4, and above 0.7 for LLaVA on TVBench/MVBench. On VidHalluc, the random-region control also shows a sizable effect, and LLaVA's Suppress curve nearly overlaps the random control. Interventions far outside the natural range show that forcing attention this way can cause errors, but not that naturally occurring errors stem from it.

W4. Missing hyperparameters and statistical uncertainty. The default values of α, β, σ, γ and λ are not given anywhere (§5.1, p.7, L363–364 points to App. D, which contains only a sensitivity plot); the sensitivity analysis (Figs. 5–6) is done on VidHalluc, which is also the main test set; and results "averaged over three seeds" come without standard deviations. Given that the margins over the stronges

Questions for authors
  1. Could you report base and SSACD results on LVBench and MLVU with 32/64 frames, along with answer-parsing failure rates? If not, would you consider softening the long-video claims?
  2. On a subset with ground truth, could you show the confusion matrix between OVER/UNDER labels and true boundaries, break SSACD's changes into wrong→right and right→wrong, and give an oracle upper bound where the perturbation mode is chosen from ground truth?
  3. Could you repeat the attention interventions within the natural range (e.g., 0.2–0.35), and report the sample sizes and variance for Fig. 1?
  4. Could you list all default hyperparameters, state which data were used for tuning, and report the standard deviation over the three seeds and the configurations of the baselines?
  5. On more recent Video LLMs, are scene-transition hallucination and omission still significant, and how much of SSACD's gain remains?
6 comments

Thank you for your constructive comments. We address your five questions below, covering (1) longer-video evaluation and claim scope, (2) boundary-grounded analysis, (3) natural-range attention interventions, (4) reproducibility, and (5) recent backbones. We distinguish supplementary controlled studies from the submitted main-result configurations and qualify conclusions where evidence remains limited.

We provide detailed responses and supporting experimental results for each concern below.

A. [Question 1] Longer-Video Evaluation and Claim Scope

Could you report base and SSACD results on LVBench and MLVU with 32/64 frames, along with answer-parsing failure rates? If not, would you consider softening the long-video claims?

To investigate the effectiveness of SSACD under larger frame budgets, we conducted additional evaluations on LVBench with 32 and 64 frames for both backbones, including corresponding answer-parsing failure rates.

1. LVBench Accuracy with 32/64 Frames

The results below report mean ± sample standard deviation across three seeds, with gains expressed in percentage points.

Table A1. LVBench accuracy with 32 and 64 frames (mean ± SD, %).

ModelFramesBaseSSACD (Ours)Gain (pp)
Qwen2.5-VL-7B3235.61 ± 1.0039.14 ± 1.64+3.53
6436.56 ± 0.5338.78 ± 1.79+2.22
LLaVA-OneVision-7B3235.96 ± 1.0437.74 ± 0.53+1.78
6435.96 ± 0.6638.15 ± 0.67+2.19

SSACD consistently improves mean LVBench accuracy over Base at both tested frame budgets for both backbones. These results demonstrate that the performance improvements extend beyond the original eight-frame setting and remain positive under larger frame budgets. However, they do not establish exhaustive scene-boundary detection or scalability to arbitrary video lengths.

2. Answer-Parsing Failure Rates with 32/64 Frames

We additionally report the corresponding LVBench answer-parsing failure rates (%), expressed as mean ± sample standard deviation across three seeds.

Table A2. LVBench answer-parsing failure rates with 32 and 64 frames (mean ± SD, %, lower is better).

ModelFramesBaseDINO-HEALVCDSSACD (Ours)
Qwen2.5-VL-7B320.258 ± 0.2240.172 ± 0.0750.000 ± 0.0000.000 ± 0.000
640.151 ± 0.1490.237 ± 0.1490.000 ± 0.0000.000 ± 0.000
LLaVA-OneVision-7B320.065 ± 0.0650.065 ± 0.0650.000 ± 0.0000.000 ± 0.000
640.065 ± 0.0650.065 ± 0.0650.000 ± 0.0000.000 ± 0.000

SSACD exhibits no observed answer-parsing failures across the evaluated 32/64-frame runs on either backbone, as does VCD. Since the Base parsing failure rates are already low (0.065–0.258%), the observed LVBench accuracy gains are unlikely to be explained solely by reductions in answer-parsing failures. Nevertheless, these downstream improvements do not establish that improved scene-boundary understanding is their sole cause.

3. Manuscript Clarification

Following the reviewer's suggestion, we will include the additional LVBench accuracy and answer-parsing results in Sec. 5.2 and explicitly narrow our longer-video claims to the observed improvements under the evaluated frame budgets. We will also clarify that sparse frame sampling may miss brief scene transitions and that the current results do not establish exhaustive boundary detection or scalability to arbitrary video lengths.

B. [Question 2] Boundary-Grounded Labels, Prediction Changes, and Oracle References

On a subset with ground truth, could you show the confusion matrix between OVER/UNDER labels and true boundaries, break SSACD's changes into wrong→right and right→wrong, and give an oracle upper bound where the perturbation mode is chosen from ground truth?

We conducted a controlled study on a balanced 74-video VidHalluc STH subset (37 transition and 37 non-transition videos), using eight frames and three seeds (1, 2, 3). The supplementary implementation differs from that used for the submitted Table 1; its scores are not directly substituted for those results.

1. Ground-Truth Scope and Boundary Matching

Transition timestamps were reconstructed from video assembly splices and screened through assistant-assisted visual inspection; negative labels came from the benchmark. These are neither independent human annotations nor exhaustive boundary labels.

A candidate matches when its sampled-frame interval, extended by 0.25 seconds at each endpoint, contains an annotated transition. This measures interval containment, not boundary localization within ±0.25 seconds. Since positive transitions are concentrated near video midpoints, this is a controlled evaluation rather than a general boundary-detection benchmark.

2. OVER/UNDER Labels versus Annotated Boundaries

OVER/UNDER are attention-based diagnostic states, not binary boundary predictions. Each model/seed has 134 candidates (32 matched, 102 unmatched).

Table B1. OVER/UNDER states versus annotated boundary matches.

ModelSeedOVER MatchedOVER UnmatchedUNDER MatchedUNDER Unmatched
Qwen2.5-VL-7B18382464
212372065
310402262
LLaVA-OneVision-7B18442458
28542448
38552447

The results show no direct correspondence between OVER/UNDER and boundary presence. Unmatched candidates are not necessarily false boundaries because annotations are non-exhaustive.

3. Prediction Changes: Wrong→Right and Right→Wrong

We compare Base and SSACD using binary YES/NO scene-change accuracy, counting parsing failures as incorrect. This differs from the free-generation STH Overall metric.

Table B2. Prediction changes from Base to SSACD.

ModelSeedWrong→RightRight→WrongNet Gain (pp)
Qwen2.5-VL-7B132+1.35
251+5.41
342+2.70
LLaVA-OneVision-7B1122+13.51
2170+22.97
3113+10.81

Wrong→right changes consistently outnumber right→wrong changes across both backbones and all seeds, yielding positive net accuracy gains.

4. GT-Conditioned Policy and Restricted Oracle Reference

We evaluate two reference strategies:

  • GT boundary rule: Applies shift perturbation to boundary-matched candidates and context perturbation otherwise.
  • Two-mode answer oracle: Retrospectively selects a correct answer, if available, from saved Shift-only and Context-only outputs.

Table B3. Binary decision accuracy (mean ± sample SD, %).

ModelBaseSSACD (Ours)GT Boundary RuleTwo-Mode Answer Oracle
Qwen2.5-VL-7B76.13 ± 3.1279.28 ± 2.0677.03 ± 3.5880.63 ± 3.12
LLaVA-OneVision-7B56.31 ± 3.9072.07 ± 4.7571.62 ± 4.8775.68 ± 3.58

The GT boundary rule does not outperform SSACD, suggesting that boundary presence alone is insufficient for optimal perturbation selection. The GT rule is not an upper bound, while the two-mode oracle provides only a restricted ceiling over the saved fixed-mode outputs, not over arbitrary per-candidate assignments or adaptive policies.

5. Manuscript Clarification

We will include this exploratory analysis with explicit annotation provenance, matching criteria, implementation differences, and oracle limitations. We will clarify that OVER/UNDER are attention diagnostics rather than boundary labels, and avoid claims of general boundary-detection accuracy or globally optimal perturbation selection.

C. [Question 3] Fig. 1 Variability and Natural-Range Interventions

Could you repeat the attention interventions within the natural range (e.g., 0.2–0.35), and report the sample sizes and variance for Fig. 1?

Thank you for this suggestion. We conducted additional analyses on VidHalluc STH to quantify Fig. 1 variability and evaluate interventions within the natural attention range.

1. Fig. 1 Sample Sizes and Variability

Attention values are reported as mean ± SD across unique videos. Paired differences are computed as generated-error attention minus teacher-forced ground-truth attention.

Table C1. Attention statistics for Fig. 1 (mean ± SD).

Model / PanelVideosTeacher-forced GTGenerated ErrorPaired Difference [95% Bootstrap CI]
Qwen2.5-VL / Hallucination930.2443 ± 0.10860.2621 ± 0.0938+0.0178 [0.0078, 0.0272]
LLaVA-OneVision / Omission810.2506 ± 0.07370.2285 ± 0.0796−0.0220 [−0.0345, −0.0101]

These within-video comparisons reveal attention differences between teacher-forced ground-truth responses and generated errors. However, they do not isolate response-content effects from teacher-forcing versus free-generation differences or establish the causes of naturally occurring errors.

2. Natural-Range Attention Interventions

We repeated the Fig. 2 intervention protocol on VidHalluc STH using LLaVA-OneVision, with target attention ratios of 0.25, 0.30, and 0.35. We evaluated hallucination under attention boosting on non-transition samples and omission under attention suppression on true-transition samples, comparing semantic-shift regions with size-matched random-region controls.

Each target used 150 paired hallucination samples and 248 paired omission samples.

Table C2. Natural-range attention intervention results on VidHalluc STH.

Target RatioError TypePaired SamplesSemantic-shift Error (%)Random-region Error (%)Difference (pp)
0.25Hallucination15030.0028.00+2.00
Omission2485.654.03+1.61
0.30Hallucination15028.6728.00+0.67
Omission2485.244.03+1.21
0.35Hallucination15027.3326.67+0.67
Omission2485.243.63+1.61

Across all three targets, semantic-shift interventions yield higher error rates than random-region controls for both failure modes. Hallucination differences range from +0.67 to +2.00 pp, while omission differences range from +1.21 to +1.61 pp.

Although modest, these consistently positive differences align with Fig. 2, suggesting that semantic-shift-specific failure tendencies remain observable within the natural attention range. The effect magnitude nevertheless varies with intervention settings.

Our motivating analysis aims to investigate whether attention manipulation can amplify scene-transition errors, rather than establish uniformly strong causal effects across all attention ranges. Fig. 2 shows stronger failure amplification under more aggressive interventions, while these additional results provide complementary evidence of smaller, directionally consistent effects within the natural range.

We will refine the Fig. 2 discussion to clarify the intervention-dependent effect magnitude and avoid claims of uniformly strong causal effects. These observations motivate SSACD, which leverages failure-amplified predictions for contrastive decoding.

Reviewer_g5H3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper begins with the effect of boosting and suppressing attention ratios towards visual tokens via attention steering. Motivated from the observation that boost/suppress increase hallucination/omission, the proposed method intentionally induces the hallucination or suppression by the attention steering. Attention scores of visual tokens that corresponds to the two frames of semantic shift are reshaped with PASTA algorithm. Also, input frame perturbation is introduced with three different heuristics. The experiments include two models on 5 benchmarks, and ablations on the proposed components.

Critical strengths
  1. The authors uncover that hallucination and omission are systematically linked to mis-calibrated attention around semantic-shift regions
  2. The paper integrates attention steering and visual contrastive decoding.
Critical weaknesses
  1. The proposed method is only applicable to scene transition related QAs, it cannot be generalized to other visual tasks.
  2. The motivation explains the observation on attention steering but the main experiments use input frame perturbation, therefore the overall logic flow is not convincing.
Questions for authors
  1. Is the method robust to diverse distance metrics in Eq. 1?
  2. At line 197, it is mentioned "hallucination increases as more attention is allocated to non-boundary semantic shifts". If the attention towards the boundary gets higher, why does hallucination increase? The claim is not intuitive and the explanation is not enough.
  3. Does the result of the observation hold consistently when higher attention is allocated to semantic shifts?
  4. Can we still use attention acceleration tools such as flash attention with the SSACD-Attn? Shouldn't it be forced to use eager?
  5. Is K=8 enough to detect the genuine semantic shift? What if there are significant compositional differences on i-th and i+1-th frames but not a scene transition? It is concerning given that the model tests on long video benchmarks.
  6. How robust is the method by steering attention on different layers? It would be ideal if authors included a graph to show how robust the solution is regarding the layers.
  7. Why can the smoothing be a necessary baseline in Table 2?
5 comments

Thank you for your insightful comments and constructive suggestions. We address your concerns by (1) clarifying the broader applicability of SSACD beyond scene-transition-specific QA and the unified motivation behind its two failure-amplification strategies, (2) providing additional evaluations of robustness to distance metrics, frame sampling, and attention intervention layers, and (3) clarifying the interpretation of our attention intervention findings, FlashAttention compatibility, and the rationale for the smoothing baseline. We provide detailed explanations and supporting experimental results for each point below.

We provide detailed responses and supporting experimental results for each concern below.

A. [Weakness 1] Applicability Beyond Scene-Transition-Specific QA

The proposed method is only applicable to scene transition related QAs, it cannot be generalized to other visual tasks.

We would like to clarify that SSACD is not limited to scene-transition-specific QA tasks. As already reported in Table 1, we evaluate SSACD not only on scene-transition benchmarks but also on general long-video understanding benchmarks, LVBench and MLVU, which cover diverse temporal reasoning tasks ([L344–L355]).

  • General long-video understanding: SSACD consistently improves performance across both backbones. On Qwen2.5-VL-7B, LVBench accuracy increases from 23.3 to 32.8 (+9.5 pp) and MLVU from 31.8 to 37.4 (+5.6 pp). Similarly, on LLaVA-OneVision-7B, LVBench improves from 25.4 to 32.0 (+6.6 pp) and MLVU from 35.1 to 39.1 (+4.0 pp) (Table 1, [L378–L395]).
  • Broader applicability: These results demonstrate that the benefits of SSACD extend beyond explicitly identifying scene transitions. As discussed in Sec. 5.2 ([L405–L410]), distinguishing genuine scene transitions from incidental semantic changes is relevant to long-range temporal understanding. The improvements on general benchmarks support the broader applicability of our approach, rather than restricting it to scene-transition-specific QA.

B. [Weakness 2] Unified Motivation for Attention-Level and Input-Level Failure Amplification

We would like to clarify that attention-level sharpening and input-level perturbation share the same objective: amplifying the identified mis-calibrated attention tendencies around semantic-shift regions to construct a failure-amplified auxiliary branch for contrastive decoding. The two methods differ only in how this amplification is induced ([L98–L103], [L278–L280]).

The motivation explains the observation on attention steering but the main experiments use input frame perturbation, therefore the overall logic flow is not convincing.

  • Two implementations of the same objective: Attention-level sharpening directly rescales attention weights to strengthen over- or under-attention. Input-level perturbation achieves the same objective by selectively weakening contextual or shift-related visual evidence, inducing the model to amplify the corresponding attention tendency through its own computation ([L284–L316]). Figure 4 confirms that input-level perturbation induces the intended attention amplification ([L430–L449]).
  • Why input-level perturbation is used in the main experiments: Direct attention sharpening imposes externally specified changes to attention weights, which may disrupt useful content-dependent relationships among visual evidence. In contrast, input-level perturbation preserves the model's learned attention computation, allowing the resulting attention distribution to reflect the remaining visual evidence ([L414–L428]). Table 2 shows that both approaches improve over the base models, while input-level perturbation achieves stronger performance: 65.7 vs. 60.5 on Qwen2.5-VL and 50.7 vs. 43.0 on LLaVA-OneVision. Table 3 further supports the effectiveness of semantic-shift-aware perturbation and adaptive perturbation selection ([L453–L461]).
  • Manuscript clarification: To make this shared motivation explicit, we will revise Sec. 4.1 to establish the common failure-amplification objective before introducing the two implementations, and Sec. 4.3.2 to explain why selective input perturbation can induce the intended attention changes while preserving the model's learned attention computation. We will also strengthen Sec. 5.3 by explicitly connecting the attention changes verified in Fig. 4 with the performance comparison in Table 2, clarifying why input-level perturbation is adopted in our main experiments.

Thus, input-level perturbation is not a departure from our attention-based motivation, but an alternative and empirically more effective implementation of the same failure-amplification principle.


**C. [Question 1] Robustness to the Distance Metric

Is the method robust to diverse distance metrics in Eq. 1?

We evaluated the sensitivity of our method to distance metrics on 445 VidHalluc STH videos, using LLaVA-OneVision, eight frames, and three random seeds (1, 2, and 3).

For L2-normalized embeddings, cosine, Euclidean, and normalized angular distances are monotonic transformations of one another. Therefore, under equivalent thresholds, they yield identical pair selections. To examine sensitivity beyond these mathematically equivalent metrics, we additionally evaluated an L1-based ranking control. Specifically, we retained cosine-based eligibility and the budget of one selected pair per eligible video, while replacing cosine distance with L1 distance for pair ranking. All other experimental settings were held fixed.

Table: Robustness to different distance metrics on VidHalluc STH (mean ± SD across three seeds).

Distance / Selection RuleSTH Overall (%)
Cosine (dcos>0.2d_{\mathrm{cos}} > 0.2)50.70 ± 1.12
Euclidean (d2>0.4d_2 > \sqrt{0.4})50.70 ± 1.12
Normalized angular (dang>0.2048d_{\mathrm{ang}} > 0.2048)50.70 ± 1.12
L1 ranking (matched eligibility and pair budget)50.52 ± 1.78

The L1 ranking control changed the selected pair in 28 of 399 eligible videos (7.02%), indicating that pair selection remained unchanged for 92.98% of eligible videos. The mean STH Overall score was 50.52% under L1 ranking, compared with 50.70% under cosine ranking, a difference of −0.18 percentage points.

The seed-wise paired differences (L1 − cosine) were −2.09, −1.28, and +2.85 percentage points, with a mean difference of −0.18 ± 2.65 percentage points (sample SD). Thus, although the average performance was similar, the seed-level differences exhibited noticeable variability, and the results should not be interpreted as establishing statistical equivalence.

Overall, these findings establish invariance to mathematically equivalent distance formulations and show that the tested L1 ranking variation preserves most pair selections while yielding similar mean performance. Importantly, the L1 control retains cosine-based eligibility rather than replacing the entire distance-based selection procedure. Therefore, our conclusions are limited to robustness against the tested ranking variation and do not imply invariance to arbitrary distance metrics.


D. [Question 2] Clarification of Non-Boundary Semantic Shifts and Hallucination

We would like to clarify that a semantic shift does not necessarily indicate a genuine scene boundary. In our formulation, semantic shifts are candidate regions of abrupt visual-semantic changes between adjacent frames, which may also arise from viewpoint changes, object motion, or visual composition changes within a continuous scene ([L45–L50], [L149–L159]).

  • Why does increasing attention lead to hallucination? Our analysis specifically examines non-transition samples, where detected semantic shifts do not correspond to genuine scene boundaries. Increasing attention to these non-boundary shifts can cause the model to interpret incidental visual changes as evidence of a scene transition, thereby increasing hallucination. In contrast, suppressing attention to genuine boundary-relevant shifts can increase omission. These distinct effects are demonstrated through the controlled interventions in Fig. 2 ([L186–L204]).
  • Manuscript clarification: We will revise the explanation in Sec. 3 (L196–L207) to explicitly distinguish non-boundary semantic shifts from genuine scene boundaries and clarify that the observed hallucination increase concerns attention amplification in non-transition cases, rather than increased attention to genuine scene boundaries.

E. [Question 3] Consistency of Attention Intervention Effects

Our controlled interventions in Fig. 2 ([L186–L207]) show that increasing attention to non-boundary semantic shifts generally promotes hallucination, while suppressing attention to boundary-relevant shifts promotes omission, across the evaluated models and benchmarks.

However, the magnitude of these effects varies across intervention strengths, and the relationship is not strictly monotonic in every setting. Importantly, the size-matched random-region controls show that semantic-shift interventions produce stronger effects in several conditions, supporting the relevance of semantic-shift regions beyond arbitrary attention manipulation.

We will clarify in Sec. 3 that our findings establish a general directional trend under controlled interventions, rather than a universally monotonic relationship between semantic-shift attention and scene-transition errors.

F. [Question 4] Compatibility with FlashAttention

We would like to clarify that SSACD does not inherently require eager attention throughout the entire inference pipeline. The requirement depends on the specific operation and implementation variant.

  • SSACD-Attn: Direct attention sharpening requires explicit access to attention weights and therefore uses eager attention at the modified layer (the final decoder layer in our implementation). However, this does not preclude using FlashAttention in the remaining, unmodified layers through a hybrid implementation.
  • SSACD-Input: Our default variant requires attention weights only during the preliminary attention-profiling pass. During subsequent contrastive decoding, neither the clean nor the perturbed branch directly modifies attention weights. Thus, the decoding stage is compatible in principle with FlashAttention, without changing the proposed contrastive decoding objective.

We will clarify these stage-specific requirements in Appendix G. While our current implementation relies on eager attention for operations requiring explicit attention weights, the proposed framework is not inherently restricted to an eager-only implementation.


G. [Question 5] Semantic-Shift Detection and Robustness to Frame Sampling

We would like to clarify that SSACD does not assume detected semantic shifts are genuine scene boundaries. Semantic shifts are candidate regions of visual-semantic changes, which can arise from viewpoint or compositional changes even within a continuous scene ([L149–L159]). Our method uses these regions to identify and amplify attention tendencies, rather than directly treating them as scene transitions.

  • Ground-truth boundary analysis: To examine the effectiveness of the 8-frame setting, we analyzed a subset with annotated scene boundaries. Among 37 transition-containing videos, 32 (86.49%) had a detected candidate interval matched to the annotated boundary (0.25-second temporal tolerance). Meanwhile, only 32 of 134 detected candidate intervals across the balanced 74-video subset matched annotated boundaries, confirming that semantic-shift candidates also include non-boundary visual changes. These results characterize candidate detection in this subset, rather than guaranteeing exhaustive boundary localization.
  • Extended long-video evaluation: To further address the concern about sparse frame sampling, we evaluated SSACD on LVBench using 32 and 64 frames, extending the original 8-frame setting. The results below report mean ± sample standard deviation across three seeds.

Table G1. LVBench results with 32 and 64 frames (mean ± SD, %).

ModelFramesBaseSSACD (Ours)Gain (pp)
Qwen2.5-VL-7B3235.61 ± 1.0039.14 ± 1.64+3.53
6436.56 ± 0.5338.78 ± 1.79+2.22
LLaVA-OneVision-7B3235.96 ± 1.0437.74 ± 0.53+1.78
6435.96 ± 0.6638.15 ± 0.67+2.19

Additional evaluation on MLVU. We also evaluated Qwen2.5-VL-7B on a 560-question subset of the 2,129-question MLVU evaluation set, comprising 80 questions from each of seven tasks. All conditions use the same questions and three seeds. We report M-Avg (%), the equally weighted mean of the seven task accuracies, as mean ± sample standard deviation across seeds.

Table G2. MLVU subset results with 32 and 64 frames (mean ± SD, %).

ModelFramesBaseSSACD (Ours)Gain (pp)
Qwen2.5-VL-7B3255.18 ± 0.9357.08 ± 1.08+1.90
6457.62 ± 0.5760.48 ± 0.52+2.86

These MLVU results are interim results on the 560-question subset, rather than results for the full 2,129-question evaluation set; the remaining evaluation is ongoing.

SSACD improves mean performance over Base at both frame budgets on LVBench and the evaluated MLVU subset, supporting that its benefits extend beyond the original 8-frame setting.

We will clarify in Sec. 3 that semantic shifts are candidate regions rather than verified boundaries, and extend Sec. 5.2 with the 32/64-frame evaluations, explicitly identifying the scope of the interim MLVU results. We will also distinguish improvements in long-video understanding from exhaustive scene-boundary detection, which is not the objective of SSACD.

Reviewer_SnFZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Semantic Shift-Aware Contrastive Decoding (SSACD), a training-free approach to mitigating hallucinations in video large language models, particularly around semantic scene transitions. The method consists of three stages: (1) Semantic-Shift Detection, which identifies scene transitions and characterizes attention patterns as over-attended or under-attended; (2) Failure Amplification, which strengthens potentially erroneous predictions through attention-level sharpening or input-level Gaussian perturbation; and (3) Contrastive Suppression, which contrasts the predictions from clean and failure-amplified branches to reduce hallucinations. The experiments demonstrate improvements in hallucination mitigation across evaluated video understanding benchmarks, with ablation studies examining the effectiveness of the proposed components.

Critical strengths
  1. Semantic-shift-aware hallucination mitigation The paper identifies semantic scene transitions as an important source of hallucinations and explicitly incorporates this observation into the decoding process.
  2. Training-free framework with empirical improvements The proposed approach does not require additional model training and demonstrates improvements across the evaluated benchmarks. The ablation studies also provide useful evidence supporting the proposed components.
Critical weaknesses
  1. Limited methodological novelty The proposed framework largely builds upon existing contrastive decoding and perturbation-based techniques. Although incorporating semantic-shift information is an interesting direction, the paper does not sufficiently distinguish its methodological contributions from prior contrastive decoding approaches.

  2. Insufficient justification for input-level perturbation The input-level Gaussian perturbation appears to be a generic corruption strategy rather than one specifically designed to address scene-transition hallucinations. Interestingly, the input-level variant outperforms attention-level sharpening on VidHalluc (STH). This raises the question of whether the improvements primarily arise from the proposed semantic-shift-aware mechanism or from generic input perturbation.

  3. Insufficient comparisons with recent methods The experimental evaluation would benefit from comparisons with recent video hallucination mitigation approaches, particularly SEASON (CVPR 2026) and Temporal-Aware Activation Engineering (NeurIPS 2025), on overlapping benchmarks such as VidHalluc. Such comparisons are important for assessing the effectiveness and novelty of the proposed method relative to recent advances.

Questions for authors

● What is the motivation for using Gaussian perturbation to address scene-transition hallucinations? Why does the input-level variant outperform attention-level sharpening on VidHalluc (STH), despite the paper's motivation being centered on attention miscalibration? ● Could the authors provide a controlled comparison between the proposed semantic-shift-aware perturbation and generic perturbation without semantic-shift detection? This would help clarify whether the improvements arise specifically from the proposed mechanism. ● Could the authors provide comparisons with recent methods (e.g., SEASON (CVPR 2026)), on shared benchmarks such as VidHalluc under comparable backbone models and evaluation settings?

3 comments

Thank you for your insightful comments and constructive suggestions. (1) We have clarified the key methodological distinctions and design rationale of SSACD, (2) supported by controlled analyses already included in the original submission, and (3) conducted the requested additional comparisons with recent baselines.

We provide detailed responses and supporting experimental results for each concern below.

A. [Weakness 1–2 & Question 1] Design Rationale for Semantic-Shift-Aware Failure Amplification

What is the motivation for using Gaussian perturbation to address scene-transition hallucinations? Why does the input-level variant outperform attention-level sharpening on VidHalluc (STH), despite the paper's motivation being centered on attention miscalibration?

  • We would like to clarify that both attention-level sharpening and input-level perturbation share the same objective and induce the same direction of attention amplification: deliberately strengthening mis-calibrated attention tendencies around semantic-shift regions to construct a failure-amplified branch for contrastive suppression. They differ only in how this amplification is achieved ([L98–L103], [L278–L280]).
  • How does input-level perturbation amplify mis-calibrated attention?
    • Our goal is to amplify the identified attention tendencies by selectively weakening visual evidence, rather than applying generic visual corruption. Unlike VCD, which broadly degrades visual information to increase reliance on language priors, SSACD selectively perturbs visual evidence to increase the relative reliance on unperturbed regions ([L303–L316]).
    • Specifically, for over-attended semantic shifts, we perturb the surrounding context to further increase the relative attention to the shifts. For under-attended shifts, we perturb the shift-related frames to further reduce their relative attention ([L309–L316]). Gaussian noise is a simple and sufficient mechanism for selectively weakening visual evidence; the key design lies in which regions are perturbed and how the perturbation amplifies the intended attention tendency, rather than in the noise itself (Appendix B, [L693–L722]).
    • Figure 4 confirms that input-level perturbation induces the intended attention amplification ([L443–L446]), while Table 3 demonstrates that semantic-shift-aware perturbation and adaptive perturbation selection contribute to mitigating hallucination and omission ([L453–L461]). These results distinguish our approach from generic visual perturbation.
  • Why does input-level perturbation outperform attention-level sharpening?
    • Although both methods amplify the same attention tendencies, direct attention sharpening explicitly rescales attention weights ([L284–L295]) without accounting for the content-dependent relationships among visual evidence. This may inadvertently suppress useful information or overemphasize irrelevant regions ([L417–L428]).
    • In contrast, input-level perturbation selectively weakens visual evidence while preserving the model's learned attention computation ([L303–L306]). The model determines the resulting attention distribution based on the remaining visual evidence and its contextual relationships, rather than following externally imposed attention weights ([L425–L428]). This allows the auxiliary branch to amplify the intended failure tendency while retaining potentially useful content-dependent information.
    • We believe this distinction helps explain why input-level perturbation achieves stronger performance than direct attention sharpening in Table 2 ([L414–L428]), despite sharing the same underlying objective.

B. [Weakness 2 & Question 2] Controlled Comparison with Generic Perturbation

Could the authors provide a controlled comparison between the proposed semantic-shift-aware perturbation and generic perturbation without semantic-shift detection? This would help clarify whether the improvements arise specifically from the proposed mechanism.

  • We would like to clarify that the requested controlled comparisons are already provided in Tables 1 and 3.
    • Generic vs. semantic-shift-aware perturbation: Unlike VCD-style uniform visual corruption, SSACD selectively perturbs shift-related or contextual frames to amplify the identified attention tendency while preserving unperturbed visual evidence. As shown in Table 3, replacing semantic-shift-aware perturbation with uniform perturbation reduces performance from 50.7 to 46.6, consistent with the VCD result in Table 1.
    • Importance of semantic-shift detection: Replacing detected semantic-shift regions with random regions also reduces performance from 50.7 to 47.3, demonstrating the importance of identifying where to apply perturbation.
  • These controlled comparisons support that the improvement comes from semantic-shift-aware perturbation design rather than generic visual corruption alone.

C. [Weakness 3 & Question 3] Comparisons with Recent Video Hallucination Mitigation Methods

Could the authors provide comparisons with recent methods (e.g., SEASON (CVPR 2026)), on shared benchmarks such as VidHalluc under comparable backbone models and evaluation settings?

  • To address the reviewer's request, we have expanded our evaluation to include four additional recent baselines, including SEASON (CVPR 2026) and Temporal-Aware Activation Engineering (NeurIPS 2025), on shared benchmarks under comparable backbone models and evaluation settings.
  • As shown in the table below, SSACD consistently outperforms all four additional baselines, further demonstrating its effectiveness relative to recent video hallucination mitigation methods. We have incorporated these results into Table 1 of the revised manuscript.
ModelMethodClass. ↑Desc. ↑Overall ↑
Qwen2.5-VL-7BBase63.348.657.4
DINO-HEAL65.849.359.2
TCD65.553.560.7
VCD69.252.562.4
AVCD70.641.458.9
Temporal68.548.160.3
SEASON53.847.751.4
SSACD (Ours)73.454.165.7
LLaVA-OneVision-7BBase32.938.635.2
DINO-HEAL31.740.835.3
TCD44.749.246.5
VCD45.048.946.6
AVCD52.229.843.2
Temporal53.846.050.6
SEASON40.345.842.5
SSACD (Ours)50.551.150.7

We sincerely thank you for your thoughtful review and constructive feedback.

We would like to kindly follow up on our rebuttal, where we have provided additional experimental results and detailed responses to your comments.

We would greatly appreciate it if you could take a moment to review our responses and share any further feedback or questions you may have.

Thank you again for your time and thoughtful consideration.

Comments on the paper

1 comment

Dear Reviewers,

We sincerely thank all reviewers for their thoughtful evaluations and constructive suggestions on strengthening our work.

We were encouraged by your recognition of several strengths of our work: the semantic-shift-aware analysis of scene-transition errors, particularly the systematic link between hallucination/omission and mis-calibrated attention around semantic-shift regions (R#SnFZ, R#g5H3, R#gxXq); the connection from diagnosis to method, including the integration of attention steering and visual contrastive decoding (R#g5H3, R#gxXq); and the training-free framework with clear empirical improvements, supported by ablation studies demonstrating the effectiveness of the proposed components (R#SnFZ, R#gxXq).

We also appreciate the concerns raised by the reviewers and have carefully addressed them through clearer methodological explanations and additional experiments. Below, we summarize our responses to the major concerns shared across the reviews.

1. Clarifying the Novelty and Design Rationale of SSACD (R#SnFZ, R#g5H3)

We would like to clarify that our key methodological novelty lies in designing a targeted contrastive signal grounded in our diagnostic finding: over-attention to non-boundary semantic shifts promotes hallucination, while under-attention to boundary-relevant shifts promotes omission. Contrastive decoding serves as a mechanism for suppressing these failure-prone predictions, while Gaussian noise is a simple means of constructing the targeted perturbations.

  • Semantic-shift-aware contrastive signal design: Rather than applying generic visual corruption as in conventional VCD, SSACD identifies semantic-shift regions, profiles their attention, and selectively perturbs shift or contextual evidence to amplify the corresponding failure tendency. Table 3 demonstrates the benefit of this targeted design over generic perturbation.
  • Rationale for input-level perturbation: SSACD supports both direct attention sharpening and input-level perturbation. Directly rescaling attention weights may disrupt useful relationships among visual evidence by imposing externally specified attention changes. In contrast, targeted input perturbation allows the model to adjust its attention through its own learned computation, amplifying the intended failure tendency without prescribing internal attention weights. Figure 4 confirms the intended attention changes, while Table 2 demonstrates the stronger effectiveness of input-level perturbation.

2. Extended Long-Video Evaluation (R#g5H3, R#gxXq)

To address concerns about the limited number of sampled frames, we extended our long-video experiments to higher frame counts. The consistent performance improvements support the effectiveness of SSACD beyond the original 8-frame setting.

3. Comparisons with Additional Baselines and Backbones (R#SnFZ, R#gxXq)

We expanded our evaluation to include more recent hallucination mitigation baselines and Video LLM backbones. The additional results preserve the overall performance trends, providing further evidence for the effectiveness and generalizability of SSACD.

We provide detailed responses and additional analyses addressing the remaining reviewer-specific concerns in our individual replies. We appreciate these suggestions, which have helped us clarify our contributions and strengthen the experimental validation of our work.

We thank all reviewers again for their valuable feedback.

All Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.