acceptodds
Under review as a conference paper at ICLR 2027

ACTIV: Reducing Spatiotemporal Hallucinations in Video Language Models through Token Interventions

Abstract

Video language models have gained adoption in applications such as robotics and autonomous driving, where reliability and low inference latency are essential. Yet, these models still misreport where objects are and the order in which events occur. Contrastive decoding can reduce these errors without retraining by comparing predictions from original and altered visual inputs. However, constructing negatives by re-encoding altered videos increases inference latency. We target a useful accuracy–latency tradeoff by constructing spatial and temporal negatives directly from already-computed visual tokens. We introduce Adaptive Contrast with Token Interventions for Video (ACTIV), a training-free framework that constructs spatial and temporal views with only a single video encoding pass. ACTIV averages visual features across locations or time to bottleneck the corresponding information. During answer generation, ACTIV contrasts predictions from these alternatives with those from the original features, weighting each comparison by its effect on the prediction to favor words supported more strongly by the original video. ACTIV requires a single visual encoding and three decoder branches, achieving competitive hallucination mitigation with substantially lower latency than re-encoding-based contrastive decoding. Across three models and three hallucination benchmarks, ACTIV outperforms the evaluated training-free baselines in average score for each model; on LLaVA-Video-7B, it improves VidHalluc and EventHallusion scores by 3.87 and 5.38 points over the unmodified model, respectively, while reducing estimated model-inference latency by approximately **22–26**% relative to the state-of-the-art.

Primary area:
foundation or frontier models, including LLMs
Keywords:
video-language models, spatiotemporal hallucinations, contrastive decoding, visual token interventions, training-free inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xQ22OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Adaptive Contrast with Token Interventions for Video (ACTIV), a training-free framework for mitigating spatial and temporal hallucinations in video-language models. ACTIV constructs spatial and temporal negative representations by averaging already-encoded visual tokens across spatial locations or time steps, respectively, avoiding the repeated vision-encoder passes required by approaches such as SEASON. During decoding, it combines the two contrastive signals using weights derived from Jensen–Shannon divergence between next-token prediction distributions. The paper also introduces the Video Evidence Margin (VEM), which compares the likelihood of a generated answer under original visual tokens with its average likelihood under random feature replacements, as a diagnostic proxy for visual-feature dependence.

Critical strengths
  1. Practical reduction in vision-encoder computation.
    ACTIV constructs contrastive negatives directly from existing visual tokens, avoiding repeated vision-encoder passes. The reported latency breakdown supports this design choice: visual-processing costs are substantially lower than those of SEASON, while total latency is reduced by approximately 22–26% in the principal comparison. This is a meaningful systems contribution, although the method remains slower than standard decoding and retains considerable decoder-prefill overhead.
Critical weaknesses
  1. Insufficiently established novelty relative to existing works.
    ACTIV shares the central design of spatial–temporal contrastive decoding with SEASON. Its main changes are to construct negative representations by averaging existing visual tokens and to weight the contrastive branches using JSD. Although these changes reduce visual-encoder computation, the paper needs to establish more convincingly that they offer a substantive methodological advance beyond a computationally motivated replacement of existing components.

  2. Limited validation on newer model generations.
    The main evaluation uses Qwen2.5-VL-7B, LLaVA-OneVision-7B, and LLaVA-Video-7B, the same established model families used in prior work in this area. Newer model generations, including Qwen3-VL and LLaVA-OneVision-2, were available before this review, yet ACTIV is not evaluated on them.

Questions for authors
  1. Clarification of the methodological contribution.
    Given the similarity to SEASON and the existence of other single-encoding interventions, what evidence demonstrates that ACTIV's token-space averaging and JSD-based weighting provide a substantive methodological advantage rather than primarily replacing existing components to reduce encoder cost? A convincing answer supported by existing ablations or additional controlled results could materially change my assessment of the paper's novelty.

  2. Evidence of generalization to newer model generations.
    Can the authors provide results on at least one newer publicly available video-language backbone on a representative hallucination benchmark?

5 comments
AuthorsOpenReview ↗

Controlled comparison with SEASON, and a newer backbone (Gemma 4)

Thank you for the clear review. We are glad that you find the reduction in vision-encoder computation "a meaningful systems contribution", and that you find the paper "clear and easy to follow". Thank you also for the statement of what can change your assessment. You raised two concerns. We answer each one with a new result.

Short answers

  1. Is ACTIV more than a lower-cost replacement of SEASON's components? Yes. In a controlled comparison, ACTIV's negatives and ACTIV's weights each increase accuracy. The lower cost is an additional benefit.
  2. Results on a newer backbone? On Gemma 4 E4B, ACTIV improves EventHallusion by 2.20 points with no tuning, and the 95% interval is above zero. SEASON does not improve it.

Detailed answers

1. Clarification of the methodological contribution

You asked for controlled results. We ran a controlled comparison that changes one component at a time:

  • the negatives: SEASON's (re-encoded video) or ACTIV's (token averages);
  • the weights: SEASON's (JSD of attention over frames) or ACTIV's (JSD of next-token distributions, Eq. 5).

All rows use LLaVA-Video-7B, the full benchmarks, the same frames, the same decoder and α = 1. Rows 2–5 also use the same candidate filter (λ = 0.1, lines 267–269).

#NegativesWeightsEncoder passesVidHallucVideoHallucerEventHallusionOverallLatency (s)
0none (unmodified model)–178.2649.8667.2465.120.41
1SEASONAttention JSD, no filter (SEASON as published)379.2247.0871.1565.821.12
2SEASONAttention JSD379.5447.0871.1565.921.11
3SEASONOutput JSD379.9447.5272.1366.531.11
4ACTIVAttention JSD182.1049.6471.8867.870.82
5ACTIVOutput JSD182.1550.0272.6268.260.82

Overall is the mean of the three benchmark scores. Latency is the median on VidHalluc. Rows 1–3 use our implementation of SEASON (its code is not public), with a setting from the SEASON paper: α = 1 and β = 0.33. Rows 4–5 use ACTIV's setting from the paper (lines 300–301). We tuned no setting for this comparison. ACTIV's advantage does not depend on this α: at each tested α from 0.25 to 2, ACTIV is 1.17–3.14 points above the unmodified model. SEASON is 0.70 points above it (row 1).

The table shows three facts:

  • The token negatives are more accurate, not only cheaper. With the weights fixed, ACTIV's negatives add 1.95 points (row 2 → 4) and 1.73 points (row 3 → 5) to the overall score. The score increases on all three benchmarks in both cases. Eq. 4 and Figure 3 give the reason: token averaging removes an exact fraction of the variation along time or space. A change before or inside the encoder does not give this control.
  • The output-JSD weights add a further gain at no cost. They add 0.39 points with ACTIV's negatives (row 4 → 5) and 0.61 points with SEASON's own negatives (row 2 → 3). The gain is positive on all three benchmarks in both cases. These weights need no attention maps.
  • Together, the two changes add 2.44 points over SEASON as published (row 1 → 5). At the same time, ACTIV uses one encoder pass instead of three and has 26% lower latency.

Thus, ACTIV is not SEASON with cheaper parts: the largest gain comes from the new negatives. The paper agrees with this on the other backbones. Table 4(b) isolates the temporal negative on LLaVA-OneVision: ACTIV's negative is 3.62 and 1.72 points above SEASON's. In Table 1, the full method is 2.73 and 2.83 points above SEASON overall on Qwen2.5-VL and LLaVA-OneVision.

Other single-encoding methods. STEAR needs attention maps, a selection of layers and uncertainty thresholds (lines 462–465). ACTIV needs only the encoded tokens and the next-token distributions. Thus, it has no attention layers or uncertainty thresholds to select.

2. Evidence of generalization to newer model generations

We evaluated Gemma 4 E4B on EventHallusion (all 409 questions). Gemma 4 is from a third model family (not Qwen and not LLaVA), so it tests a newer model and a different decoder at the same time. Both methods use transferred settings, with no tuning on Gemma 4. ACTIV uses its Qwen2.5-VL settings without change. SEASON uses the same setting as above (α = 1, β = 0.33), and its attention weights read the two global-attention layers that are nearest to the relative depth of the layers in the SEASON paper.

MethodEncoder passesAccuracyChange from baseline [95% CI]Latency (s)
Baseline153.06–0.27
SEASON351.10−1.96 [−5.43, +1.47]0.67
ACTIV155.26+2.20 [+0.24, +4.15]0.48

ACTIV is 4.16 points above SEASON (95% CI [+1.21, +7.43]) and 28% faster. The intervals use bootstrap resampling of the source videos. On the two subsets where each correct answer is "no", ACTIV does not add "yes" answers: it gives 78, the same number as the unmodified model. SEASON gives 95.

Changes in the revision. We will add the controlled comparison and the Gemma 4 result to Section 4.5.

You wrote that a convincing controlled result can change your assessment of the novelty. We hope that the comparison above is this result. If a concern remains, please tell us. We will answer it during the discussion period.

AuthorsOpenReview ↗

Thank you for your time and comments

Dear Reviewer xQ22,

We hope all is well and would like to thank you again for providing reviews to our submission.

We are wondering if you have a chance to read our rebuttal response. If so, do you have any more issue that you would like us to clarify and/or provide any more results, given the author-reviewer exchanges will be closing in less than a day?

Thank you in advance for your timely feedback.

best, The Authors

Reviewer_KkaEOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free method, ACTIV, to reduce hallucinations in video language models under a low latency setting using contrasting decoding to shift model outputs towards those that are supported more strongly by the original video. ACTIV achieves low latency by constructing the contrastive spatial and temporal samples by interpolating between tokens from the encoded video frames, thus avoiding re-encoding the contrasting video.

The authors start from defining an answer-level margin score to measure whether the model uses video evidence derived from the difference between the actual features and feature-wise mean/SD random sample. The main ACTIV method involves the construction of temporal and spatial negatives by interpolating between the actual and mean features along the temporal/spatial dimension of the video, computing the outputs for these contrastive variants, and then shifting the original output logits using JSD-weighted differences from the contrastive logits. A variant also uses the answer-level margin to reweigh this shift.

The authors evaluate ACTIV on three hallucination benchmarks and across three base VLMs, showing improvements in performance at a relatively low latency cost. They then generalize these improvements to video understanding benchmarks. Further ablations and parameter generalization experiments provide additional evidence for their claims.

I generally liked that the paper had a simple but effective method that works across tasks. I have some minor concerns about the writing and differentiation from prior work but have an overall positive attitude.

Edit: The rebuttal helped clarify my relatively minor concerns and hence I have raised my score.

Critical strengths
  • The method is well-motivated and simple to implement. It works across multiple models and reduces hallucination without affecting general video understanding.
Critical weaknesses
  • The SEASON baseline also uses two negatives though a slightly different procedure. While the latency improvement is clear, its not clear why there is a large difference in benchmark performance.
Questions for authors

Can you explain why SEASON, which also uses contrastive decoding with related sampling strategies, fares much worse on hallucination benchmarks?

Minor: \citet and \citep are not used correctly, fix the usage.

2 comments
AuthorsOpenReview ↗

Why SEASON scores lower (controlled comparison)

Thank you for the positive and careful review. We are glad that you find the method simple, well-motivated and effective.

Summary

  1. Why is SEASON lower? ACTIV has better negatives and better weights. In a controlled comparison, ACTIV's negatives add 1.7–2.0 points and ACTIV's weights add 0.4–0.6 points. The better negatives are also the cheaper negatives: one encoder pass instead of three, and 22–26% lower latency than SEASON (Table 2).
  2. Is the comparison fair to SEASON? Yes. In this comparison, SEASON has the same frames, decoder and candidate filter as ACTIV, and it uses a setting from its own paper.
  3. Prior work and citations. We will state the differences from SEASON in Section 5 and correct the citations.

Detailed answers

1. Why is SEASON lower, when both methods use two negatives?

On LLaVA-Video-7B, we change one component at a time. Each cell is the overall score: the mean of VidHalluc, VideoHallucer and EventHallusion on the full benchmarks. The unmodified model scores 65.12.

SEASON weights (attention JSD)ACTIV weights (output JSD, Eq. 5)
SEASON negatives (3 encoder passes)65.9266.53
ACTIV negatives (1 encoder pass)67.8768.26

The negatives give most of the difference (+1.73 to +1.95 points), and the weights give the remainder (+0.39 to +0.61 points). Both effects have the same sign on each of the three benchmarks.

Why are ACTIV's negatives better? SEASON changes the input before or inside the vision encoder (noisy frames and homogenization). Thus, it does not control how much the final visual tokens change. ACTIV averages the final tokens directly, and Eq. 4 gives the exact reduction along time or space. Table 4(b) shows the same result for the temporal negative on a second backbone: on LLaVA-OneVision, ACTIV's temporal negative is 3.62 points (VideoHallucer) and 1.72 points (EventHallusion) above SEASON's.

2. The comparison is fair to SEASON

  • Equal conditions. All four cells use the same items, frames, decoder, candidate filter (λ = 0.1) and α = 1. We tuned no setting for this comparison: ACTIV keeps its setting from the paper (lines 300–301). The filter is not the cause of the difference: in this comparison, SEASON scores 65.92 with the filter and 65.82 without it.
  • SEASON's own setting. SEASON uses a setting from its own paper: α = 1 and β = 0.33 (SEASON paper, Appendix A.1).
  • A checked implementation. The SEASON code is not public, so we implemented it from the equations in its paper (lines 291–292), and we checked each step against these equations. On EventHallusion, our SEASON matches or exceeds the scores in the SEASON paper on the three backbones: 70.42 against 66.01, 69.19 against 69.19, and 71.15 against 66.99.

3. Prior work and citations. We will add the comparison above to Section 4.5. In Section 5, we will state the three differences from SEASON: where the method builds the negative (final tokens, not pixels or encoder layers), which signal sets the weights (output distributions, not attention maps), and the cost (one encoder pass, not three). We will also correct all \citet and \citep uses, and we will revise the text of Sections 2 and 3 for clarity.

Thank you again for your support. If one more result or clarification can increase your overall assessment in the paper, please tell us. We will provide it during the discussion period.

Reviewer_KkaEOpenReview ↗

Response to Rebuttal

Thank you to the authors for the careful comparison with SEASON. Based on this response, it is clear that the mechanism of constructing negatives is the biggest contributor to ACTIV's performance over baselines (in terms of task performance). At the same time, the mechanism of constructing negatives from token encodings in the same pass also reduces latency. I think clarifying these details actually strengthens the main arguments in the paper, so I have raised my score to an accept.

Reviewer_CQrTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper starts with observing if randomly replacing visual tokens degrades the video understanding performance. The observation tested if randomly replacing visual tokens gives a difference on next token probability. Based on this conclusion, the proposed method first applies interpolation with the raw features and spatially/temporally averaged scalar. The final logit is derived from contrastive decoding the raw logit and interpolated logit with APC and repetition penalty. The experiments include three models on three benchmarks, latency check, different interpolation alternatives and weighting rules.

Critical strengths
  • While Contrastive Decoding approaches are well-established in the image domain, they remain significantly under-explored within Video-LLMs. This work addresses this gap and paves the way for diverse future research directions in the field.
Critical weaknesses
  • The observation in Section 2 is trivial, and the result does not support the conclusion.
  • Introduction raised a concern on video frame to token process cost, but the method is not cost-efficient.
Questions for authors
  • What is the exact implementation of S(y;Z)? Does it estimate the first token log probability?
  • Where are the visualizations of the Attention maps in line 160?
  • "Nonpositive VEM is more common among incorrect answers in all five benchmark groups" I do not think the results on Figure 2 support the claim. In all datasets, the average is above zero.
  • The latency experiment should include the FLOPs, wall clock and latency for visual encoder forward and language model forward "separately" on both single-token answer and open-ended question types. I believe the language model forward has significantly higher portion of the compute.
  • There are too many hyperparameters to tune. The paper lacks the robustness against those hyperparameters.
  • Contrastive decoding is implemented under multinomial sampling, which is prone to seed noise. The experiments should include averaged results on different seed runs. [1]

[1] Leng et al. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. CVPR 2024.

9 comments
AuthorsOpenReview ↗

Answers to all questions, with the requested cost table and new sensitivity and seed results

Thank you for the detailed questions. We address each point below.

Summary.

Each item gives the short answer and the location of the evidence. The detailed answers below use the same numbers.

  1. Section 2 and Figure 2 (Weakness 1, Question 3). The claim is about the share of answers with VEM ≤ 0 (lines 175–177). It is not about the average. This share is higher for incorrect answers in 15 of 15 model–benchmark pairs (new table). The result is also not an expected one: for 8.6–33.3% of correct answers, random features give an equal or higher likelihood than the real video (lines 170–171). ACTIV decreases the share of answers with VEM ≤ 0 from 21.9% to 18.1% (lines 442–445).
  2. S(y;Z) (Question 1). The paper defines S as the length-normalized log-probability of the full answer (lines 147–148). It is not a first-token score.
  3. Attention maps (Question 2). Section 2 has no attention-map experiment. Lines 159–160 only state how VEM is different from attention maps.
  4. Cost (Weakness 2, Question 4). We give the requested table. We agree that the language model is the main cost. The additional vision encoding is a separate cost: faster language-model execution alone does not remove it, and it can dominate the time to the first token at high resolution or for long videos [1, 2]. ACTIV removes all of it. The claim in the paper is this comparison (lines 28–30): ACTIV is 22–26% faster than SEASON, and it is also more accurate (Table 2). The new table gives 20–22%.
  5. Hyperparameters (Question 5). ACTIV has three strengths: β_T, β_S and α. Both LLaVA models improve at all four tested values of α (new table).
  6. Sampling (Question 6). ACTIV does not sample tokens. At each step, it selects the highest-scoring token (lines 269–270). Thus, there is no seed noise to average.

To be clear on items 2, 3 and 6: VEM does not use first-token scoring or attention maps, and ACTIV does not use multinomial sampling. We will make these three statements more explicit in the text. We hope that this removes the related soundness concerns.

On the contribution relative to SEASON: our response to Reviewer xQ22 gives a controlled comparison on LLaVA-Video-7B. ACTIV's negatives add 1.7–2.0 points and ACTIV's weights add 0.4–0.6 points, in addition to the lower cost.

Detailed answers

1. Section 2 and Figure 2 (Weakness 1, Question 3). Our claim is about the percentage of answers with VEM ≤ 0. A positive mean or median does not contradict this claim. The markers in Figure 2 show medians.

Each cell gives the percentage of answers with VEM ≤ 0 for correct / incorrect answers:

BenchmarkQwen2.5-VLLLaVA-OneVisionLLaVA-Video
VidHalluc11.6 / 44.48.9 / 25.814.8 / 40.5
VideoHallucer24.0 / 26.68.6 / 18.016.7 / 23.5
EventHallusion29.9 / 42.211.1 / 52.214.5 / 50.0
TempCompass33.3 / 46.720.7 / 34.624.4 / 42.7
Video-MME24.3 / 25.49.8 / 17.69.0 / 18.1

The percentage is higher for incorrect answers in all 15 model–benchmark combinations. Each percentage uses its own group (correct or incorrect) as the denominator. VidHalluc pools BQA, MCQ and TSH. VideoHallucer uses individual questions for this diagnostic.

One can expect that random features make an answer less probable. The finding is more specific: 8.6–33.3% of correct answers receive no likelihood advantage from real features over matched random replacements. Accuracy alone cannot identify these answers. This finding is the motivation for ACTIV, and ACTIV decreases the share of answers with VEM ≤ 0 from 21.9% to 18.1% (lines 442–445).

In the revision, we will add this table and state in the caption of Figure 2 that the claim is about shares.

2. Exact implementation of S(y;Z) (Question 1). We score the full fixed answer (lines 147–148), without its end token:

S(y;Z)=1L∑t=1Llog⁡P(yt∣q,Z,y<t)S(y;Z)=\frac{1}{L}\sum_{t=1}^{L}\log P(y_t \mid q, Z, y_{<t})

Here, L is the number of answer tokens. For example, token log-probabilities −0.2, −0.4 and −0.3 give S = −0.3. VEM subtracts the mean score under three random replacements from the score with the real features. It does not score only the first token.

Detailed answers

3. Attention maps (Question 2). Lines 159–160 distinguish attention weights from changes in answer likelihood. They do not refer to an attention-map experiment or figure. We will reword this sentence.

4. Compute and latency (Weakness 2, Question 4). We agree that the language model has the larger share of the compute. This cost is common to all contrastive decoding methods: VCD, TCD, SEASON and ACTIV each run one decoder branch for each negative. Our efficiency claim is about a different cost: the additional vision encoding. There are three reasons.

  • It is a separate source of cost. Faster language-model execution alone does not remove the additional vision-encoder passes. Thus, a method that avoids these passes removes a cost that other speed-ups do not remove.
  • It is a large cost. In our measurements, vision encoding is 30–36% of SEASON's latency from 8 to 32 frames (the table below and Table 10), although it is only about 20% of its FLOPs. Recent studies show that vision encoding can dominate the time to the first token at high resolution or for long videos, although both vision encoding and language-model prefill become more expensive [1, 2]. The paper cites these studies as the motivation to reuse the encoded video (lines 454–456).
  • ACTIV removes all of it. ACTIV builds both negatives from one video encoding. SEASON encodes the video three times.

Thus, the claim in the paper is a comparison with SEASON (lines 28–30), not with the unmodified model.

The table gives the requested measurements. It uses LLaVA-Video-7B, an eight-frame cap and one RTX A6000. Each stage cell gives seconds / TFLOPs for one answer.

TaskMethodVisionDecoder prefillDecoder generationTotal (s)
EventHallusion (short answer)Baseline0.146 / 5.330.294 / 23.000.056 / 0.030.497
EventHallusion (short answer)SEASON0.443 / 16.000.834 / 69.000.071 / 0.091.347
EventHallusion (short answer)ACTIV0.147 / 5.330.841 / 69.000.068 / 0.091.056
VidHalluc STH (open-ended)Baseline0.135 / 5.330.243 / 19.710.219 / 0.120.596
VidHalluc STH (open-ended)SEASON0.414 / 16.000.702 / 59.140.271 / 0.351.384
VidHalluc STH (open-ended)ACTIV0.136 / 5.330.701 / 59.140.260 / 0.351.102

The table shows three results:

  • ACTIV adds no vision cost. Its vision cost is equal to the baseline's (0.147 s against 0.146 s, and 5.33 TFLOPs for both). SEASON's vision cost is three times larger (0.443 s and 16.00 TFLOPs).
  • ACTIV and SEASON have the same decoder cost. Each method runs three decoder branches.
  • ACTIV is faster than SEASON for both answer types. Its total latency is 21.6% lower on EventHallusion and 20.4% lower on VidHalluc STH. This agrees with the approximately 22–26% in the paper (lines 28–29). ACTIV is also more accurate (Table 2).

Measurement details. EventHallusion has 409 yes/no questions. STH has 445 questions that require scene descriptions. Each cell is the median over all questions. The total is the median of the per-question sums, so the stage medians do not add up exactly to the total. Times exclude video loading. FLOPs count matrix multiplications, convolutions and attention products, with two FLOPs for each multiply-add. We checked these counts against PyTorch's FLOP counter during real runs of the three methods on 24 questions, with the same rule for causal attention. The totals differ by less than 0.01%.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised paper uploaded

Dear Reviewers and Area Chair,

We have uploaded a revised version of the paper based on your concerns and feedback, with all changes shown in blue. The revision only adds the clarifications and results that you asked for in your reviews, and we hope that it makes it easy to see how each of your concerns was addressed.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.