acceptodds
Withdrawn from ICLR 2027

TRAINING-FREE POSITIVE SPATIOTEMPORAL ENHANCEMENT FOR MITIGATING TEMPORAL HALLUCINATIONS IN VIDEO-LLMS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video Large Language Models (Video-LLMs) recognize objects and scenes reliably, yet they still misread event order, motion direction and object-state transitions, which produces *temporal hallucinations*. We refer to the change of the same object's state across frames as *object-centric motion*. Existing training-free approaches boost *per-frame* object-centric evidence to reduce hallucination. Yet, they do not explicitly boost *cross-frame* object-centric motions, the temporal relationships of the same object. Our empirical analysis suggests that this information is partially decodable from object-level visual representations yet, the final layer does not decode them in the final prediction. From this diagnosis, we propose Feature Enhance, a training-free scheme that strengthens object-centric visual evidence by emphasizing foreground regions and making inter-frame changes more explicit in the visual representations. It reinforces foreground regions that persist across frames and injects foreground-gated inter-frame feature residuals, leaving the model parameters untouched. Across 3 backbones, LLaVA-OneVision, Qwen2.5-VL and LLaVA-Video, Feature Enhance improves average accuracy over the corresponding unmodified model in all 6 backbone and benchmark combinations on VideoHallucer and EventHallusion, with gains of up to 4.67 and 4.00 percentage points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video Large Language Models, Temporal Hallucination, Training-Free Inference, Visual Evidence Enhancement, Inter-Frame Representation, Hallucination Mitigation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_UwsVOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Feature Enhance, a training-free method for mitigating temporal hallucinations in Video-LLMs. It emphasizes foreground visual tokens and adds inter-frame feature differences before the frozen language model. Experiments on VideoHallucer and EventHallusion across three backbones show consistent gains over the base models.

Critical strengths
  1. The method is training-free and simple.
  2. All methods are fairly compared with the same fixed frames and decoding settings.
Critical weaknesses

My concerns are mainly in the experiments.

  1. In Table 2, the gains mainly come from non-temporal categories (e.g., SDH up to +8.50), while TPH gains only 1.71–3.87. This does not align what this works aim to address.
  2. The reimplemented SEASON [2] is much weaker than reported in its original paper, in both accuracy (Table 2 vs. Table 1 of [2]) and latency.
  3. Figure 1 implies that SEASON fails on temporal hallucination, but SEASON outperforms Feature Enhance on EventHallusion MIX for all backbones (Table 2). Also, most of the VidHalluc TSH gain comes from adding temporal-negative contrastive decoding, not from Feature Enhance itself (Table 14).
  4. No general video benchmark (e.g., Video-MME [1]) is evaluated.
Questions for authors

Please refer to the weakness part.

Reviewer_SqHrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper begins with three observations. First, they claim that moving foreground contributes to temporal errors with bad-step ratio if foreground attention is over background attention. Object recognition is independent of temporal changes, and motion direction is encoded in the visual representation. The proposed method first uses segmentation models to obtain a foreground map, and spatial terms are weighted and summed by object occupancy. Such object motions are encoded as temporal terms. The experiments include three models on three benchmarks, compared against three training-free methods, and ablation study of each component.

Critical strengths
  1. The proposed method directly enhances the positive visual token representations from three observations
  2. Incorporating motion information for temporal understanding is interesting
Critical weaknesses
  1. There are a lot of rooms to improve in terms of writing. Results of F2, F3 are verbally described without any tables or graphs, too many notations are defined, etc. Therefore the writing is not reader-friendly.
  2. The method is overall too heuristic, does not
Questions for authors
  1. How can we obtain "pixel-level foreground mask over each token’s pixel footprint"? The rho is not well defined unless authors provide the exact algorithm.
  2. Previous literature on visual token attention analysis have often found that the visual tokens often fail to attend to wrong tokens but still be able to predict answers correctly [1]. Given such observations, does "(F1) the model does not attend to the moving foreground" still be considered a failure? I believe this is a human-centric generalization.
  3. Is there any supporting experimental results that explain foreground objects are moving due to large RGB pixel value changes? (line 202)
  4. How can having more attention on the background over the foreground be concluded as F1?
  5. Is it okay to determine tau threshold post-hoc? It looks like cherrypicked.
  6. Are accuracies on two benchmarks consistent if we switch two models? "Quantitatively, LLaVA-OV-7B achieves 88.5-93.0% accuracy on non-temporal basic questions of VideoHallucer, compared with 54.0% on temporal ones. Qwen2.5-VL-7B exhibits temporal- blindness rates of 59.5% (P1) and 52.7% (P2) under paired reversal and option-order checks on TempCompass"
  7. Is there any reason to use only the attentions in the final layer?

[1] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. ECCV 2024.

Reviewer_2YbnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Feature Enhance to mitigate the temporal hallucinations in Video-LLMs. Feature Enhance is a training-free method that combines BiRefNet foreground weighting, a clip-level persistence prior, and foreground-gated inter-frame feature residuals together. The paper applies Feature Enhance to three 7B backbones and evaluates them on VideoHallucinate, EventHallucinate, and MotionBench. The experimental results show consistent improvements on these benchmarks compared with the raw baselines.

Critical strengths
  1. Feature Enhance is a training-free method and can plug-and-play to many Video-LLMs. The design to combine spatial and temporal cues is reasonable and different than existing methods.
  2. The paper provides ablations on the main components. For example, Table 4 compares foreground weighting, persistence, and other variants across four tasks, Table 14 also compares Feature Enhance alone with its combination with temporal-negative decoding.
Critical weaknesses
  1. According to the official benchmark papers, VideoHallucer TPH contains 176 qa pairs, and the ENTIRE, MISLEADING, and MIX subsets of EventHallusion contain 114, 102, and 193 questions. But the number of qa that evaluate Feature Enhance seems different from the baseline methods. In Table 2, most of the scores for the baselines, TCD, and DINO-HEAL are consistent with these sample sizes, but many scores for Feature Enhance do not correspond to an integer number of correct responses when calculated against these same denominators. For example, the 49.32% for LLaVA-OV on TPH cannot be derived from any integer count of correct predictions where N∈[176,200]N \in [176, 200] (as Table 5 states), even accounting for rounding to two decimal places. The same issue also happens on EventHallusion. So I am concerned about the fairness of the evaluation presented in Table 2. If the baselines and Feature Enhance were evaluated using different dataset scales, it would undermine the significance of the results.
  2. Also, the distinction between the development set and the evaluation set is unclear. The paper states that the development data and evaluation data do not overlap. But Table 4 and Appendix D indicate that 100 TPH samples and 100 EventHallusion samples were used for parameter selection. As I posted in the first point, the evaluation for TPH seems to use the full set of 176 qa pairs. If these 100 TPH development samples are drawn from the same version, they should overlap with the full evaluation set. This is important because the parameters of Feature Enhance are from testing 109 different configurations, and the evaluation results on TPH will be unreliable if Feature Enhance is tuned on evaluation data.
  3. The temporal residual is an important new element in Feature Enhance, and I feel it makes Feature Enhance really different from previous methods. But the existing ablations do not isolate the contribution of the temporal residual. The ablations in Table 4 and Table 13 used a hyperparameter-tuning development set, and different configurations also varied coefficients. For example, the Temporal-only variant uses (β=0.6)(\beta=0.6), but the full method employs (β=0.8)(\beta=0.8). Thus, these differences reflect both changes in components and parameter adjustments.
  4. A minor weakness in presentation: Figure 1 is pretty hard to read now since there are some layout and formatting errors. It would be difficult for people who do not have a solid background in this topic.
Questions for authors
  1. Could you explain how the Table 2 scores were calculated? For example, the sample counts, correct counts and failure handling, especially for values such as 49.32%.

  2. Could you provide the development/evaluation sample IDs to verify that they are disjoint, and clarify if the 100 TPH development samples are questions or pairs?

  3. Could you provide component ablations on held-out data, keeping the remaining coefficients fixed when removing each component?

Reviewer_Q67TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Feature Enhance, an inference-time intervention for reducing temporal hallucinations in frozen Video-LLMs. Motivated by foreground-attention analyses and linear probes of motion-direction information, it modifies projected visual tokens using three components: BiRefNet-derived foreground weighting, a spatial persistence prior obtained by averaging foreground occupancy across frames, and foreground-gated, norm-matched differences between adjacent temporal steps. Its principal methodological distinction is injecting temporal-change signals directly into the positive visual representation without updating backbone parameters or adding autoregressive decoding branches. Experiments report improvements over the unmodified models across three backbones on VideoHallucer and EventHallusion, together with a smaller improvement on MotionBench. The approach is practical and conceptually simple, although the evidence more clearly establishes an effective feature intervention than the proposed explanation involving unused object-centric motion information.

Critical strengths
  1. Simple, broadly applicable intervention. Equations 5–8 specify a compact transformation that operates on projected visual tokens and preserves frozen backbone parameters. Combining foreground localization with explicit temporal differences is a plausible extension beyond spatial saliency reweighting.
  2. Consistent benchmark-average improvements. Table 2 shows gains over the unmodified backbone in all six backbone–benchmark combinations, reaching +4.67 percentage points on VideoHallucer and +4.00 on EventHallusion. Shared sampled frames and deterministic decoding provide useful controls, while MotionBench offers additional evidence beyond hallucination-specific benchmarks.
  3. Candid reporting of qualifications and tradeoffs. The authors acknowledge small diagnostic effect sizes, post hoc thresholds, the distinction between decodability and causal use, and limitations of fixed-position differencing. They also report substantial inference overhead and explicitly recognize settings in which SEASON performs better.
Critical weaknesses
  1. The diagnostic evidence does not sufficiently establish the proposed failure mechanism. Section 3.4 reports strong visual-token direction-probe accuracy but omits essential details about label construction, dataset sizes, source-level splitting, probe fitting, and controls for static appearance or question-derived cues. The decision-state probe’s 63/101 accuracy versus the model’s 55/101 is also inconclusive: the reported 33 versus 25 discordant outcomes yield a two-sided exact paired-test p≈0.36p\approx0.36. Source-disjoint probes with static-frame, shuffled-frame, and question-only controls would substantially strengthen the interpretation.
  2. The component ablations conflate component inclusion, coefficient selection, and development-set optimization. Tables 4 and 13 evaluate the same 400-example development set used to select among 109 configurations, while changing coefficients between variants—for example, temporal-only uses β=0.6\beta=0.6, whereas the full method uses β=0.8\beta=0.8. The full method’s 1.50-point advantage over temporal-only corresponds to six additional correct predictions on this set. Independently held-out, fixed-coefficient component-removal experiments are needed to establish complementarity.
Questions for authors
  1. How were the visual and decision-state probes constructed and split? Please specify label definitions, probe training and regularization, class balance, and whether original/reversed versions of a source video remain in the same split. Strong performance under source-disjoint, static-frame, and shuffled-frame controls would substantially increase my confidence that the probes measure temporal information rather than shortcuts.
  2. Do the component gains persist on held-out data with coefficients controlled? Please provide fixed-coefficient removal ablations and temporal-residual controls, with paired confidence intervals. A reliable advantage over temporal-only and matched perturbation baselines would strengthen the claimed necessity of the full design.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.