acceptodds
Withdrawn from ICLR 2027

Perception or Prior? Motion-Blind Controls for Video Hallucination Mitigators

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Hallucination mitigators for video LLMs are judged by benchmark deltas, yet many benchmark questions are answerable without temporal information, and preference optimization can raise scores by shifting language priors alone. How much of any reported gain requires motion remains unmeasured. We provide the measurement. Five DPO arms share one base model, one public preference dataset, and one training recipe, differing only in the visual signal seen during training: the original video, uniformly permuted frames, black frames, no visual tokens, or order-preserving jitter. Arm equivalence is established by recipe digests and per-run consumption hashes. A preregistered linear contrast adjudicates the threshold claim without division; recovered-gain fractions are reported behind a denominator gate with Fieller intervals, conditioned on a motion-necessary stratum identified by five held-out model families. We report a rank order of bounds, not an additive decomposition. Two evaluation-side findings stand on their own. Freezing every frame to a repeated median costs Qwen2.5-VL-7B 13.3 VideoHallucer pair-accuracy points; black frames cost 43.1—the blind control overstates the video's contribution roughly threefold. Separately, switching answer extraction from generative parsing to per-option log-likelihood moves Video-LLaVA-7B 15 points (9.5 to 24.5), bracketing the published 17.8 and exceeding the margin between most mitigators and their baselines. Neither benchmark specifies which extraction rule was used. Checkpoints, the motion-necessary stratum, and all intervention code are released.

Primary area:
datasets and benchmarks
Keywords:
video language models, hallucination, direct preference optimization, evaluation, control experiments, temporal understanding, benchmark validity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZVfoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether the performance improvements achieved by methods for mitigating hallucinations in large video models truly stem from enhanced temporal awareness, or whether they can be obtained through other information sources and evaluation settings. The authors analyze the impact of different visual occlusion benchmarks and answer scoring rules on evaluation results, and design a DPO control experiment that alters only the training video input to analyze the source of the performance gains. The experiments show that different evaluation settings significantly affect performance results.

Critical strengths

The research subjects and starting point of this study are well-chosen.

Critical weaknesses
  1. The writing in this paper could be significantly improved. It is evident that the current manuscript strives to present the authors’ work, but the current writing is obscure and difficult to understand. It reads more like an initial draft generated directly from code by a LLM, with little or no human review, as the paper contains numerous elements that defy common sense in human writing: for example, the abstract’s phrasing “with matching machine-verified and every arm evaluated on untouched video,” the use of subheadings to introduce every paragraph, and the presentation of content in Section 5. Therefore, I consider this to be a significant weakness.

  2. The writing and experimental sections appear to contain many inconsistencies or contradictions. Please see the “Questions” section for details.

Questions for authors
  1. In Section 2.5, the authors note that the “whole-set result” may arise naturally from arithmetic because many samples do not require motion at all, and that the truly informative question is what happens with motion-necessary items. However, this MNS conditional analysis was not performed. Therefore, I would like the authors to further explain: in the absence of results for MNS, to what extent should the core interpretation in the Abstract and Conclusion be understood?

  2. In Section 2.1, A2 performs a uniform permutation of 32 frames. Based on this, the authors state that A2’s training video carried no motion. However, the paper also explicitly acknowledges that A2 preserves all frame-, object-, scene-, and set-level cues. Although the shuffled frame set lacks temporal order, it may still contain elements such as the range of object displacement and state changes across multiple frames. Therefore, the authors need to clarify the boundaries of the motion-blind definition.

Reviewer_LaDTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that benchmark deltas reported by hallucination-mitigation methods for video LLMs do not establish what the model learned to use. It first presents two evaluation-side observations. On VideoHallucer, replacing frames with black costs Qwen2.5-VL-7B 43.1 points of pair accuracy while repeating a single static frame costs 13.3, so the usual black-frame control attributes the static scene to the video. Separately, changing only how an answer is read off (generative parsing vs. per-option log-likelihood) moves Video-LLaVA-7B from 9.5 to 24.5, with the published 17.8 between the two rules. The main contribution is a training-side control: five LoRA-DPO runs from Qwen2.5-VL-7B on LLaVA-Hound-DPO, identical except for the video seen during training (full, frame-shuffled, black, absent, order-preserving jitter), all evaluated on untouched video.

Critical strengths
  1. The matched-recipe design is well constructed. The five arms share the same base model, data, recipe, data order and text (verified by hashes) and differ only in the training video. This directly tests whether a DPO gain needs motion in the training data, which test-time perturbations cannot do. The preregistration and automated matching checks are careful.

  2. The two measurement findings are concrete and useful. Benchmarks in this area do not state which decision rule or which blinded control they use, and the paper shows that each choice moves scores by more than most reported gains.

Critical weaknesses
  1. The introduction presents findings before motivating the problem: the two evaluation-side results appear on page 1, and the motivating gap only in the “What none of it supplies” paragraph. Notation (Δ, RGF, ISP, MNS, arm labels A1–A5) is used in the abstract and introduction but defined only in §2 and Figure 1, so claims such as “the motion-blind arm recovers 106% of the standard arm’s gain” cannot be followed until page 3; the “matched-recipe control grid” paragraph has the same problem. The contributions are never listed — the paper says at length what it does not claim, but not concisely what it does. Several passages read as working notes rather than exposition (e.g. “Three quantities carry the audit”).

  2. The main result is supported on one benchmark only. The main claim, that a DPO gain does not require motion in the training video holds on TempCompass alone, on one base model and one preference set. On VideoHallucer the standard arm gains only +1.7 and the contrast is not significant; on EventHallusion and DyBench the recipe reduces accuracy, so there is no gain to decompose. The analysis the authors themselves call load-bearing, the motion-necessary stratum (Sec 2.5, Sec 4.3), was not run, nor were the training-free leg, the second base model, or the artifact-floor correction. DyBench's A2 and A4 cells are missing and all arms stopped at 350/396 steps.

  3. The reported results are internally inconsistent. Sec 5 states that no benchmark clears the +4.0 denominator gate and that three preregistered criteria failed; the abstract and Sec4.2 state that TempCompass clears it and two failed. Table 1 reports least-favourable-seed contrasts while Appendix B.2 states the per-seed analysis was not run, and its values do not follow from Table 2 (e.g. EventHallusion: 2(−2.2) − (−1.7) = −2.7, reported as −3.4). A3 and A5 have one seed and A4 two, so the seed rule cannot apply to them.

  4. No dataset or benchmark is proposed. The synthetic corpus (Appendix D.5) and the motion-necessary stratum are released but produce no result, so for this primary area the submission offers artifacts without validation.

Questions for authors
  1. Sec 4.2 and the abstract say TempCompass passes the denominator gate and two preregistered criteria failed; Sec 5 says no benchmark passes the gate and three failed. Which is correct?
  2. Table 1 reports least-favourable-seed contrasts, but Appendix B.2 says the per-seed analysis was not run. How were these values obtained? Please report the contrast for each seed.
  3. The contrasts in Table 1 do not follow from the Δ values in Table 2 (e.g. EventHallusion gives −2.7, not −3.4). Do the two tables use different seeds? Please clarify.
  4. Seed is treated as a sensitivity factor, yet A3 and A5 were trained with one seed and A4 with two. Why?
  5. Can the motion-necessary stratum analysis be run on the available held-out models as a sensitivity check? The paper itself says that without it the whole-set result is "close to arithmetically implied."
  6. The submission is in the datasets and benchmarks area. What is the dataset or benchmark contribution?
Reviewer_U58EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether gains from video hallucination mitigation depend on learning motion information, or whether they can be obtained without temporally ordered training video. It introduces matched training controls. It also examines how two evaluation choices affect reported benchmark results: the type of visual ablation and the answer-scoring method.

Critical strengths

Matched training controls. Five variants share the base model, preference dataset and optimisation procedure, differing only in the visual information seen during training, with machine-verified configuration and data-matching checks and a preregistered statistical protocol. This is consequential because it isolates what a gain requires from the training procedure itself, which test-time perturbations cannot do, and the design is cheap enough for others to adopt.

Evaluation-sensitivity findings. The black-frame vs frozen-frame controls differ sharply on VideoHallucer and the answer-scoring method alone moves a score by 15 points. Both effects exceed typical reported gains, so they matter for how the field reads benchmark improvements, independent of the paper’s main claim.

The authors are open about unsuccessful reproductions, incomplete experiments and limitations.

Critical weaknesses

The central claim rests on a narrow base. The key result is from TempCompass, where shuffled-frame training recovers 106% of the standard gain. However, standard training yields meaningful improvement on only one of four benchmarks. The motion-necessary subset analysis was not completed, even though the authors themselves present it as the analysis that separates genuine temporal reasoning from shortcuts. TempCompass questions being temporal does not mean each one requires temporal information, so the gain may still come from static cues or dataset-specific shortcuts. As it stands, the evidence supports a narrower claim: temporally ordered training frames are not necessary to reproduce the observed gain on one benchmark under the tested recipe.

Generalisability is limited. There is one model (Qwen2.5-VL-7B), one preference dataset and one LoRA-DPO recipe. The planned second model and the training-free mitigation experiments were not completed.

The shuffled-frame control should be interpreted more precisely. Shuffling removes reliable frame order, but it does not remove temporal cues within individual frames or in the preference texts. All models are also evaluated on intact video and may draw on temporal capabilities from pretraining. The training/inference distinction is acknowledged, but what the control establishes should be stated more precisely.

Experimental validation is incomplete. Training stops before the planned number of steps, some benchmark evaluations are missing, and the number of seeds differs across conditions. The jitter control is much weaker than shuffling, which limits its ability to isolate augmentation effects.

Reporting is inconsistent. §5 says no benchmark passes the denominator gate and that the threshold test fails. Table 1 and §4.2 report a passing result on TempCompass. This needs reconciling, especially given the paper’s emphasis on transparent evaluation.

Questions for authors

Can you provide evidence on genuinely motion-dependent examples?

Precisely what does the shuffled-frame control establish, given temporal cues within frames, in the text, and from pretraining?

Which account of the preregistered outcomes is correct, §5 or §4.2/Table 1?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.