acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking, Diagnosing, and Mitigating Temporal Positional Bias in Video Understanding

Abstract

Multimodal large language models (MLLMs) are sensitive to where relevant evidence appears in a video. Existing evaluations often manipulate evidence position by concatenating unrelated clips, which may introduce clip-matching shortcuts and disrupt the contextual relationships needed for reasoning. We investigate temporal positional bias through a unified study of benchmarking, diagnosis, and mitigation. We introduce TempPosBench, which shifts observation windows over continuous videos to vary evidence position while preserving temporal continuity. It comprises 1,322 question–answer instances across 13 tasks spanning perception, reasoning, and streaming awareness. Evaluating seven MLLMs reveals model- and task-specific positional preferences, with temporal reasoning exhibiting the strongest sensitivity. We further examine attention concentrated toward early or late video frames. Targeted prefix calibration and visual-key rotary position embedding (RoPE) compression provide evidence that attention sinks and positional-encoding-induced recency preferences can contribute to positional bias. For mitigation, we extend the benchmark construction to produce positionally balanced training candidates, then select positions that emphasize each model’s weaknesses. Fine-tuning on continuous video contexts improves both accuracy and positional robustness on spatial and temporal reasoning compared with concatenated clips. Under comparable training budgets, model-specific selection can also outperform training on all available positions. Fine-tuning additionally yields flatter frame-attention distributions, connecting the training-based mitigation to our attention-level diagnosis. Together, these findings highlight temporal continuity and model-specific positional supervision as complementary ingredients for mitigating temporal positional bias in video understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
temporal positional bias, temporal continuity, video understanding, multimodal large language models, spatiotemporal reasoning, benchmarking, supervised fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KsyhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates positional bias in video understanding by multimodal large language models (MLLMs). The authors point out, and further validate through attention analysis, that existing concatenation-based evaluation methods may introduce semantic shortcuts, thereby coupling the measurement of positional bias with the way contextual inputs are constructed. Based on this observation, the authors conduct a relatively systematic study from three perspectives. First, they propose a temporally continuous benchmark to evaluate positional bias based on model performance. Second, they diagnose the potential causes of positional bias by analyzing attention distributions and identify two positional-bias attention profiles. Finally, they explore bias mitigation through positionally balanced training data, and the experimental results demonstrate promising improvements.

Critical strengths
  1. The temporally continuous benchmark proposed in this paper represents a meaningful improvement over previous concatenation-based evaluation methods. This design is more representative of real-world video understanding scenarios, and the experiments demonstrate its advantages over the concatenation-based variants.
  2. The paper presents a relatively systematic investigation, progressing from observing positional-bias phenomena at the performance level, to analyzing the underlying attention mechanisms, and finally to mitigating the bias through fine-tuning with specially constructed training data. The proposed analysis and mitigation are supported by relatively comprehensive experimental evaluations.
Critical weaknesses

The paper requires substantial improvement in terms of presentation and writing. Many parts of the paper lack appropriate citations, making it difficult for readers to trace the sources of the claims or methods. For example, the entire Introduction and Approach sections are largely missing citations. The main paper omits many necessary details. For instance, it does not provide concrete examples of the evaluation data, including the complete video and question inputs. It also lacks essential details about the training data, such as the dataset size and data sources. In addition, it is unclear which model's fine-tuning results are presented in Table 3 and Figure 5.

Questions for authors
  1. Are the target clips and the source videos sampled from the same video? Since the target clip is the same across different evaluation positions, is there also a discontinuity between the target clip and the surrounding background video?
  2. There may be information leakage between the target clip and the background video. If a continuous background segment contains information relevant to the question associated with the target clip, it could potentially lead to inaccurate evaluation results. How do the authors address this issue?
  3. The entire benchmark contains only 1,322 samples, and some experiments are conducted on only one of the three tasks. Given the relatively small amount of evaluation data, how reliable are the conclusions drawn from these experiments?
  4. Have the models fine-tuned for positional-bias mitigation also been evaluated on general-purpose video understanding benchmarks? Intuitively, successful mitigation of positional bias should also benefit general video understanding tasks.
  5. As shown in Figure 1(a), some models may exhibit different patterns of positional bias across different tasks. For example, this appears to be the case for InternVL3.5. Have the authors investigated this phenomenon further?
Reviewer_UCLDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work proposes one position-controlled yet continuity-preserving benchmark named as TempPosBench which varies the temporal position of the target clip while retaining the original temporal continuity. This dataset comprises 1,322 question–answer instances across 13 tasks spanning perception, reasoning, and streaming awareness. Based on this dataset, this work found distinct head-heavy and tail-heavy patterns across MLLMs with sharp bias. This work further shows show effective mitigation of temporal positional bias in video reasoning requires both model-specific supervision and temporal continuity in the training videos.

Critical strengths
  • The motivation is clear and practically relevant. Continuous video contexts can better preserve the intended reasoning demands during positional evaluation while most previous works concatenates semantically unrelated clips which disrupts temporal continuity.
  • Diagnosis on temporal reasoning probe two interesting potential mechanisms of temporal positional bias: early-token attention sinks and positional-encoding effects via visual-key RoPE compression.
  • The comparison of continuous vs. concatenated training data built from the same QA pairs is a useful controlled experiment, and the reasoning-vs-perception split in Figure 5 is informative.
Critical weaknesses
  • Comparision with the SOTA model like GPT6/Gemini/Claude or larger open-source model like Qwen3-VL-32/72 is missing. The author need to showthe raised question is not addressed by advanced training or larger weights.
  • The prefix calibration is under-specified. A_ref comes from a pass with extra black frames, so it is unclear how A_cal = (1−λ)A_orig + λA_ref is aligned across different sequence lengths.
  • Inconsistent baselines across tables. For GLM-4.6V-Flash on Temporal Reasoning, the vanilla numbers are 49.18/29.68 (Table 1), 34.24/18.19 (Table 2), and 35.25/21.61 (Table 7). On the 20% tuning subset, vanilla is 24.76/28.85 (Table 8) and 25.00/28.57 (Table 9). Why there is such one large performance gap between different Tables.
  • The mitigation results may be inflated by in-distribution learning and possible leakage.
  • Reference settings in Table 4 may reveal the experimental noise given the current range level shown in the paper.
Questions for authors

Mainly refer to weakness but some additional question are listed under:

  • Please provide report benchmark results on the same QAs under concatenation-based construction, to show directly how the measured bias differs?
Reviewer_djzeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TempPosBench to test how video LLMs perform when the same evidence appears at different positions in a continuous video. It examines how models attend to video frames and proposes training on shifted windows where each model performs poorly. Experiments on several models show position-dependent performance while finetuning on shifted windows improves results on this benchmark compared with training on concatenated clips.

Critical strengths
  1. A useful benchmark based on continuous video. Shifting windows preserves natural event order and avoids artificial joins between unrelated clips. Coverage of multiple tasks makes the benchmark useful for studying position effects across different video-understanding abilities.

  2. Some investigation beyond performance scores is provided, although similar to prior works. The paper tests whether changes to attention and position encoding affect accuracy and position sensitivity. These additional interventions provide evidence about possible causes of the bias, making the study relatively more informative than simply documenting where models fail.

Critical weaknesses
  1. The training comparison is confounded by matching the evaluation format. Both training methods are tested on shifted-window videos, which directly match the continuous-window training construction. Without testing both models on concatenated inputs or an independent benchmark, the results do not establish a broader advantage beyond matching training and testing conditions.

  2. The experiment is insufficient. (1) The baseline evidence is incomplete. Raw-video fine-tuning reduces accuracy from 38.91 to 20.59, but this severe failure is not adequately explained. It is difficult to use that control to establish the need for positional balancing. (2) Comparisons with existing position-bias mitigation methods are also missing. (3) Models are evaluated on the TempPosBench, which share the same distribution with the training data format. To supprot the claim on reducing positional bias to improve video understanding, evaluation on more general long video understanding benchmarks are required.

  3. The benchmark does not isolate position from other input changes. Moving the window changes surrounding events and may change which evidence frames are sampled. Performance differences could therefore reflect different context or evidence visibility, rather than positional bias alone.

Questions for authors

Please refer to Critical Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.