- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates positional bias in video understanding by multimodal large language models (MLLMs). The authors point out, and further validate through attention analysis, that existing concatenation-based evaluation methods may introduce semantic shortcuts, thereby coupling the measurement of positional bias with the way contextual inputs are constructed. Based on this observation, the authors conduct a relatively systematic study from three perspectives. First, they propose a temporally continuous benchmark to evaluate positional bias based on model performance. Second, they diagnose the potential causes of positional bias by analyzing attention distributions and identify two positional-bias attention profiles. Finally, they explore bias mitigation through positionally balanced training data, and the experimental results demonstrate promising improvements.
Critical strengths
- The temporally continuous benchmark proposed in this paper represents a meaningful improvement over previous concatenation-based evaluation methods. This design is more representative of real-world video understanding scenarios, and the experiments demonstrate its advantages over the concatenation-based variants.
- The paper presents a relatively systematic investigation, progressing from observing positional-bias phenomena at the performance level, to analyzing the underlying attention mechanisms, and finally to mitigating the bias through fine-tuning with specially constructed training data. The proposed analysis and mitigation are supported by relatively comprehensive experimental evaluations.
Critical weaknesses
The paper requires substantial improvement in terms of presentation and writing. Many parts of the paper lack appropriate citations, making it difficult for readers to trace the sources of the claims or methods. For example, the entire Introduction and Approach sections are largely missing citations. The main paper omits many necessary details. For instance, it does not provide concrete examples of the evaluation data, including the complete video and question inputs. It also lacks essential details about the training data, such as the dataset size and data sources. In addition, it is unclear which model's fine-tuning results are presented in Table 3 and Figure 5.
Questions for authors
- Are the target clips and the source videos sampled from the same video? Since the target clip is the same across different evaluation positions, is there also a discontinuity between the target clip and the surrounding background video?
- There may be information leakage between the target clip and the background video. If a continuous background segment contains information relevant to the question associated with the target clip, it could potentially lead to inaccurate evaluation results. How do the authors address this issue?
- The entire benchmark contains only 1,322 samples, and some experiments are conducted on only one of the three tasks. Given the relatively small amount of evaluation data, how reliable are the conclusions drawn from these experiments?
- Have the models fine-tuned for positional-bias mitigation also been evaluated on general-purpose video understanding benchmarks? Intuitively, successful mitigation of positional bias should also benefit general video understanding tasks.
- As shown in Figure 1(a), some models may exhibit different patterns of positional bias across different tasks. For example, this appears to be the case for InternVL3.5. Have the authors investigated this phenomenon further?