acceptodds
Withdrawn from ICLR 2027

A Systematic Evaluation of Positional Bias in Multi-Video Summarization with MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) are increasingly used for video understanding, yet their reliability under multi-video inputs remains poorly understood. We study positional bias in multi-video summarization, where the quality of a per-video summary can change with the video's input slot even when the underlying content is unchanged. We construct a benchmark from ActivityNet and News videos, covering Cooking, Domestic, Leisure, and News settings with two- and four-video inputs. We evaluate nine open-source and proprietary MLLMs and measure position effects with three complementary metrics: Coverage, Directional Positional Bias (DPB), and Middle-Edge Gap (MEG). Our results show that positional effects are domain- and model-dependent: signed directional bias can be small even when middle positions underperform, and increasing visual or generation budget does not uniformly remove the imbalance. We further analyze prompt-level mitigation methods. Together, the results show that multi-video summarization remains sensitive to input protocol and position, motivating more robust order-invariant multimodal systems.

Primary area:
datasets and benchmarks
Keywords:
benchmarking, evaluation methodologies, evaluation, metrics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QrYkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates positional bias in multi-video summarization using multimodal large language models. It introduces an evaluation protocol with cyclic video orderings and three metrics, Coverage, Directional Positional Bias, and Middle-Edge Gap, and evaluates nine models across four settings. Results reveal model- and domain-dependent positional effects, including middle-position weaknesses that are not consistently eliminated by larger input budgets or prompt-level interventions.

Critical strengths
  • Underexplored research problem.
  • Good cyclic-order evaluation protocol.
  • Evaluation across nine MLLMs and four settings.
  • Useful statistical analyses and diagnostic experiments.
Critical weaknesses
  • The measured positional effects are often small relative to sampling variability and evaluation uncertainty, weakening some model-specific conclusions.

  • The benchmark has limited input scale, and several experimental factors are insufficiently controlled.

  • The evaluation methodology and released code contain inconsistencies, while incomplete reproduction materials and limited mitigation experiments weaken the reliability and practical value of the findings.

Questions for authors
  1. Many of the reported positional effects are small relative to the uncertainty of the evaluation protocol. Table 4 reports a pooled MEG of -0.944%, while Section 4.4 reports an average cyclic-order estimation error of 1.21% relative to full-permutation evaluation. Table 1 also presents numerous model- and domain-specific conclusions based on only 27 video groups per configuration, without per-cell confidence intervals. Could the authors report group-level confidence intervals for DPB and MEG in Table 1 and identify which positional effects remain statistically distinguishable from zero? It would also help to quantify the variability introduced by different cyclic orderings and repeated generations.

  2. Some results appear inconsistent across the main paper and appendix. For MiniCPM-o-4.5, Table 1 reports MEG values of -0.51%, 1.92%, and 0.21% for Cooking-4, Domestic-4, and Leisure-4 (Short), whereas the Base column of Table 11 reports -2.03%, 0.86%, and -0.85%, respectively. The Leisure result even changes sign. For Qwen3-VL-8B, the MEG values agree, but the DPB values differ between the two tables. Are these results obtained from different inference runs or aggregation procedures? Please clarify the experimental configurations, explain these discrepancies, and report the variability across repeated runs.

  3. The evaluation protocol introduces several potential confounding factors. Cyclic ordering balances positional exposure but also changes neighboring-video context, while longer videos use more sampled frames than shorter ones. Moreover, the fallback alignment in Section 3.2 uses ground-truth reference summaries, and invalid outputs assigned zero Coverage may distort positional metrics. Could the authors disentangle position from neighboring context and visual budget, and report results with invalid outputs excluded? Please also provide the frequency of fallback alignment by model and position and clarify how the held-out position contrasts in Table 3 are computed.

  4. The benchmark scope and evaluation reliability remain limited. The experiments consider at most four videos of up to two minutes, with only 27 groups per domain-duration configuration. Furthermore, ActivityNet and News use different Coverage estimators, and the GPT-5.1 judge is validated on 245 summaries without reporting agreement by model, domain, or position. Could the authors extend the evaluation to longer multi-video contexts and provide more detailed judge validation? Clarification of the inference configurations for proprietary models would also improve reproducibility.

  5. The mitigation and novelty analyses require stronger experimental support. The evaluated interventions are limited to prompt-level variants on a small subset of models, and Table 6 shows that single-target prompting has opposite effects for two-video and four-video inputs. Could the authors compare against order-aggregation methods and existing training-free positional-bias mitigation approaches, such as SoFA (CVPR 2025)? The paper should also better distinguish its contributions from prior studies of positional bias in summarization, including "On Context Utilization in Summarization with Large Language Models" (ACL 2024) and "Summary of a Haystack" (EMNLP 2024).

[1] Tian, Xinyu, et al. "Identifying and Mitigating Position Bias of Multi-image Vision-Language Models." CVPR. 2025.

[2] Ravaut, Mathieu, et al. "On context utilization in summarization with large language models." ACL. 2024.

[3] Laban, Philippe, et al. "Summary of a haystack: A challenge to long-context llms and rag systems." EMNLP. 2024.

  1. The released code contains several inconsistencies with the evaluation protocol and raises reproducibility concerns. Section 3.3.2 defines DPB as the average of per-instance scores, whereas "scripts/aggregate_position_metrics.py" first averages Coverage across positions and then computes DPB. This difference may explain the inconsistent DPB values between Tables 1 and 11. Furthermore, "scripts/compute_position_metrics.py" groups records without distinguishing cyclic orders, potentially producing incorrect coverage vectors. The reference-based fallback alignment described in Section 3.2 is absent from the released parser, which instead assigns zero Coverage to missing summary slots. The GPT-5.1 judge prompt, output schema, and score normalization also differ from Appendix A.2. More importantly, the repository does not include the raw model generations, judge decisions, or analysis scripts needed to reproduce the main results, and the released manifests contain 676 distinct video IDs rather than the 708 reported in Section 3.1. Could the authors reconcile these discrepancies, clarify which implementations produced the reported results, and release the complete evaluation pipeline and intermediate outputs necessary for reproducibility?

  2. Minor: Several presentation and reference issues should be corrected. Section 4.3 discusses Cooking and Domestic for InternVL3.5-8B, whereas Figure 3 presents Domestic and News. The description of the single-target labeling experiment is also unclear about the indexing convention used across tables. In the references, QuestEval is incorrectly attributed to Rebuffel et al. (2021), which corresponds to Data-QuestEval rather than the summarization metric by Scialom et al. (EMNLP 2021). Some references also retain arXiv versions despite subsequent formal publication.

[4] Scialom, Thomas, et al. "QuestEval: Summarization Asks for Fact-based Evaluation." EMNLP. 2021.

Reviewer_Ash5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the problem of positional bias in multi-video summarization across MLLMs through the lens of positional bias. The authors create a benchmark using videos from Activity-Net and News Video dataset across four scenario domains and an associated evaluation protocol based on pairwise and listwise inputs through cyclic permutation orderings. For evaluating order dependent performance shifts, the paper introduces coverage metric across two diagnostic dimensions: Directional positional bias and Middle-Edge gap. Based on experiments associated with open-source MLLMs, the study reveals order sensitivity based on severe middle-slot information loss and cross-video detail leakage.

Critical strengths
  • First systematic study targeting positional bias in multi-video summarization in terms of generating aligned per-video summary descriptions.
  • Introduction of novel metrics like MEG, DPB for isolating middle position weakness that standard linear metrics conceal.
  • Balanced cyclic permutation design that places every video in input slot position.
  • Extensive evaluation involving nine open-source and proprietary MLLMs across four distinct domains with additional ablations across frame counts, image resolutions and prompt placement.
Critical weaknesses
  • The problem design is artificial since it tries to evaluate positional bias when summaries are generated for multiple videos as input. There is a mismatch with the real world deployment pipeline where video summarization is run parallelly for multiple videos instead of a single input.
  • The rationale for middle position bias is not well explained. It is not clear whether the middle position weakness is due to perception failure of MLLMs or autoregressive output-allocation bottleneck.
  • While the proposed metrics might be novel, the computation technique doesn’t add any methodological innovation in terms of video evaluation. It relies on direct averaging of coverage scores across input slots rather than probabilistic or learned evaluation models for scoring.
Questions for authors
  • Currently the dataset has 708 videos covering limited domains from Activity net and News sources. Have the authors considered expanding the data to larger sample size?
  • Have the authors considered additional prompt level interventions beyond boundary separators or equal attention instructions ?
  • In the current multi-video setup, MLLMs suffer from cross-video detail leakage causing actions and objects from adjacent shots to contaminate unrelated video summaries. In the current multi-video setup, how can this cross-video detail leakage be tackled ?
Reviewer_cVuaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigated the positional bias issue in multi-video summarization: how the order of input videos will affect summarization results using MLLM. Along this motivation, the authors constructed a benchmark dataset from several existing video datasets and formulated the evaluation with two bias settings: pairwise of two videos and cycling of 4 videos. The evaluation was conducted with 9 MLLMs against three metrics. Various discussions were conducted in terms of evaluation results.

Critical strengths

1)Identifying an interesting perspective for video summarization. 2)Constructed a dataset for evaluation 3)Designed evaluation metrics

Critical weaknesses
  1. The scientific objective is not clear. For example, why does it matter for investigating positional bias for multi-video summarization, not just because it is valid in NLP. In particular, the NLP literature generally referred to the position of sentences in a document, which is different with the position of one video in a set of videos.

  2. It is not clear how the videos are selected and paired or grouped. I am not convinced 1-2 minutes is long enough, which makes the investigation less convincing and worthwhile. I am not sure how topic is chosen. It may be more suitable to group videos of an events or relevant theme/tasks.

  3. It lacks clear justification on the three metrics designed.

  4. Presentation could be further improved. For example, In section 2.2, it would be better to focus on video summarization, not necessarily on text; Statistics of the dataset needs to be provided. It seems that the results shown in Figure 3 and Figure 4 indicate marginal impact of the position order. It would have been better to provide qualitative samples for case study.

Questions for authors
  1. Stronger justification on the motivation of the work.

  2. Stronger justification of the evaluation metrics.

  3. More details of the dataset. The anonymous website contains various json files, which do not help gain a good understanding of the dataset easily.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.