- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates positional bias in multi-video summarization using multimodal large language models. It introduces an evaluation protocol with cyclic video orderings and three metrics, Coverage, Directional Positional Bias, and Middle-Edge Gap, and evaluates nine models across four settings. Results reveal model- and domain-dependent positional effects, including middle-position weaknesses that are not consistently eliminated by larger input budgets or prompt-level interventions.
Critical strengths
- Underexplored research problem.
- Good cyclic-order evaluation protocol.
- Evaluation across nine MLLMs and four settings.
- Useful statistical analyses and diagnostic experiments.
Critical weaknesses
-
The measured positional effects are often small relative to sampling variability and evaluation uncertainty, weakening some model-specific conclusions.
-
The benchmark has limited input scale, and several experimental factors are insufficiently controlled.
-
The evaluation methodology and released code contain inconsistencies, while incomplete reproduction materials and limited mitigation experiments weaken the reliability and practical value of the findings.
Questions for authors
-
Many of the reported positional effects are small relative to the uncertainty of the evaluation protocol. Table 4 reports a pooled MEG of -0.944%, while Section 4.4 reports an average cyclic-order estimation error of 1.21% relative to full-permutation evaluation. Table 1 also presents numerous model- and domain-specific conclusions based on only 27 video groups per configuration, without per-cell confidence intervals. Could the authors report group-level confidence intervals for DPB and MEG in Table 1 and identify which positional effects remain statistically distinguishable from zero? It would also help to quantify the variability introduced by different cyclic orderings and repeated generations.
-
Some results appear inconsistent across the main paper and appendix. For MiniCPM-o-4.5, Table 1 reports MEG values of -0.51%, 1.92%, and 0.21% for Cooking-4, Domestic-4, and Leisure-4 (Short), whereas the Base column of Table 11 reports -2.03%, 0.86%, and -0.85%, respectively. The Leisure result even changes sign. For Qwen3-VL-8B, the MEG values agree, but the DPB values differ between the two tables. Are these results obtained from different inference runs or aggregation procedures? Please clarify the experimental configurations, explain these discrepancies, and report the variability across repeated runs.
-
The evaluation protocol introduces several potential confounding factors. Cyclic ordering balances positional exposure but also changes neighboring-video context, while longer videos use more sampled frames than shorter ones. Moreover, the fallback alignment in Section 3.2 uses ground-truth reference summaries, and invalid outputs assigned zero Coverage may distort positional metrics. Could the authors disentangle position from neighboring context and visual budget, and report results with invalid outputs excluded? Please also provide the frequency of fallback alignment by model and position and clarify how the held-out position contrasts in Table 3 are computed.
-
The benchmark scope and evaluation reliability remain limited. The experiments consider at most four videos of up to two minutes, with only 27 groups per domain-duration configuration. Furthermore, ActivityNet and News use different Coverage estimators, and the GPT-5.1 judge is validated on 245 summaries without reporting agreement by model, domain, or position. Could the authors extend the evaluation to longer multi-video contexts and provide more detailed judge validation? Clarification of the inference configurations for proprietary models would also improve reproducibility.
-
The mitigation and novelty analyses require stronger experimental support. The evaluated interventions are limited to prompt-level variants on a small subset of models, and Table 6 shows that single-target prompting has opposite effects for two-video and four-video inputs. Could the authors compare against order-aggregation methods and existing training-free positional-bias mitigation approaches, such as SoFA (CVPR 2025)? The paper should also better distinguish its contributions from prior studies of positional bias in summarization, including "On Context Utilization in Summarization with Large Language Models" (ACL 2024) and "Summary of a Haystack" (EMNLP 2024).
[1] Tian, Xinyu, et al. "Identifying and Mitigating Position Bias of Multi-image Vision-Language Models." CVPR. 2025.
[2] Ravaut, Mathieu, et al. "On context utilization in summarization with large language models." ACL. 2024.
[3] Laban, Philippe, et al. "Summary of a haystack: A challenge to long-context llms and rag systems." EMNLP. 2024.
-
The released code contains several inconsistencies with the evaluation protocol and raises reproducibility concerns. Section 3.3.2 defines DPB as the average of per-instance scores, whereas "scripts/aggregate_position_metrics.py" first averages Coverage across positions and then computes DPB. This difference may explain the inconsistent DPB values between Tables 1 and 11. Furthermore, "scripts/compute_position_metrics.py" groups records without distinguishing cyclic orders, potentially producing incorrect coverage vectors. The reference-based fallback alignment described in Section 3.2 is absent from the released parser, which instead assigns zero Coverage to missing summary slots. The GPT-5.1 judge prompt, output schema, and score normalization also differ from Appendix A.2. More importantly, the repository does not include the raw model generations, judge decisions, or analysis scripts needed to reproduce the main results, and the released manifests contain 676 distinct video IDs rather than the 708 reported in Section 3.1. Could the authors reconcile these discrepancies, clarify which implementations produced the reported results, and release the complete evaluation pipeline and intermediate outputs necessary for reproducibility?
-
Minor: Several presentation and reference issues should be corrected. Section 4.3 discusses Cooking and Domestic for InternVL3.5-8B, whereas Figure 3 presents Domestic and News. The description of the single-target labeling experiment is also unclear about the indexing convention used across tables. In the references, QuestEval is incorrectly attributed to Rebuffel et al. (2021), which corresponds to Data-QuestEval rather than the summarization metric by Scialom et al. (EMNLP 2021). Some references also retain arXiv versions despite subsequent formal publication.
[4] Scialom, Thomas, et al. "QuestEval: Summarization Asks for Fact-based Evaluation." EMNLP. 2021.