- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SumReason, a multi-annotator benchmark for standard and query-conditioned video summarization. Its queries are organized into four categories: Direct Visual, Temporal Reasoning, High-Level Semantics, and External Knowledge Reasoning. The authors benchmark heuristic, supervised, unsupervised, LLM-based zero-shot, and zero-shot MLLM scene-scoring pipelines under 2-second and KTS segmentation. They conclude that TR is the main bottleneck and that SumReason is harder than SumMe/TVSum.
Critical strengths
-
The four-way query taxonomy makes the benchmark diagnostically useful, because it supports per-category failure analysis that a single aggregate F1 on SumMe/TVSum cannot provide. Separating perception-level queries (DV) from reasoning-level queries (TR, EKR) is a sensible design for the MLLM era.
-
The empirical coverage is broad and reasonably up to date. The paper spans classical heuristics, more than a dozen supervised and unsupervised summarizers, an LLM-as-judge pipeline, and four open and closed MLLMs under two segmentation schemes.
-
Reporting both a random floor and human agreement is good practice, because it places absolute F1 values in context.
Critical weaknesses
-
The novelty relative to closely related benchmarks is not clearly delineated. In particular, Barbara & Maalouf (2025) already released VidSum-Reason [1], a query-driven summarization dataset that emphasizes long-tailed concepts and multi-step reasoning, and the present paper uses their P2S pipeline as a baseline. The paper should state explicitly how SumReason differs from VidSum-Reason (videos, queries, annotation protocol, overlap) and from QFVS [2]. A comparison table against QFVS, VidSum-Reason, Mr. HiSum, and VideoXum would make the contribution concrete, covering #videos, #queries, #annotators per query, query types, metrics,
-
The evaluation relies solely on F1 under KTS/Knapsack, a protocol known to be dominated by segmentation and largely insensitive to the predicted importance scores. In Table 3, the best query-conditioned model (InternVL3-8B, 55.78) is only about 6 points above PoR (49.9), and LGRLN (47.58) falls below PoR. Two “random” baselines also differ by about 27 points (RS 23.2 vs. PoR 49.9), and this gap is not explained. It points to a fragment-length or Knapsack artifact rather than a difference in summary quality. I ask the authors to report rank-correlation metrics (Kendall’s τ, Spearman’s ρ), as LLMVS does [3] and report PoR separately for the 2-second and KTS segmentations;
References
- Barbara, M. & Maalouf, A. Prompts to Summaries: Zero-Shot Language-Guided Video Summarization. arXiv 2025. DOI: 10.48550/arXiv.2506.10807
- Sharghi, A. et al. Query-Focused Video Summarization: Dataset, Evaluation, and a Memory Network Based Approach. CVPR 2017. DOI: 10.1109/CVPR.2017.229
- Lee, M. J. et al. Video Summarization with Large Language Models. CVPR 2025. DOI: 10.1109/CVPR52734.2025.01768
Questions for authors
Do the category rankings and the “hardest benchmark” conclusion hold under Kendall’s τ / Spearman’s ρ? /
4 comments
We thank the reviewer for their detailed and constructive feedback. As the discussion period draws to a close, we would like to highlight the additional evidence presented in our rebuttal and shared results comments. We provided detailed comparisons with VidSum-Reason, QFVS, and other recent benchmarks, clarifying SumReason's distinctive contributions. To address evaluation concerns, we introduced Spearman's ρ, Kendall's τ, normalized F1 gains, and confidence intervals, and evaluated PoR separately under frame-level, 2-second, and KTS segmentation. Importantly, SumReason exhibits the lowest human-normalized rank correlation for 13/17 methods (76.5%) compared with SumMe and TVSum. Additionally, TR has the lowest rank correlation in 9/21 query-conditioned configurations under both metrics, providing complementary evidence beyond F1. We believe these additions strengthen our findings and address your concerns, and hope you will reconsider your score. If any concerns remain, please share them here. We would be happy to respond as promptly as possible. Thank you again for your constructive review!
After reviewing the author’s response and the comments from other reviewers, I still consider SumReason to be of value; therefore, my final recommendation remains ‘weak accept’.
We thank the reviewer for recognizing the value of our work and for their constructive feedback, which helped us strengthen our experiments, results, and conclusions.
We thank the reviewer for recognizing our contributions and appreciate the constructive feedback that motivated additional analyses.
Regarding W1: comparisons with existing video summarization benchmarks. We thank the reviewer for highlighting these important benchmarks. Although they make valuable contributions, their task formulations, annotation protocols, and evaluation objectives differ from SumReason. VidSum-Reason: Our closest predecessor introduced reasoning-oriented queries in four categories, primarily as a proof of concept for its summarization method rather than a standalone dataset. It contains 9 videos and 20 video-query pairs, scored by one annotator and reviewed by two others. Its 1–5 scores combine query relevance with general video importance: query-irrelevant segments are scored for narrative importance. Furthermore, It was not evaluated against MLLMs or query based methods, and places less emphasis on systematically evaluating complex temporal reasoning. In contrast, SumReason has 37 videos, 60 pairs, and ten independent annotations per query, separates query relevance from query-independent importance, and uses a differentiated DV/TR/HLS/EKR taxonomy to analyze reasoning-specific performance. QFVS: As discussed in Section 2 and Appendix A1, it contains four long egocentric videos (3–5 hours), each evaluated using roughly 46 concept-based queries. It employs concept-labeled 5-second shots and semantic matching against human-selected summaries. While an established query-focused summarization benchmark, its queries primarily identify visual concepts and their combinations, rather than explicitly testing temporal dependencies or external knowledge. Moreover, it does not provide ten independent graded relevance annotations per query. S-VideoXum: This benchmark contains ~12k videos and ten script-summary pairs per video. MLLM-generated scripts describe pre-existing human summaries, supporting script-driven selection. SumReason instead independently formulates reasoning queries and collects human relevance judgments for each query, rather than deriving the query from an existing summary. QVHighlights: With ~10k videos and ~10k natural-language queries, it annotates relevant moments and five-level clip saliency. Its main objectives are moment retrieval and highlight detection, whereas SumReason evaluates duration-constrained extractive summaries against multiple human references. VideoXum: It contains ~14k videos and 140k aligned visual–textual summary pairs for cross-modal summarization and semantic alignment. Although it offers multiple human visual summaries per video, these are not organized by independent reasoning-oriented user queries. SumReason instead measures which segments satisfy a specified query and how those selections differ from general video importance. Mr. HiSum: Its ~32k videos receive frame-importance labels derived from aggregated viewer-replay behavior. Such labels capture audience engagement and commonly replayed highlights, not whether segments satisfy specific temporal-reasoning or external-knowledge queries as in SumReason.
Regarding W2: F1 evaluation protocol and random baselines. To examine this concern, we additionally report rank correlation metrics, providing complementary evaluation of predicted importance rankings independently of final summary selection. We also evaluated PoR under different segmentation settings, as requested: 48.09% (2-second) and 43.69% (KTS). Regarding the discrepancy between RS and PoR. It is caused by procedures RS directly samples random frames, whereas PoR assigns random importance scores before budget-constrained summary selection. Thus, they are not equivalent random controls, and fragment selection may contribute to the difference.
Regarding the question of category ranking and hardest benchmark conclusion under rank-correlation metrics. Following the reviewer's suggestion, we report rank correlation across reasoning categories and evaluated methods. Temporal Reasoning has the lowest correlation in 9/21 configurations under both metrics, followed by External Knowledge Reasoning. These results support our F1 findings. Notably, TR exhibits lower model agreement despite relatively strong human rank agreement, suggesting that the observed difficulties cannot be attributed to annotation disagreement. For standard summarization, SumReason exhibits the lowest human-normalized rank correlation for 13/17 methods (76.5%) compared with TVSum and SumMe, providing further evidence of benchmark difficulty beyond F1. Due to limited space we refer the reviewer to tables titled "Standard summarization gain, CI, and rank correlation datasets comparison" and "Query conditioned summarization rank correlation category comparison" for detailed results.
We believe our additional experiments and comparisons address the concerns and strengthen the evidence for SumReason. We hope they will encourage the reviewer to increase their score.