acceptodds
Under review as a conference paper at ICLR 2027

SumReason: A Benchmark for Reasoning-Driven Query-Conditioned and Standard Video Summarization

Abstract

Video summarization benchmarks have largely evolved along two disconnected tracks: standard summarization datasets that assume a single notion of importance, and query/text-conditioned datasets whose queries often reduce to simple object or concept retrieval/matching. As a result, current benchmarks do not adequately test whether models can adapt summaries to user intent or perform the temporal, semantic, and knowledge-based reasoning required in realistic settings. We introduce SumReason, a unified benchmark for both standard and query-conditioned video summarization that explicitly targets this gap of reasoning-intensive video understanding. The dataset contains 37 videos, 60 video-query pairs, and 25 query-independent annotation sets, with 850 total annotations from 10 annotators. Its query hierarchy spans Direct Visual, Temporal Reasoning, High-Level Semantic Understanding, and External Knowledge Reasoning, and is designed to shift focus toward both salient and secondary events. SumReason combines strong annotation reliability with increased task difficulty: it achieves higher inter-annotator agreement than legacy benchmarks in the standard setting, while query-conditioned summaries overlap with standard summaries by only 30–40% on average, confirming substantial query-induced semantic shifts. Across extensive evaluations of >15 video summarization methods and frontier open- and closed-source MLLMs, current systems exhibit substantial limitations on SumReason: standard summarization methods show markedly smaller gains than on established datasets, while MLLMs reveal a persistent reasoning gap in the query-conditioned setting. We position SumReason as a benchmark for studying intent-aware summarization moving beyond surface-level visual matching.

Primary area:
datasets and benchmarks
Keywords:
Video summarization, Query-conditioned summarization, Multimodal learning, Temporal reasoning, Vision-language models, Benchmark dataset

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WYTGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SumReason, a multi-annotator benchmark for standard and query-conditioned video summarization. Its queries are organized into four categories: Direct Visual, Temporal Reasoning, High-Level Semantics, and External Knowledge Reasoning. The authors benchmark heuristic, supervised, unsupervised, LLM-based zero-shot, and zero-shot MLLM scene-scoring pipelines under 2-second and KTS segmentation. They conclude that TR is the main bottleneck and that SumReason is harder than SumMe/TVSum.

Critical strengths
  1. The four-way query taxonomy makes the benchmark diagnostically useful, because it supports per-category failure analysis that a single aggregate F1 on SumMe/TVSum cannot provide. Separating perception-level queries (DV) from reasoning-level queries (TR, EKR) is a sensible design for the MLLM era.

  2. The empirical coverage is broad and reasonably up to date. The paper spans classical heuristics, more than a dozen supervised and unsupervised summarizers, an LLM-as-judge pipeline, and four open and closed MLLMs under two segmentation schemes.

  3. Reporting both a random floor and human agreement is good practice, because it places absolute F1 values in context.

Critical weaknesses
  1. The novelty relative to closely related benchmarks is not clearly delineated. In particular, Barbara & Maalouf (2025) already released VidSum-Reason [1], a query-driven summarization dataset that emphasizes long-tailed concepts and multi-step reasoning, and the present paper uses their P2S pipeline as a baseline. The paper should state explicitly how SumReason differs from VidSum-Reason (videos, queries, annotation protocol, overlap) and from QFVS [2]. A comparison table against QFVS, VidSum-Reason, Mr. HiSum, and VideoXum would make the contribution concrete, covering #videos, #queries, #annotators per query, query types, metrics,

  2. The evaluation relies solely on F1 under KTS/Knapsack, a protocol known to be dominated by segmentation and largely insensitive to the predicted importance scores. In Table 3, the best query-conditioned model (InternVL3-8B, 55.78) is only about 6 points above PoR (49.9), and LGRLN (47.58) falls below PoR. Two “random” baselines also differ by about 27 points (RS 23.2 vs. PoR 49.9), and this gap is not explained. It points to a fragment-length or Knapsack artifact rather than a difference in summary quality. I ask the authors to report rank-correlation metrics (Kendall’s τ, Spearman’s ρ), as LLMVS does [3] and report PoR separately for the 2-second and KTS segmentations;

References

  1. Barbara, M. & Maalouf, A. Prompts to Summaries: Zero-Shot Language-Guided Video Summarization. arXiv 2025. DOI: 10.48550/arXiv.2506.10807
  2. Sharghi, A. et al. Query-Focused Video Summarization: Dataset, Evaluation, and a Memory Network Based Approach. CVPR 2017. DOI: 10.1109/CVPR.2017.229
  3. Lee, M. J. et al. Video Summarization with Large Language Models. CVPR 2025. DOI: 10.1109/CVPR52734.2025.01768
Questions for authors

Do the category rankings and the “hardest benchmark” conclusion hold under Kendall’s τ / Spearman’s ρ? /

4 comments

We thank the reviewer for recognizing our contributions and appreciate the constructive feedback that motivated additional analyses.

Regarding W1: comparisons with existing video summarization benchmarks. We thank the reviewer for highlighting these important benchmarks. Although they make valuable contributions, their task formulations, annotation protocols, and evaluation objectives differ from SumReason. VidSum-Reason: Our closest predecessor introduced reasoning-oriented queries in four categories, primarily as a proof of concept for its summarization method rather than a standalone dataset. It contains 9 videos and 20 video-query pairs, scored by one annotator and reviewed by two others. Its 1–5 scores combine query relevance with general video importance: query-irrelevant segments are scored for narrative importance. Furthermore, It was not evaluated against MLLMs or query based methods, and places less emphasis on systematically evaluating complex temporal reasoning. In contrast, SumReason has 37 videos, 60 pairs, and ten independent annotations per query, separates query relevance from query-independent importance, and uses a differentiated DV/TR/HLS/EKR taxonomy to analyze reasoning-specific performance. QFVS: As discussed in Section 2 and Appendix A1, it contains four long egocentric videos (3–5 hours), each evaluated using roughly 46 concept-based queries. It employs concept-labeled 5-second shots and semantic matching against human-selected summaries. While an established query-focused summarization benchmark, its queries primarily identify visual concepts and their combinations, rather than explicitly testing temporal dependencies or external knowledge. Moreover, it does not provide ten independent graded relevance annotations per query. S-VideoXum: This benchmark contains ~12k videos and ten script-summary pairs per video. MLLM-generated scripts describe pre-existing human summaries, supporting script-driven selection. SumReason instead independently formulates reasoning queries and collects human relevance judgments for each query, rather than deriving the query from an existing summary. QVHighlights: With ~10k videos and ~10k natural-language queries, it annotates relevant moments and five-level clip saliency. Its main objectives are moment retrieval and highlight detection, whereas SumReason evaluates duration-constrained extractive summaries against multiple human references. VideoXum: It contains ~14k videos and 140k aligned visual–textual summary pairs for cross-modal summarization and semantic alignment. Although it offers multiple human visual summaries per video, these are not organized by independent reasoning-oriented user queries. SumReason instead measures which segments satisfy a specified query and how those selections differ from general video importance. Mr. HiSum: Its ~32k videos receive frame-importance labels derived from aggregated viewer-replay behavior. Such labels capture audience engagement and commonly replayed highlights, not whether segments satisfy specific temporal-reasoning or external-knowledge queries as in SumReason.

Regarding W2: F1 evaluation protocol and random baselines. To examine this concern, we additionally report rank correlation metrics, providing complementary evaluation of predicted importance rankings independently of final summary selection. We also evaluated PoR under different segmentation settings, as requested: 48.09% (2-second) and 43.69% (KTS). Regarding the discrepancy between RS and PoR. It is caused by procedures RS directly samples random frames, whereas PoR assigns random importance scores before budget-constrained summary selection. Thus, they are not equivalent random controls, and fragment selection may contribute to the difference.

Regarding the question of category ranking and hardest benchmark conclusion under rank-correlation metrics. Following the reviewer's suggestion, we report rank correlation across reasoning categories and evaluated methods. Temporal Reasoning has the lowest correlation in 9/21 configurations under both metrics, followed by External Knowledge Reasoning. These results support our F1 findings. Notably, TR exhibits lower model agreement despite relatively strong human rank agreement, suggesting that the observed difficulties cannot be attributed to annotation disagreement. For standard summarization, SumReason exhibits the lowest human-normalized rank correlation for 13/17 methods (76.5%) compared with TVSum and SumMe, providing further evidence of benchmark difficulty beyond F1. Due to limited space we refer the reviewer to tables titled "Standard summarization gain, CI, and rank correlation datasets comparison" and "Query conditioned summarization rank correlation category comparison" for detailed results.

We believe our additional experiments and comparisons address the concerns and strengthen the evidence for SumReason. We hope they will encourage the reviewer to increase their score.

We thank the reviewer for their detailed and constructive feedback. As the discussion period draws to a close, we would like to highlight the additional evidence presented in our rebuttal and shared results comments. We provided detailed comparisons with VidSum-Reason, QFVS, and other recent benchmarks, clarifying SumReason's distinctive contributions. To address evaluation concerns, we introduced Spearman's ρ, Kendall's τ, normalized F1 gains, and confidence intervals, and evaluated PoR separately under frame-level, 2-second, and KTS segmentation. Importantly, SumReason exhibits the lowest human-normalized rank correlation for 13/17 methods (76.5%) compared with SumMe and TVSum. Additionally, TR has the lowest rank correlation in 9/21 query-conditioned configurations under both metrics, providing complementary evidence beyond F1. We believe these additions strengthen our findings and address your concerns, and hope you will reconsider your score. If any concerns remain, please share them here. We would be happy to respond as promptly as possible. Thank you again for your constructive review!

After reviewing the author’s response and the comments from other reviewers, I still consider SumReason to be of value; therefore, my final recommendation remains ‘weak accept’.

We thank the reviewer for recognizing the value of our work and for their constructive feedback, which helped us strengthen our experiments, results, and conclusions.

Reviewer_jrc4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SumReason, a benchmark for standard and query-conditioned video summarization, emphasizing temporal, semantic, and knowledge-based reasoning. The dataset contains 37 videos, 60 video-query pairs, and 850 annotations from 10 annotators. The authors evaluate traditional video summarization methods and recent MLLMs, highlighting their limitations, particularly in temporal reasoning.

Critical strengths
  1. Meaningful benchmark motivation. The paper addresses an important limitation of existing video summarization benchmarks by introducing queries requiring temporal reasoning, high-level semantic understanding, and external knowledge.

  2. Extensive evaluation and annotation analysis. The authors evaluate various traditional summarization methods and recent MLLMs. The multi-annotator design and analysis of annotation consistency provide useful insights into benchmark reliability.

Critical weaknesses
  1. The benchmark scale is relatively limited especially for some specific query categories.

Although the benchmark includes multiple human annotations, it contains only 60 video-query pairs, with particularly limited coverage for High-Level Semantic Understanding (9 queries) and External Knowledge Reasoning (13 queries), which somehow weaken the credibility of Table 3.

Furthermore, while the authors report strong inter-annotator agreement, agreement alone may not guarantee correctness for complex reasoning queries. Additional independent validation, potentially using MLLMs as auxiliary judges followed by human verification, could strengthen annotation reliability.

  1. Insufficient validation of reasoning capabilities.

Although the paper emphasizes long-range temporal reasoning, the zero-shot MLLM pipeline evaluates each scene independently using sampled frames (in Appendix D). This may prevent models from accessing the global temporal context necessary to answer certain queries, such as "identifying an event after the third occurrence of another event". Consequently, it is unclear whether the reported reasoning gaps reflect limitations of the models or the evaluation protocol. Comparisons with whole-video evaluation may strengthen the conclusions.

Questions for authors

How representative are the reasoning categories given the limited number of queries (9 and 13)? Could the authors provide additional evidence of annotation correctness beyond inter-annotator agreement, potentially through independent human or AI-assisted verification?

4 comments

We thank the reviewer for recognizing SumReason's contributions and extensive evaluation. We appreciate the constructive feedback, which motivated additional validation evidence and clarification, strengthening our work.

Regarding W1: benchmark scale and reasoning-category representativeness. SumReason was deliberately designed to address gaps in reasoning-oriented evaluation and annotation depth, rather than compete solely on video count. Established standard summarization benchmarks, such as SumMe (25 videos) and TVSum (50 videos), provide valuable human annotations but do not explicitly evaluate query-dependent temporal reasoning, semantic interpretation, or external knowledge requirements. Query-focused benchmarks such as QFVS support multiple queries per video but primarily use concept-based queries over four egocentric videos. In contrast, SumReason contains 37 videos, 60 video-query pairs, and ten independent human annotations per query, alongside standard summarization annotations. Its four query categories—DV, TR, HLS, and EKR—were deliberately constructed to evaluate distinct understanding and reasoning requirements. We prioritized detailed, independently assessed annotations and carefully defined queries over increasing dataset size without comparable annotation depth. To fully address your comment, we will revise the manuscript to ensure that our observations, interpretations, and conclusions are specific to our evaluated data rather than presented as general or universal claims. Nevertheless, the observed gap remains meaningful. Our experiments demonstrate that, within this controlled and independently annotated setting, existing summarization methods and MLLMs exhibit persistent difficulties with queries requiring temporal reasoning and external knowledge. These results identify concrete limitations under the tested conditions and establish SumReason as a complementary testbed for investigating them. Expanding category coverage and assessing generalization to broader query distributions are important directions for future work.

Regarding W1: annotation correctness beyond inter-annotator agreement. We thank the reviewer for this concern. As mentioned in Appendix Section B, annotation quality was assessed beyond inter-annotator agreement through a separate procedure involving 3 independent human validators.

First, the validators watched the videos without audio, verified that each query could be satisfied by identifiable visual events, and independently assessed its reasoning and knowledge requirements. More importantly, blinded to the original annotations and summaries, each validator directly selected the most query-relevant temporal intervals under the same 15% summary budget, without being restricted to predefined scenes. Unlike the original 1–5 importance-scoring protocol, this procedure produced independently selected binary summaries. We compared each validator's selections against all ten annotation-derived summaries using F1. The validators achieved 69.54% mean pairwise F1. Their mean agreement with the annotation-derived summaries was 65.94% (DV), 79.02% (TR), 60.16% (HLS), and 60.83% (EKR). Furthermore, annotations with F1 below 50% against any validator were flagged for manual examination, with reannotation performed only when an issue was confirmed. As documented in Appendix B.4, only 15 of 850 annotations (1.76%) required reannotation, all from standard summarization; none of the 600 query-conditioned annotations required revision. Second, an independent MLLM evaluation has already been tested in Table 3. It demonstrates the models ability to reproduce human relevance judgments. When we ask the model to score each 2 second or KTS segments, as shown, MLLM struggled to simulate human score, specifically once we look at rank correlation. Thus we didn’t include them as a validation for the annotation.

Regarding W2: Insufficient validation of reasoning capabilities. We agree that independently scoring short scenes may limit long-range temporal reasoning. However, our original evaluation included Gemini 3.1 Pro with full-video input, evaluated using both 2-second and KTS-based scoring units. To strengthen this evaluation, we additionally evaluated Qwen3-VL-8B and InternVL3-8B with full-video input under both granularities, yielding six full-video configurations. TR is the lowest-performing category in 4/6 configurations and falls below HLS in all six. Even the strongest full-video TR result, Gemini 3.1 Pro (KTS), achieves 53.05% F1 versus 63.8% human-reference F1(a 10.75-point gap). Thus, TR difficulty persists despite global context. Due to limited space, we refer the reviewer to the table titled “Query conditioned summarization new F1 score category comparison” for full results.

We believe our additional experiments and clarifications address the concerns and strengthen the evidence for SumReason. We hope they will encourage the reviewer to increase their score.

We thank the reviewer for their detailed and constructive feedback. As the discussion period draws to a close, we would like to highlight the additional evidence presented in our rebuttal and shared results comments. To address annotation correctness beyond inter-annotator agreement, we presented independent validation by three blinded human evaluators, obtaining 60.16–79.02% F1 agreement with annotation-derived summaries across categories. We also clarified our quality-control procedure, under which only 15/850 annotations (1.76%) required reannotation, none query-conditioned. Regarding benchmark scale, we clarified the scope of our conclusions. Finally, we expanded the evaluation to six full-video MLLM configurations, demonstrating that temporal-reasoning difficulty persists even with access to global context. We believe these additions directly address your concerns and hope you will reconsider your score. If any concerns remain, please share them here. We would be happy to respond as promptly as possible. Thank you again for your constructive review!

Reviewer_jrc4OpenReview ↗

Official Comment by Reviewer jrc4

Thanks the authors for clarification on benchmark scale and category representativeness. Such has addresses my concerns, and after consideration, I would like to raise my score to "Weak Acceptance".

We thank the reviewer for considering our clarifications and for raising thoughtful questions about benchmark scale and category representativeness. This discussion helped us clarify the scope of our findings and contributions.

Reviewer_L66fOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SumReason, a benchmark that evaluates standard and query-conditioned video summarization on the same videos, with 60 video-query pairs over 37 videos annotated by 10 annotators. Its queries are organized into four reasoning categories, ranging from direct visual cues to temporal reasoning and external knowledge. The authors show that adding a query substantially changes which segments annotators consider important. In their experiments, standard methods barely improve over the random baseline, while MLLMs do reasonably well on visual and semantic queries but struggle with temporal reasoning.

Critical strengths

S1. The query-induced shift analysis is, to me, the most convincing part of the paper. On the same videos, query-conditioned summaries overlap with the standard ones by only 30-40% F1, and for many TR and EKR queries the overlap is close to zero. This is direct evidence that query-conditioned summarization can't be reduced to generic saliency, and it's the main reason I think a benchmark like this is worth having.

S2. Both the query design and the annotation are a step up from existing query-focused datasets. The queries ask for ordering, counting, causal links, or domain knowledge instead of just spotting a concept. Each item also comes with 10 non-aggregated annotations and per-category agreement, which gives the community an evaluation axis that QFVS-style benchmarks don't really offer.

Critical weaknesses

W1. I'm not convinced the benchmark measures summarization quality well, which undercuts the "hardest benchmark" claim. The ground-truth summaries are not built by annotators but derived from independently scored 2-second shots through KTS averaging and a knapsack, so they cannot reflect set-level properties such as redundancy, coverage, or continuity. The KTS + knapsack + F1 protocol is also known to let random summaries approach human scores (Otani et al., CVPR 2019). Here the gap between PoR and PoH is only about 7 F1 and the best standard method already closes most of it, yet rank correlation metrics are not reported for the methods.

W2. The central finding, a temporal reasoning bottleneck in MLLMs, is not supported by the experiments. Each scene is scored independently from at most 16 frames, so a query like "after the third overtake" cannot be answered no matter how capable the model is, and the 2s vs. KTS comparison does not change this. With only 60 queries and per-category standard deviations of 10-20 F1, the category gaps are also within noise. Some of the reported statistics are inconsistent as well, such as GPT-4o's Avg in Table 8 and the p-value of 0.98 for P2S with GPT-5.6-Sol in Table 9.

Questions for authors

Q1. Could you report Kendall's tau and Spearman's rho for each method, together with the normalized gain (F1 - PoR) / (PoH - PoR), with confidence intervals across splits?

Q2. How does SumReason differ from existing text-conditioned summarization datasets such as S-VideoXum and QVHighlights? It would also help to add more recent baselines such as LLMVS, V2Xum-LLM, SummDiff, TripleSumm, and AgenticVS for the standard track, and query-based saliency models such as QD-DETR or UniVTG for the query-conditioned track.

Q3. (Minor) Several method details are described inconsistently, and there are a number of typos. For example, Qwen3-VL appears as both 7B and 8B. Typos include "Gemini-3.6-Flush", "MulitModal", and "addign an importanse score ... withing".

4 comments

We thank the reviewer for recognizing our analysis, queries, and multi-annotator annotations, and for the constructive feedback, which motivated additional experiments and clarifications that strengthened our work.

Regarding W1: summarization quality, evaluation validity and rank correlation. To establish that the adopted protocol is more robust to redundancy, coverage, and continuity, we argue the following. First, regarding the annotation protocol, our benchmark follows the established video summarization protocol: fine-grained human importance scores assigned to 2-second shots are aggregated into time-based fragments, which serve as the building blocks for constructing each summary. Second, regarding relevance concentration of Knapsack, we averaged the ten annotators' scores within each KTS scene and identified top-scoring scenes (mean ≥ 4.5). These occupy only 5.6±7.3% of video duration, and in approximately 90% of video-query pairs (80–95% across categories), they collectively fit within the 15% summary budget, leaving room for contextual content. Thus, the budget generally accommodates the most relevant events without substantial selection trade-offs. Third, to address the limitations of F1-based evaluation, we additionally report rank correlation, and confidence intervals for both settings. In standard summarization, SumReason exhibits the lowest rank correlation for 13/17 evaluated methods (76.5%), in which they also contextualize performance relative to the PoR–PoH gap. In query-conditioned summarization, the results also reveal persistent challenges in TR and EKR. We ask the reviewer to look at the tables titled: “Standard summarization gain, CI ,and rank correlation datasets comparison” and “Query-conditioned summarization rank correlation category comparison“. Fourth, independent human validation. Three validators, blinded to the original annotations, directly selected keyshots under the same 15% budget. Their selections achieved 69.54% mean pairwise F1 and 60.16–79.02% F1 against annotation-derived summaries across categories, providing independent evidence of cross-protocol agreement. Due to limited space we refer the reviewer to see our response to Reviewer Jrc4.

Regarding W2: central finding for temporal reasoning and full-video MLLM evaluation. We agree that scene-level scoring may limit long-range temporal reasoning. However, our original evaluation already included Gemini 3.1 Pro with full-video input, generating relevance scores under both fixed 2-second and KT- based segmentation. We will clarify this distinction from scene independent evaluation in the revised manuscript. To further assess the effect of full video context, we evaluated two additional full video MLLMs (Qwen3-VL-8B and InternVL3-8B) under both scoring granularities. We also added 2 new methods, QD-DETR and UniVTG, in zero-shot and supervised settings. As shown in the table titled "Query conditioned summarization new F1 score category comparison" access to full-video context does not consistently eliminate TR difficulty. TR remains the lowest-performing category in 4/6 full-video configurations and scores below HLS in all six.

Regarding W2+Q3: inconsistent statistical reporting and typos. We will verify the reported results, and will ensure that all numerical inconsistencies, model identifiers, and typo errors are corrected in the revised manuscript.

Regarding Q1: rank correlation, normalized gain, and confidence interval over splits. To fully address the reviewers question, we report rank correlation, normalized gain, and confidence interval over splits, for both the standard and query conditioned track, including the new methods, and MLLM that uses full context as an input. We refer the reviewer to tables titled: “Standard summarization gain, CI ,and rank correlation datasets comparison”, “Query conditioned summarization normalized gain and CI”, and “Query conditioned summarization rank correlation category comparison".

Regarding Q2: evaluating recent methods. Following the reviewer's suggestions, we added 2 full-video MLLMs (Qwen3-VL-8B and InternVL3-8B), 2 query-conditioned methods (QD-DETR and UniVTG), each evaluated in zero-shot and supervised settings, and 3 recent standard summarization methods (SumDiff, LLMVS, and V2Xum-LLM). TripleSumm was excluded because its visual–audio–text fusion requires modalities and temporal alignment beyond our visual-only setting.

Regarding Q2: comparisons with existing video summarization benchmarks. We provide a detailed comparison with existing video summarization benchmarks, emphasizing differences in task formulation, reasoning requirements, query diversity, and annotation protocols. Due to limited space we refer the reviewer to our response to Reviewer WYTG.

We believe our additional experiments and clarifications address the concerns and strengthen the evidence for SumReason. We hope they will encourage the reviewer to increase their score.

We thank the reviewer for their detailed and constructive feedback. As the discussion period draws to a close, we would like to highlight the additional evidence presented in our rebuttal and shared results comments. We conducted independent human validation, examined relevance concentration under the 15% summary budget, and introduced Spearman's ρ, Kendall's τ, normalized gains, and confidence intervals to complement F1. We also expanded our evaluation with recent summarization methods and six full-video MLLM configurations. Notably, TR remains the lowest-performing category in 4/6 full-video configurations, demonstrating that its difficulty persists even with global temporal context. We also clarified the evaluation protocols and methodological details. We believe these additions substantially address your concerns and hope you will reconsider your score. If any concerns remain, please feel free to share them here. We would be happy to respond as promptly as possible. Thank you again for your constructive review!

I thank the authors for the additional experiments and detailed responses. The new results and clarifications have substantially addressed my concerns about the paper, particularly the independent human validation, comparisons with additional baselines, and clarification of the evaluation protocol. I will therefore raise my score to Weak Accept. However, these additions and clarifications do not yet appear to be reflected in the submitted paper. I ask the authors to revise the paper to include the experimental results and explanations provided in the rebuttal.

We thank the reviewer for reassessing our submission in light of the rebuttal. We have now uploaded a revised manuscript that includes the independent human validation, the additional baseline and full-video experiments , and the evaluation protocol clarifications. We also cross-checked the reported results and corrected numerical inconsistencies and typographical errors.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Results (1/2): Overall performance and cross-dataset rank correlations

We thank all reviewers for their constructive feedback, which motivated additional experiments and analyses. Given the space limits and for your convenience, we are sharing the results in two comments for ease of reference. This first comment contains two tables covering overall performance in standard and query-conditioned video summarization. The second comment contains two further tables.

During the rebuttal period, we expanded our standard video summarization evaluation with SumDiff, LLMVS, and V2Xum-LLM. For query-conditioned summarization, we added zero-shot and supervised evaluations of QD-DETR and UniVTG, along with additional full-video MLLM evaluations. Both tables report normalized F1 gains and approximate 95% confidence intervals across five splits. For standard summarization, SumReason had the lowest human-normalized Spearman’s ρ and Kendall’s τ for 13/17 methods (76.5%) across SumReason, SumMe, and TVSum. For query-conditioned summarization, several configurations showed modest F1 gains over the shared frame-level random baseline. Full-video context did not consistently improve overall F1 over scene-level scoring.

Standard summarization gain, CI ,and rank correlation datasets comparison

MethodNormalized gain G (F1)Approx. 95% CI for F1 (%)SumReason ρ / τTVSum ρ / τSumMe ρ / τ
PoR0.000[48.78, 60.50]~0 / ~0~0 / ~0~0 / ~0
PoH1.000[56.49, 65.91]0.394 / 0.3480.204 / 0.1770.213 / 0.205
DR-DSN-3.161[26.61, 33.95]0.024 / 0.0180.026 / 0.0200.048 / 0.047
VASNet-0.324[46.43, 61.59]0.035 / 0.0270.170 / 0.1600.170 / 0.160
SUM-FCN0.273[48.53, 64.29]0.035 / 0.0260.016 / 0.0120.186 / 0.167
DSNet0.669[52.24, 66.14]0.088 / 0.0650.083 / 0.0640.181 / 0.162
PGL-SUM-0.014[45.49, 63.28]0.030 / 0.0250.206 / 0.1570.116 / 0.104
STVT0.731[54.72, 69.23]~0.130 / ~0.1200.131 / 0.1000.172 / 0.154
CSTA0.000[45.57, 63.45]0.152 / 0.1690.255 / 0.1940.274 / 0.246
SUM-GAN-AAE-0.105[49.70, 57.83]~0 / ~0-0.062 / -0.0470.230 / 0.207
AC-SUM-GAN0.206[51.80, 60.10]0.040 / 0.0300.041 / 0.0310.088 / 0.102
CA-SUM0.384[50.57, 63.92]0.030 / 0.0200.210 / 0.1600.084 / 0.063
Uformer0.701[54.62, 64.25]~0 / ~00.164 / 0.1260.139 / 0.124
RS-SUM0.149[47.63, 63.83]0.070 / 0.0600.106 / 0.0800.015 / 0.007
SegSum0.572[56.92, 62.29]0.050 / 0.0300.057 / 0.0440.154 / 0.138
mpA_uvS0.794[55.09, 65.07]0.040 / 0.0300.070 / 0.0530.200 / 0.179
SumDiff0.220[45.78, 66.25]0.208 / 0.1620.255 / 0.1950.285 / 0.256
V2Xum-LLM-2.057[26.50, 53.59]———
LLMVS0.159[43.90, 66.93]0.082 / 0.0750.275 / 0.2110.282 / 0.253
P2S0.512[54.18, 62.33]0.121 / 0.1130.100 / 0.0760.060 / 0.050
Lowest normalized ρ and τ——13/17 (76.5%)2/17 (11.8%)2/17 (11.8%)

Query conditioned summarization normalized gain and CI

MethodF1 (%)Normalized gain GApprox. 95% CI for F1 (%)
Baselines
PoR (Frame)49.900.000[46.71, 52.65]
PoR-2s48.09-0.123[45.03, 51.15]
PoR-KTS43.69-0.422[41.72, 45.66]
PoH64.601.000[58.88, 70.08]
Prior SOTA
LGRLN47.58-0.158[42.37, 52.79]
FCSNA53.520.246[49.72, 57.31]
Additional query-conditioned methods
QD-DETR (zero-shot)48.15-0.119[44.08, 52.21]
QD-DETR (supervised)53.220.226[49.14, 57.31]
UniVTG (zero-shot)45.59-0.293[39.56, 51.63]
UniVTG (supervised)50.010.007[47.65, 52.36]
P2S (LLM)
GPT-5.6-Sol53.820.267[47.49, 56.04]
Claude-Fable-550.940.071[48.50, 53.39]
Qwen3.6 27B50.020.008[44.34, 55.71]
Gemini-3.6-Flash49.86-0.003[45.13, 54.60]
Qwen3-VL-8B (P2S)49.57-0.022[43.83, 55.31]
Scene-level MLLM — 2s
Qwen3-VL-8B52.010.144[48.69, 55.33]
InternVL3-8B55.780.400[51.72, 59.83]
GPT 5.6 Luna55.750.398[53.28, 62.21]
Scene-level MLLM — KTS
Qwen3-VL-8B52.050.146[47.76, 56.35]
InternVL3-8B53.590.251[49.60, 57.58]
GPT 5.6 Luna53.370.236[48.40, 58.35]
Full-video MLLM — 2s scoring
Gemini 3.1 Pro54.670.324[49.69, 59.66]
Qwen3-VL-8B50.850.065[46.81, 54.88]
InternVL3-8B50.230.022[45.58, 54.88]
Full-video MLLM — KTS scoring
Gemini 3.1 Pro54.990.346[50.96, 59.03]
Qwen3-VL-8B52.060.147[48.31, 55.81]
InternVL3-8B48.76-0.078[42.37, 55.16]
AuthorsOpenReview ↗

Results (2/2): Performance by reasoning category

The two tables below report F1 and rank correlations by query category. Temporal Reasoning has the lowest F1 in four of the six full-video MLLM configurations, and the strongest full-video result in this category remains 10.75 percentage points below the human reference. The rank correlations also show difficulty with External Knowledge Reasoning, which has the lowest Spearman’s ρ and Kendall’s τ for full-video Gemini 3.1 Pro under both scoring schemes.

Query conditioned summarization new F1 score category comparison

MethodDVTRHLSEKRAvg. F1
Baselines
PoR (Frame)50.25 ± 13.2947.10 ± 13.0154.91 ± 11.5749.25 ± 14.4549.90
PoR-2s48.59 ± 12.0745.58 ± 12.2853.28 ± 10.4047.67 ± 12.3448.09
PoR-KTS43.85 ± 11.1341.95 ± 10.0849.16 ± 7.9742.37 ± 8.8443.69
PoH63.60 ± 12.1063.80 ± 11.8565.40 ± 10.4065.50 ± 11.2064.60
Additional query-conditioned methods
QD-DETR (zero-shot)47.77 ± 17.6046.89 ± 18.1752.44 ± 18.1747.62 ± 15.9648.15
QD-DETR (supervised)55.37 ± 16.6152.29 ± 16.8953.26 ± 18.9551.65 ± 15.8853.22
UniVTG (zero-shot)43.36 ± 21.7341.76 ± 17.1848.57 ± 25.2552.50 ± 19.7445.59
UniVTG (supervised)49.17 ± 15.8147.88 ± 15.5655.82 ± 7.9750.41 ± 15.0050.01
Full-video MLLM — 2s scoring
Gemini 3.1 Pro54.47 ± 19.1151.54 ± 15.5757.90 ± 19.6457.53 ± 17.3954.67
Qwen3-VL-8B52.24 ± 17.5047.94 ± 15.3557.31 ± 18.8148.92 ± 13.0550.85
InternVL3-8B50.20 ± 14.5348.51 ± 14.3451.62 ± 20.8951.95 ± 20.3450.23
Full-video MLLM — KTS scoring
Gemini 3.1 Pro55.14 ± 19.2553.05 ± 18.1555.89 ± 20.1257.17 ± 17.7654.99
Qwen3-VL-8B52.59 ± 13.6849.95 ± 18.5860.23 ± 15.0248.92 ± 20.0852.06
InternVL3-8B42.91 ± 19.6845.62 ± 16.9960.19 ± 14.3053.80 ± 21.4048.76
Lowest F1 frequency1/24 (4.2%)16/24 (66.7%)0/24 (0.0%)7/24 (29.2%)—

Query conditioned summarization rank correlation category comparison

MethodDV ρ/τTR ρ/τHLS ρ/τEKR ρ/τAvg. ρ/τ
Baselines
PoR (Frame)-0.014/-0.0110.003/0.0020.007/0.006-0.016/-0.012-0.006/-0.004
PoR-2s0.002/0.0020.002/0.0010.003/0.0020.005/0.0040.003/0.002
PoR-KTS-0.003/-0.003-0.002/-0.002-0.001/-0.001-0.009/-0.007-0.004/-0.003
PoH (human, pairwise)0.647/0.6070.568/0.5430.509/0.4610.592/0.5590.588/0.554
CLIP
Prior SOTA
LGRLN-0.090/-0.072-0.003/-0.002-0.122/-0.094-0.003/-0.003-0.047/-0.037
FCSNA0.027/0.022-0.001/0.0000.026/0.022-0.015/-0.0120.009/0.007
Additional query-conditioned methods
QD-DETR (zero-shot)0.023/0.0200.203/0.1610.153/0.1190.136/0.1080.127/0.101
QD-DETR (supervised)0.173/0.1630.091/0.0850.061/0.0570.052/0.0510.096/0.090
UniVTG (zero-shot)0.022/0.0160.000/0.0000.036/0.0300.040/0.0300.000/0.000
UniVTG (supervised)0.079/0.0660.122/0.0980.104/0.0800.023/0.0200.085/0.070
P2S (LLM)
GPT-5.6-Sol0.054/0.0420.019/0.0150.029/0.0230.062/0.0490.040/0.031
Claude-Fable-50.064/0.0500.022/0.0170.044/0.0330.077/0.0610.050/0.039
Qwen3.6 27B0.004/0.0030.044/0.035-0.009/-0.0070.038/0.0290.023/0.018
Gemini-3.6-Flash
Qwen3-VL-8B (P2S)
Scene-level MLLM — 2s
Qwen3-VL-8B0.459/0.4190.186/0.1720.333/0.2990.225/0.2020.298/0.272
InternVL3-8B0.402/0.3700.128/0.1150.370/0.3390.199/0.1810.263/0.240
GPT 5.6 Luna0.471/0.4250.244/0.2260.359/0.3190.244/0.2200.327/0.296
Scene-level MLLM — KTS
Qwen3-VL-8B0.470/0.4250.210/0.1960.359/0.3220.198/0.1810.311/0.283
InternVL3-8B0.452/0.4170.122/0.1150.318/0.2900.182/0.1700.268/0.247
GPT 5.6 Luna0.463/0.4190.252/0.2330.392/0.3500.291/0.2650.344/0.312
Full-video MLLM — 2s scoring
Gemini 3.1 Pro0.099/0.0960.103/0.0990.167/0.1560.041/0.0390.100/0.095
Qwen3-VL-8B0.139/0.1260.069/0.0630.125/0.1120.153/0.1400.116/0.106
InternVL3-8B-0.008/-0.0090.113/0.107-0.058/-0.0560.032/0.0280.036/0.033
Full-video MLLM — KTS scoring
Gemini 3.1 Pro0.134/0.1290.159/0.1530.132/0.1220.053/0.0510.127/0.121
Qwen3-VL-8B0.324/0.2950.034/0.0320.302/0.2750.157/0.1410.181/0.165
InternVL3-8B0.158/0.1470.097/0.0910.139/0.1230.016/0.0160.105/0.097
Lowest category frequency (ρ/τ)1/21 (4.8%)9/21 (42.9%)3/21 (14.3%)7/21 (33.3%)

We thank all reviewers for their constructive feedback and for engaging with us throughout the discussion. Your comments helped us strengthen the experiments and clarify what the results support.

We thank Reviewer L66f for asking us to clarify the evaluation protocol and for encouraging us to incorporate the rebuttal results into the manuscript.

We thank Reviewer jrc4 for suggesting independent human validation and asking whether the temporal reasoning gap persists when models receive the full video.

We thank Reviewer WYTG for prompting a more detailed comparison with related benchmarks and asking us to clarify how the random baselines differ.

We have uploaded a revised manuscript that incorporates the independent human validation, additional baseline and full-video experiments, and evaluation protocol clarifications discussed in the rebuttal. We also cross-checked the reported results and corrected numerical inconsistencies and typographical errors.

Dear reviewers,

As we wrap up the rebuttal period, we would like to thank all three reviewers for their feedback, which helped us further strengthen our work.

We are also pleased that all three reviewers reached a consensus and recommended Weak Acceptance, each giving our paper a score of 3 with high confidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.