acceptodds
Under review as a conference paper at ICLR 2027

Do Not Stop at Frame Scoring: Learning to Compose Video Summaries

Abstract

Video summarization is a task to extract a sequence of segments to preserve the key content of a video. Recent work on this task tends to focus only on predicting frame importance scores, leaving the segmentation and budget-constrained selection as a fixed pipeline. However, our empirical analysis reveals that a more precise importance score does not reliably translate into a better summary, suggesting that summary composition itself should be trained. Directly learning to compose a summary, however, requires structured summary-level supervision, which is prohibitive to obtain at scale. To address this bottleneck, we introduce offline pseudo-summary construction, transforming existing frame-importance annotations and video content into a structured, budget-aware pseudo-summary. We then introduce SumComposer, a model to autoregressively add a segment to the summary, conditioned on preceding selections. Once trained, it requires neither importance annotations nor language model calls to summarize a video. Our experiments verify that our pseudo-summaries are better aligned with human annotations than fixed construction, and SumComposer trained on these pseudo-labels substantially improves matching and boundary alignment over fixed composition.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video summarization, Video understanding, Temporal modeling

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_xMcQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles the problem of video summarization by considering summary composition as part of the problem. Instead of relying on frame importance scores followed by post-processing pipelines like KTS segmentation and knapsack selection, the authors introduce an offline pipeline for constructing pseudo summaries as synthetic supervision targets. These synthetic targets are used for training an auto-regressive VL model called SumComposer that predicts segment boundaries conditioned on previous choices. At inference time, it can be used to predict summaries from video inputs, thus resulting in better segment matching and boundary alignment as compared to fixed composition.

Critical strengths
  • Providing the evaluation and task disconnect in terms of low correlation between frame importance scores and summary F1.
  • Offline pipeline that transforms frame importance scores and structured descriptions from videos via VLM/LLMs into segment level summaries.
  • Autoregressive decoder based VL model called SumComposer that predicts start and end timings of intervals based on previously selected segment embeddings.
  • Test time prediction of summaries from visual features without any additional post-processing pipelines.
Critical weaknesses
  • No quality verification of the synthetic targets which are used as ground truths.
  • Absence of any human preference study on the final summaries generated by SumComposer.
  • No strong zero-shot baselines using frontier VLMs to summarize videos in a single pass.
  • The metrics SmatchS_{match} and SalignS_{align} are defined against synthetic targets in the form of pseudo-summaries.
  • Since there is a budget associated with the segment decoding, the process can stop prematurely if a predicted segment is slightly too long.
  • Further since the segment boundaries are defined by local gradient peaks of frame importance scores, any sub-optimal segment split is permanently baked into training targets without any possibility of timestamp modification.
Questions for authors
  • In the segment estimation from the raw videos, the authors use the frame importance signal to determine the variations via temporal gradients. Have the authors considered segmenting the videos into shots and further combining into semantic scenes as prior segments ? It will be good to check if semantic scenes provide better initial segments.
  • In the synthetic data generation process, have the authors considered other models for creating pseudo summary targets ? If yes, is there any tradeoff between quality and latency in data generation process ? It is not clear if there is any significant quality tradeoff in using frontier models vs smaller + faster open source models.
  • How much does the target alignment quality (as measured by Smatch and Salign) degrades if lighter teacher model is used ?
  • Why does the performance of SumComposer drop as compared to fixed composition methods when older backbones like VASNet, Joint-VA and PGL-SUM are used ?
7 comments
AuthorsOpenReview ↗

Response to Reviewer xMcQ (1/4)

We thank the reviewer for the detailed feedback on pseudo-summary quality, evaluation references, and the limitations of our composition pipeline. We conducted additional human evaluations and controlled experiments to address these concerns.


W1. No quality verification of the synthetic targets which are used as ground truths.

A. Tab. 1 in the paper evaluates pseudo-summary quality against human annotations on SumMe and TVSum. Our constructed targets achieve higher segment-matching and boundary-alignment scores than KTS+knapsack on both datasets.

To further validate their quality, we conducted a blind human preference study with six participants on five randomly selected SumMe videos. Participants watched each original video, and then two anonymized summaries, one produced by our method and another by KTS+knapsack applied on the ground-truth importance scores, in random order. They were then asked to answer which one they prefer in terms of content coverage, temporal coherence, and overall preference. Both methods used the same 15% duration budget. The table below reports preference rates from 30 judgments per criterion.

ComparisonContent coverageTemporal coherenceOverall preference
Prefer our pseudo-summary50.056.763.3
Neutral30.026.713.3
Prefer KTS+knapsack on GT20.016.723.3

Our pseudo-summaries received more preference judgments than the baseline across all three criteria. These results provide complementary evidence of their quality beyond agreement with human annotations.
We added this user study in App. D.8 (Tab. VII) of the revised paper.


W2. Absence of any human preference study on the final summaries generated by SumComposer.

A. To address this concern, we conducted a separate blind preference study on the final summaries generated by SumComposer. Six participants evaluated five randomly selected Mr. HiSum videos using the same protocol as W1. Unlike W1, however, this experiment evaluates the final model outputs rather than the pseudo-summary training targets. Both SumComposer and fixed KTS+knapsack composition used TripleSumm as the temporal backbone and followed the same 15% duration budget. The table below reports preference rates from 30 judgments per criterion.

ComparisonContent coverageTemporal coherenceOverall preference
Prefer our summary53.360.053.3
Neutral10.010.010.0
Prefer KTS+knapsack on importance scores estimated by TripleSumm36.730.036.7

Participants favored the summaries produced by SumComposer in content coverage, temporal coherence, and overall preference.
We added this study in App. E.3 (Tab. X) of the revised paper.


W3. No strong zero-shot baselines using frontier VLMs to summarize videos in a single pass.

A. Following the suggestion, we added a direct zero-shot video summarization baseline using Qwen3.8-27B. The model receives video frames sampled at 1 fps, their timestamps, and a 15% summary-duration budget. It is prompted to return important, nonoverlapping intervals with start and end timestamps. The full prompt and the post-processing procedure used to enforce the 15% duration cap are provided in App. C.1 of the revised paper. We evaluated this baseline on the same Mr. HiSum test split, using the same constructed targets and evaluation metrics as the main comparison. The table below reports the results.

MethodSmatchS_{\mathrm{match}}BalignstartB_{\mathrm{align}}^{\mathrm{start}}BalignendB_{\mathrm{align}}^{\mathrm{end}}
Qwen3.8-27B9.2014.309.90
SumComposer + UMT17.6326.2518.79
SumComposer + TripleSumm21.9428.5520.49

SumComposer achieves consistently higher agreement with the constructed temporal targets across all three metrics.
Please see Tab. 3 in Sec. 5.3 of the revised paper for the added zero-shot comparison.

AuthorsOpenReview ↗

Response to Reviewer xMcQ (2/4)

W4. The metrics SmatchS_{match} and BalignB_{align} are defined against synthetic targets in the form of pseudo-summaries.

A. We clarify that the proposed metrics are not restricted to synthetic references. Both metrics compare a predicted temporal summary with a reference summary, regardless of how that reference is obtained. The reference sources we use differ by datasets:

  • SumMe: Temporal summaries directly annotated by humans.
  • TVSum: Temporal references constructed from human-rated importance scores, greedily selecting the frames until the 15% budget is filled. No KTS+knapsack used.
  • Mr. HiSum: We use constructed pseudo-summary targets because human-authored temporal summaries are not available in this dataset.

Thus, Tab. 1 uses human-created summaries on SumMe and temporal targets derived from human importance ratings on TVSum. Only the Mr. HiSum scores reported in Tabs. 2 and 3 measure agreement with the constructed pseudo-summaries. We clarified this distinction in Sec. 5.1 of the revised paper.

We further examined whether these metrics capture temporal errors that frame-level F1 can overlook. On the human-labeled summaries in SumMe, we applied controlled perturbations to segment positions, boundaries, and fragmentation. When continuous clips were divided into shorter pieces while preserving most selected frames, the F1 metric remained nearly unchanged, whereas SmatchS_{match} decreased sharply (See App. F (Tab. XI) for the full results). This result illustrates the complementary roles of the metrics: the segment matching captures differences in segment structure beyond frame-level overlap, while the boundary alignment measures the placement of individual segment boundaries.


W5. The process can stop prematurely if a predicted segment is slightly too long.

A. We agree that the current stopping rule can leave part of the summary budget unused. When a predicted segment exceeds the remaining budget, the decoding process indeed terminates with the previously accepted segments.

To examine the impact of this limitation, we compared the original rule with a variant that allows the last segment to be partially included into the remaining budget. (In such a case, we retained a centered subinterval whose duration matches the remaining budget.) Both used the same SumComposer checkpoint with the TripleSumm backbone and applied the same 15% duration budget.

Clipping increased average budget utilization from 80% to 100%, while SmatchS_{match} decreased from 21.94 to 21.23. In other words, greater budget utilization did not improve segment matching. We therefore retain the original stopping rule while acknowledging that it may leave part of the summary budget unused.
We clarified this behavior in Sec. 4.3 and added the comparison in App. E.2 (Tab. IX) of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer xMcQ (3/4)

W6 & Q1. Fixed candidate boundaries may limit pseudo-summary quality.

A. We agree that the selection LLM cannot modify boundaries within a predefined candidate. As it is beyond the scope of this paper to propose a novel approach to allow segment modification, we conducted two additional experiments to examine how sensitive the pseudo-summary quality is to the candidate granularity and the source of candidate boundaries.

Candidate granularity. We compared the default segments with finer-grained ones by splitting each segment into two, assuming that finer-grained segmentation would potentially mitigate the limitations imposed by the rigidly fixed segments. Both settings use the same captioner, selection model, prompt, importance scores, and 15% duration budget. We evaluated all 25 SumMe videos against human annotations. The table below reports the results.

Candidate segmentationEvaluation levelBudget utilizationSmatchS_{match}BalignstartB_{align}^{start}BalignendB_{align}^{end}
Twice fine-grainedVideo level73%56.345.047.1
OriginalVideo level96%61.954.749.9
Twice fine-grainedUser level73%23.231.329.1
OriginalUser level96%27.031.831.8

Surprisingly, the finer-grained candidates did not improve alignment with human summaries. They also reduced budget utilization under the same selection procedure.

Boundary sources. We additionally compared five segmentation sources at a matched density of 16 candidate boundaries per minute. The comparison includes PySceneDetect, an open-source tool for detecting shot boundaries, and TransNetV2 [1], a deep neural network for shot transition detection.

For this diagnostic, the importance signal was derived from other human annotations of each video, excluding the reference annotator under a leave-one-out setting. The table below reports boundary detection and summary-alignment results.

Boundary sourceRecall@1sPrecision@1sSmatchS_{match}BalignstartB_{align}^{start}BalignendB_{align}^{end}
Uniform intervals0.540.1473.464.966.0
PySceneDetect0.400.1452.348.548.6
TransNetV20.440.1554.849.853.4
KTS0.510.1665.858.162.2
Importance-gradient (Ours)0.690.2175.674.673.8

The importance-gradient boundaries, our default setting, achieved the strongest alignment among the tested sources.

Putting these two experiments together, we claim that our default settings were at least reasonable. We believe the reviewer's proposed shot-to-semantic-scene aggregation remains worth investigating.
We clarified the fixed-candidate limitation in Sec. 3 of the revised paper, and added the candidate granularity and boundary-source experiments in App. D.6 (Tab. V) and App. D.7 (Tab. VI), respectively.

[1] Soucek et al. Transnet v2: An effective deep network architecture for fast shot transition detection. ACM MM 2024.

Reviewer_K3yvOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that better frame-importance prediction does not necessarily yield better video summaries, and that segment selection and boundaries should therefore be learned directly. The authors partition each video into candidate segments using the importance signal, construct offline pseudo-summaries via VLM descriptions and LLM-based ranking, and then train SumComposer to autoregressively predict segment start and end positions. The pseudo-summaries are compared with human annotations on SumMe/TVSum. The trained model is evaluated mainly on Mr. HiSum, against reference sequences produced by the same pseudo-summary construction pipeline.

Critical strengths

S1. Clear and practically meaningful motivation. Moving attention from frame scoring to the composition of the final summary is a reasonable direction. The metric-correlation analysis and the fixed-composition case study in Section 2 provide intuitive motivation, and the paper also acknowledges earlier methods that compose summaries directly.

S2. Reasonable empirical and controlled analyses. Table 1 and Appendix D show that the pseudo-summaries align with human annotations better than the fixed pipeline does. Table 2 covers seven encoders and honestly reports the degradation on earlier encoders. Appendix E provides a 2×2 controlled comparison of target source × composer, so the paper cannot be said to lack ablations entirely.

Critical weaknesses

W1. The contribution of the diversity instruction to reducing redundancy is not verified. The paper argues that the value of a segment depends on what has already been selected. Once several similar segments are in the summary, adding another similar one mostly increases repetition and contributes little (Sections 1-2). In the pseudo-summary construction, this is handled only through the "COVERAGE & DIVERSITY" instruction in the LLM prompt (Appendix B.3). However, I could not find any experiment testing whether this instruction actually reduces redundancy. diversity instruction, and no measure of semantic repetition. In fact, some selected candidates in these examples have nearly identical descriptions (e.g., Rank 4 and Rank 6 in Example 2), so the examples alone cannot demonstrate reduced redundancy.

W2. Pre-defined candidate segments may limit the redundancy-reduction effect of the pseudo-summaries. As described on page 4 and in Appendix B (pp. 15-16), the candidate boundaries are fixed in advance by the frame-importance signal. The LLM selects among candidates based on their descriptions, scores, and durations, and cannot generate new interval boundaries. If an important event lies within the same candidate segment as repetitive or irrelevant content, the LLM can only keep or discard the whole segment, even if it recognizes the redundancy, and cannot trim the redundant part. Ranking and merging adjacent segments do not directly address this issue.

Questions for authors

Q1. Given that the candidate segments, importance scores, language model, and summary length budget are all identical, can the authors compare pseudo-summaries generated with and without diversity requirements? Could they supplement the experiment with an evaluation of semantic redundancy and report segment-matching performance to verify whether redundancy can be reduced while retaining important content?

Q2. Have the authors examined cases where important events and redundant content appear in the same candidate segment? On a representative subset, using the same model, prompt selection, and length budget, can the authors compare the original candidate segment divisions with finer-grained divisions (or those allowing for local boundary optimization)? Evaluating coverage of important content, semantic redundancy, and consistency with human references under this setup would help quantify the limitations imposed by pre-fixed candidate segment boundaries.

3 comments
AuthorsOpenReview ↗

Response to Reviewer K3yv

We thank the reviewer for the comments on diversity-aware selection and the limitations of predefined candidate segments. Following the suggestions, we conducted controlled experiments to examine semantic redundancy and alignment with human summaries.


W1 & Q1. The effect of diversity guidance on semantic redundancy has not been verified.

A. Following the reviewer's suggestion, we compared on all 25 SumMe videos pseudo-summaries generated using three variants of selection prompts: removing the opening instruction and the “COVERAGE & DIVERSITY” guideline while retaining all other selection guidelines (Base), emphasizing informative moments (Critical), and using our coverage-and-diversity guidance (Diversity). We kept the candidate segments, visual descriptions, importance scores, selection model (gpt-oss-120b), and 15% duration budget fixed. We measured alignment against human annotations and used Qwen3.8-27B to quantify semantic redundancy among selected candidates. The table below reports the results.

InstructionSmatchS_{match}BalignstartB_{align}^{start}BalignendB_{align}^{end}Candidate-level redundancy (%)
Base63.358.553.28.6
Critical59.657.652.110.9
Diversity61.954.749.95.9

Among the tested prompt variants, Diversity yields the lowest semantic redundancy, while Base achieves higher alignment scores, revealing a tradeoff between redundancy and temporal alignment.
Please see App. D.5 (Tab. IV) of the revised paper for the prompt comparison and candidate-level redundancy evaluation.


W2 & Q2. Predefined candidate segments can limit redundancy reduction.

A. As suggested, we compared the original candidate segmentation with a finer-grained alternative to examine the effect of fixed boundaries. The finer-grained setting divides each candidate into two shorter intervals. Both settings use the same captioner, selection model, prompt, importance signal, and 15% duration budget.

The table below reports alignment with human annotations and average budget utilization across all 25 SumMe videos.

Candidate segmentationEvaluation levelBudget utilizationSmatchS_{match}BalignstartB_{align}^{start}BalignendB_{align}^{end}
Twice fine-grainedVideo level73%56.345.047.1
OriginalVideo level96%61.954.749.9
Twice fine-grainedUser level73%23.231.329.1
OriginalUser level96%27.031.831.8

The original segmentation achieves better alignment under the tested selection procedure, while finer segmentation reduces average budget utilization from 96% to 73%. In addition to the results reported in the table, finer segmentation increases semantic redundancy from 4.3% to 10.9%, where the redundancy is measured between two clips in the composed pseudo-summary.

We acknowledge that fixed candidates impose a limitation, since the selection LLM cannot remove redundant content within a retained segment. Our experiment compares two levels of granularity, and direct local boundary optimization remains worth investigating. In a separate leave-one-annotator-out diagnostic, our importance-gradient boundaries achieved the highest alignment among five boundary sources at a matched density of 16 boundaries per minute (App. D.7, Tab. VI).
Please see Sec. 3 and App. D.6 (Tab. V) of the revised paper for the limitations of fixed candidates and our experiments on candidate granularity.

It seems like the “base” version is more balanced? That sounds interesting. I'll keep the current rating and increase the confidence level.

We thank the reviewer for this thoughtful follow-up. As the reviewer mentioned, we agree that the ultimate goal is overall summary quality under the duration budget. We clarify, however, that low redundancy is one factor to constitute a good summary, but high relevance of the participating segments also does matter.

Diversity aims to focus more on reducing redundancy, even if it has to compromise relevance in some cases. Tab. IV shows that it achieves the lowest redundancy (5.9%), as intended. Critical is the opposite option: focusing more on choosing a more informative one, even if it might have some overlap. As expected, it reports the highest redundancy (10.9%). Base aims at balancing between the two extremes, equally considering both high relevance and low redundancy. Again, as expected, its redundancy is between Diversity and Critical (8.6%).

Tab. IV shows that Base achieves higher alignment with human annotations across all three metrics (SmatchS_{\mathrm{match}}, BalignstartB_{\mathrm{align}}^{\mathrm{start}}, and BalignendB_{\mathrm{align}}^{\mathrm{end}}) than Diversity or Critical does. These results are consistent with a balanced tradeoff between the two conflicting goals, having more relevant contents and reducing redundancy among them. In other words, since the fixed duration budget constrains how much content can be retained, redundancy reduction should be considered together with its effect on content preservation.

However, we do not claim that Base is the best option in general. The experimental results above illustrate a single case of this balance on a particular dataset with particular prompts for pseudo-summary construction, but the actual degree of emphasis on each goal should be tuned individually to achieve optimal balance on each circumstance. We believe that human alignment is also not a perfect single metric to assess the quality of a summary, and evaluation of video summarization is still an open problem.

We have revised App. D.5 to make this interpretation explicit. We have also revised the discussion of Tab. 1 in Sec. 5.2 to clarify that the reported gains characterize the complete construction pipeline. We thank the reviewer again for deeply engaging with this discussion.

Reviewer_EZE4OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes a method that composes a video summary. Instead of training a frame importance predictor, this work develops a composer that autoregressively predicts the next segment to be added. To train the composer, pseudo summaries are generated with a VLM and an LLM. The empirical results demonstrate that the proposed method outperforms existing methods.

Critical strengths
  • This paper describes a critical issue in video summarization well: a common pipeline for composing a summary throws frame-level importance scores away. The limitation of video summarization is empirically validated, and the method is clearly motivated by the result.
  • The design of the proposed method is reasonable.
Critical weaknesses
  • The experimental results depend on newly proposed measures, but the measures are not validated. The design of the measures seems reasonable, but validating whether they behave as intended would make the results more convincing. For instance, we could check whether the measures can identify clearly bad or good summaries, or check how the measure behaves under perturbations to a summary.
  • My main concern is that the reference summaries used for evaluation also depend on the problematic pipeline (frame importance + KTS + knapsack). As discussed in the paper, this pipeline tends to ignore frame importance, so the composed reference summaries may not reflect human judgment and may not be proper gold-standard summaries. I understand that preparing valid reference summaries is difficult in practice, not only because it is costly, but also because defining a good summary is itself a complicated problem. Providing an explanation for this point would help readers interpret the results.
Questions for authors
  • Please provide validation for the newly introduced measures.
  • Explain how using summary datasets as gold standards can be justified, despite their dependence on the pipeline that this paper criticizes.
5 comments
AuthorsOpenReview ↗

Response to Reviewer EZE4 (1/2)

We thank the reviewer for the suggestions on metric validation and the reliability of evaluation references. We conducted additional experiments to examine the behavior of the proposed metrics and clarify how references are constructed for each dataset.


W1 & Q1. The proposed evaluation metrics have not been validated.

A. Following the reviewer's suggestion, we conducted a controlled perturbation study to examine whether the proposed metrics distinguish different types of changes in temporal structure.

Specifically, we used 384 human-created summaries from all 25 SumMe videos and applied three perturbations: relocating selected clips, shifting their boundaries, and fragmenting continuous clips into shorter segments. The degree of perturbation α\alpha ranges from 0 to 1, and the reported results are averaged over ten random initializations.

We evaluated the perturbed summaries against their original human references in the tables below:

Content relocation

α\alphaSmatchS_{match}BalignstartB_{align}^{start}BalignendB_{align}^{end}F1
0.0100.0100.0100.0100.0
0.190.190.290.290.2
0.370.770.770.770.8
0.549.649.649.649.5
1.00.00.20.20.0

Boundary perturbation

α\alphaSmatchS_{match}BalignstartB_{align}^{start}BalignendB_{align}^{end}F1
0.0100.0100.0100.0100.0
0.1100.098.898.995.0
0.398.586.285.985.1
0.575.870.169.974.9
1.029.344.049.652.2

Structural fragmentation

α\alphaSmatchS_{match}BalignstartB_{align}^{start}BalignendB_{align}^{end}F1
0.0100.0100.0100.0100.0
0.145.278.678.699.6
0.310.052.752.798.4
0.58.842.342.397.4
1.02.528.028.094.6

The metrics respond differently to the three perturbations. Content relocation drops all scores. Under smaller boundary shifts, boundary-alignment scores drop while segment matching remains largely preserved. Under fragmentation, frame F1 remains high because most selected frames are retained, whereas segment matching drops sharply as continuous segments are divided into fragments.

These results support the intended roles of our metrics: SmatchS_{match} measures segment-level correspondence, while boundary alignment evaluates the accuracy of the start and end points. They provide complementary information to the frame-level F1 to assess a composed summary.
Please see Sec. 5.1 and App. F (Tab. XI) of the revised paper for the metric definitions and controlled perturbation study.

AuthorsOpenReview ↗

Response to Reviewer EZE4 (2/2)

W2 & Q2. The evaluation references may depend on the problematic KTS+knapsack pipeline.

A. We appreciate the concern about the reliability of evaluation references. We clarify that none of our evaluation references are generated by the KTS+knapsack pipeline criticized in the paper. Their construction differs across datasets:

  • SumMe: Temporal summaries directly annotated by humans.
  • TVSum: Temporal references constructed from human-rated importance scores, selecting the highest-rated frames until the 15% budget is filled. No KTS+knapsack used.
  • Mr. HiSum: We use constructed pseudo-summary targets because human-authored temporal summaries are not available in this dataset.

Consequently, Tab. 1 uses human-created summaries on SumMe and temporal targets derived from human importance ratings on TVSum. The Mr. HiSum scores reported in Tabs. 2 and 3 measure agreement with our constructed pseudo-summary targets.

We also provide two complementary evaluations of summary quality. Tab. 1 compares our pseudo-summary construction against human annotations on SumMe and TVSum, where it achieves better segment matching and boundary alignment than the fixed KTS+knapsack composition.

To obtain direct human judgments of summary quality, we conducted two blind preference studies, one on five randomly selected SumMe videos and the other on five randomly selected Mr. HiSum videos. Each study involved six participants. Participants watched each original video before comparing two anonymized summaries presented in randomized order. Both methods applied the same 15% duration budget.

On SumMe, we compared our pseudo-summary targets with KTS+knapsack summaries constructed from ground-truth importance scores. On Mr. HiSum, we compared SumComposer's final outputs with fixed composition, using TripleSumm as the backbone.

The table below reports preference rates (%) from 30 judgments per criterion in each study.

DatasetComparisonContent coverageTemporal coherenceOverall preference
SumMePrefer our pseudo-summary50.056.763.3
Neutral30.026.713.3
Prefer KTS+knapsack on GT20.016.723.3
Mr. HiSumPrefer our summary53.360.053.3
Neutral10.010.010.0
Prefer KTS+knapsack on TripleSumm36.730.036.7

Our summaries received more preference judgments than the baseline across all three criteria in both studies, complementing the quantitative evaluations.
We revised Sec. 5.1 with the sources of the evaluation targets, and App. D.8 (Tab. VII) and App. E.3 (Tab. X) with human preference studies.

Thank you for your response and the additional perturbation study. I believe that observed responses to the three different perturbations validate the proposed measures to some extent.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Discussion

We thank all three reviewers for their time and effort in evaluating our work and for their thoughtful and constructive feedback. We are pleased that all three reviewers now recommend acceptance, with ratings of 4, 3, and 3. Below is a brief summary of the reviews and discussion.


Reviewer EZE4: The reviewer recognized the empirical motivation for learning summary composition and the reasonable design of SumComposer. During the discussion, we added a controlled perturbation study of the proposed metrics, clarified the sources and interpretation of the evaluation references, and provided complementary human evaluations. The reviewer stated that their main concern had been adequately addressed and raised their rating from 3 to 4.

Reviewer K3yv: The reviewer highlighted the practical motivation and controlled empirical analyses. During the discussion, we added prompt ablations with semantic-redundancy evaluation and candidate-granularity experiments, and clarified the limitations of fixed candidates. We also discussed the observed tradeoff between annotation alignment and model-assessed redundancy, and revised the discussion of Table 1 to clarify that its gains characterize the complete construction pipeline. The reviewer maintained their rating of 3 and increased their confidence to 4.

Reviewer xMcQ: The reviewer recognized our analysis of the limitations of frame scoring, offline pseudo-summary construction, and autoregressive summary composition. During the discussion, we added human preference studies on both pseudo-summary targets and final outputs, a zero-shot VLM baseline, and analyses of the stopping rule, candidate boundaries, and teacher-model quality and latency. The reviewer raised their rating from 2 to 3.


The additional experiments and clarifications have been incorporated into the revised manuscript. Following the remaining suggestions, we will further clarify the backbone-related discussion and add qualitative examples of pseudo-summaries preferred by participants alongside the corresponding baseline summaries in the final version.

Thank you again for your time and careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.