- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper tackles the problem of video summarization by considering summary composition as part of the problem. Instead of relying on frame importance scores followed by post-processing pipelines like KTS segmentation and knapsack selection, the authors introduce an offline pipeline for constructing pseudo summaries as synthetic supervision targets. These synthetic targets are used for training an auto-regressive VL model called SumComposer that predicts segment boundaries conditioned on previous choices. At inference time, it can be used to predict summaries from video inputs, thus resulting in better segment matching and boundary alignment as compared to fixed composition.
Critical strengths
- Providing the evaluation and task disconnect in terms of low correlation between frame importance scores and summary F1.
- Offline pipeline that transforms frame importance scores and structured descriptions from videos via VLM/LLMs into segment level summaries.
- Autoregressive decoder based VL model called SumComposer that predicts start and end timings of intervals based on previously selected segment embeddings.
- Test time prediction of summaries from visual features without any additional post-processing pipelines.
Critical weaknesses
- No quality verification of the synthetic targets which are used as ground truths.
- Absence of any human preference study on the final summaries generated by SumComposer.
- No strong zero-shot baselines using frontier VLMs to summarize videos in a single pass.
- The metrics and are defined against synthetic targets in the form of pseudo-summaries.
- Since there is a budget associated with the segment decoding, the process can stop prematurely if a predicted segment is slightly too long.
- Further since the segment boundaries are defined by local gradient peaks of frame importance scores, any sub-optimal segment split is permanently baked into training targets without any possibility of timestamp modification.
Questions for authors
- In the segment estimation from the raw videos, the authors use the frame importance signal to determine the variations via temporal gradients. Have the authors considered segmenting the videos into shots and further combining into semantic scenes as prior segments ? It will be good to check if semantic scenes provide better initial segments.
- In the synthetic data generation process, have the authors considered other models for creating pseudo summary targets ? If yes, is there any tradeoff between quality and latency in data generation process ? It is not clear if there is any significant quality tradeoff in using frontier models vs smaller + faster open source models.
- How much does the target alignment quality (as measured by Smatch and Salign) degrades if lighter teacher model is used ?
- Why does the performance of SumComposer drop as compared to fixed composition methods when older backbones like VASNet, Joint-VA and PGL-SUM are used ?
7 comments
Response to Reviewer xMcQ (2/4)
W4. The metrics and are defined against synthetic targets in the form of pseudo-summaries.
A. We clarify that the proposed metrics are not restricted to synthetic references. Both metrics compare a predicted temporal summary with a reference summary, regardless of how that reference is obtained. The reference sources we use differ by datasets:
- SumMe: Temporal summaries directly annotated by humans.
- TVSum: Temporal references constructed from human-rated importance scores, greedily selecting the frames until the 15% budget is filled. No KTS+knapsack used.
- Mr. HiSum: We use constructed pseudo-summary targets because human-authored temporal summaries are not available in this dataset.
Thus, Tab. 1 uses human-created summaries on SumMe and temporal targets derived from human importance ratings on TVSum. Only the Mr. HiSum scores reported in Tabs. 2 and 3 measure agreement with the constructed pseudo-summaries. We clarified this distinction in Sec. 5.1 of the revised paper.
We further examined whether these metrics capture temporal errors that frame-level F1 can overlook. On the human-labeled summaries in SumMe, we applied controlled perturbations to segment positions, boundaries, and fragmentation. When continuous clips were divided into shorter pieces while preserving most selected frames, the F1 metric remained nearly unchanged, whereas decreased sharply (See App. F (Tab. XI) for the full results). This result illustrates the complementary roles of the metrics: the segment matching captures differences in segment structure beyond frame-level overlap, while the boundary alignment measures the placement of individual segment boundaries.
W5. The process can stop prematurely if a predicted segment is slightly too long.
A. We agree that the current stopping rule can leave part of the summary budget unused. When a predicted segment exceeds the remaining budget, the decoding process indeed terminates with the previously accepted segments.
To examine the impact of this limitation, we compared the original rule with a variant that allows the last segment to be partially included into the remaining budget. (In such a case, we retained a centered subinterval whose duration matches the remaining budget.) Both used the same SumComposer checkpoint with the TripleSumm backbone and applied the same 15% duration budget.
Clipping increased average budget utilization from 80% to 100%, while decreased from 21.94 to 21.23. In other words, greater budget utilization did not improve segment matching. We therefore retain the original stopping rule while acknowledging that it may leave part of the summary budget unused.
We clarified this behavior in Sec. 4.3 and added the comparison in App. E.2 (Tab. IX) of the revised paper.
Response to Reviewer xMcQ (3/4)
W6 & Q1. Fixed candidate boundaries may limit pseudo-summary quality.
A. We agree that the selection LLM cannot modify boundaries within a predefined candidate. As it is beyond the scope of this paper to propose a novel approach to allow segment modification, we conducted two additional experiments to examine how sensitive the pseudo-summary quality is to the candidate granularity and the source of candidate boundaries.
Candidate granularity. We compared the default segments with finer-grained ones by splitting each segment into two, assuming that finer-grained segmentation would potentially mitigate the limitations imposed by the rigidly fixed segments. Both settings use the same captioner, selection model, prompt, importance scores, and 15% duration budget. We evaluated all 25 SumMe videos against human annotations. The table below reports the results.
| Candidate segmentation | Evaluation level | Budget utilization | |||
|---|---|---|---|---|---|
| Twice fine-grained | Video level | 73% | 56.3 | 45.0 | 47.1 |
| Original | Video level | 96% | 61.9 | 54.7 | 49.9 |
| Twice fine-grained | User level | 73% | 23.2 | 31.3 | 29.1 |
| Original | User level | 96% | 27.0 | 31.8 | 31.8 |
Surprisingly, the finer-grained candidates did not improve alignment with human summaries. They also reduced budget utilization under the same selection procedure.
Boundary sources. We additionally compared five segmentation sources at a matched density of 16 candidate boundaries per minute. The comparison includes PySceneDetect, an open-source tool for detecting shot boundaries, and TransNetV2 [1], a deep neural network for shot transition detection.
For this diagnostic, the importance signal was derived from other human annotations of each video, excluding the reference annotator under a leave-one-out setting. The table below reports boundary detection and summary-alignment results.
| Boundary source | Recall@1s | Precision@1s | |||
|---|---|---|---|---|---|
| Uniform intervals | 0.54 | 0.14 | 73.4 | 64.9 | 66.0 |
| PySceneDetect | 0.40 | 0.14 | 52.3 | 48.5 | 48.6 |
| TransNetV2 | 0.44 | 0.15 | 54.8 | 49.8 | 53.4 |
| KTS | 0.51 | 0.16 | 65.8 | 58.1 | 62.2 |
| Importance-gradient (Ours) | 0.69 | 0.21 | 75.6 | 74.6 | 73.8 |
The importance-gradient boundaries, our default setting, achieved the strongest alignment among the tested sources.
Putting these two experiments together, we claim that our default settings were at least reasonable. We believe the reviewer's proposed shot-to-semantic-scene aggregation remains worth investigating.
We clarified the fixed-candidate limitation in Sec. 3 of the revised paper, and added the candidate granularity and boundary-source experiments in App. D.6 (Tab. V) and App. D.7 (Tab. VI), respectively.
[1] Soucek et al. Transnet v2: An effective deep network architecture for fast shot transition detection. ACM MM 2024.
Response to Reviewer xMcQ (1/4)
We thank the reviewer for the detailed feedback on pseudo-summary quality, evaluation references, and the limitations of our composition pipeline. We conducted additional human evaluations and controlled experiments to address these concerns.
W1. No quality verification of the synthetic targets which are used as ground truths.
A. Tab. 1 in the paper evaluates pseudo-summary quality against human annotations on SumMe and TVSum. Our constructed targets achieve higher segment-matching and boundary-alignment scores than KTS+knapsack on both datasets.
To further validate their quality, we conducted a blind human preference study with six participants on five randomly selected SumMe videos. Participants watched each original video, and then two anonymized summaries, one produced by our method and another by KTS+knapsack applied on the ground-truth importance scores, in random order. They were then asked to answer which one they prefer in terms of content coverage, temporal coherence, and overall preference. Both methods used the same 15% duration budget. The table below reports preference rates from 30 judgments per criterion.
Our pseudo-summaries received more preference judgments than the baseline across all three criteria. These results provide complementary evidence of their quality beyond agreement with human annotations.
We added this user study in App. D.8 (Tab. VII) of the revised paper.
W2. Absence of any human preference study on the final summaries generated by SumComposer.
A. To address this concern, we conducted a separate blind preference study on the final summaries generated by SumComposer. Six participants evaluated five randomly selected Mr. HiSum videos using the same protocol as W1. Unlike W1, however, this experiment evaluates the final model outputs rather than the pseudo-summary training targets. Both SumComposer and fixed KTS+knapsack composition used TripleSumm as the temporal backbone and followed the same 15% duration budget. The table below reports preference rates from 30 judgments per criterion.
Participants favored the summaries produced by SumComposer in content coverage, temporal coherence, and overall preference.
We added this study in App. E.3 (Tab. X) of the revised paper.
W3. No strong zero-shot baselines using frontier VLMs to summarize videos in a single pass.
A. Following the suggestion, we added a direct zero-shot video summarization baseline using Qwen3.8-27B. The model receives video frames sampled at 1 fps, their timestamps, and a 15% summary-duration budget. It is prompted to return important, nonoverlapping intervals with start and end timestamps. The full prompt and the post-processing procedure used to enforce the 15% duration cap are provided in App. C.1 of the revised paper. We evaluated this baseline on the same Mr. HiSum test split, using the same constructed targets and evaluation metrics as the main comparison. The table below reports the results.
SumComposer achieves consistently higher agreement with the constructed temporal targets across all three metrics.
Please see Tab. 3 in Sec. 5.3 of the revised paper for the added zero-shot comparison.