- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CSSI, an efficient long-video understanding framework that represents selected keyframes with spatial detail tokens and compresses other frames into single timeline tokens. It introduces a query-conditioned autoregressive frame selector trained with an answer-margin reward. Experiments demonstrate improvements on several long-video QA benchmarks.
Critical strengths
- CSSI achieves consistent improvements over ViLAMP on several long-video QA benchmarks, particularly on longer videos.
- The paper provides extensive evaluations, including frame-selection comparisons, needle-in-a-haystack experiments, and efficiency scaling analysis.
Critical weaknesses
1. Limited novelty compared with existing methods.
The mixed-resolution representation emphasized as a main contribution has already been introduced by ViLAMP, which retains detailed tokens for selected frames and compresses unselected frames into single tokens. Moreover, ReFoCUS has already explored autoregressive frame selection trained with an answer-margin reward and policy-gradient optimization. The paper should better clarify its methodological differences and contributions beyond these existing approaches.
2. Insufficient experimental validation.
- Table 1 lacks comprehensive comparisons with existing keyframe-selection methods on long-video QA benchmarks. Table 2 only provides limited comparisons on Video-MME.
- An ablation removing the timeline tokens is missing, making their contribution unclear.
- Although Figure 1 reports latency and memory, a detailed inference-time breakdown and direct efficiency comparison with ViLAMP are missing.
- The autoregressive selection strategy provides only a 0.5-point improvement over independent ranking in Table 5, raising questions about its effectiveness.
3. Unclear method presentation.
Figure 2 does not clearly illustrate the autoregressive frame-selection process, particularly how previous selections influence subsequent decisions.
Questions for authors
Q1. Novelty relative to ViLAMP and ReFoCUS. What are the precise methodological differences between CSSI and these two existing works? ViLAMP already introduces mixed-resolution video representation, while ReFoCUS proposes autoregressive frame selection with a prediction-margin reward and group-relative policy optimization. Could the authors explain which components are fundamentally new in CSSI and provide controlled experiments demonstrating their contributions?
Q2. Missing comparisons and ablations. Could the authors compare CSSI with established keyframe-selection methods, especially learned or RL-based selectors, across the full benchmarks in Table 1 rather than only the limited categories in Table 2? For the ViLAMP comparison, does the baseline retain its original differential feature merging, or is it replaced by average pooling? Additionally, could the authors report an ablation without timeline tokens, keeping the selected keyframes and their spatial resolution unchanged, to clarify the contribution of the mixed-resolution representation?
Q3. Training data provenance and reward design. CSSI reports approximately 962K candidate QA pairs and 393K filtered training samples, identical to the numbers reported in ReFoCUS. Were these data independently collected and filtered, or was any existing dataset or pretrained selector reused? Additionally, how does the reward in Equation (12) differ substantively from the normalized answer-margin reward in ReFoCUS?
Q4. Inference efficiency. Could the authors report a detailed inference-time breakdown, including visual encoding, selection-policy execution, visual-token construction, and language-model inference? How do the additional costs introduced by the 24-layer Gated DeltaNet-2 selector compare with ViLAMP under the same hardware, input lengths, and visual-token budgets? Including ViLAMP in the efficiency scaling comparison would be particularly informative.
Q5. Autoregressive selection mechanism. Figure 2 does not clearly show how previously selected frames affect subsequent selection decisions. It also suggests a different input ordering from Section 3.3. Could the authors clarify the exact execution sequence? Given the relatively small improvement of autoregressive selection over independent ranking in Table 5 (64.9 vs. 64.4), is this difference statistically significant and consistent across multiple benchmarks?