acceptodds
Withdrawn from ICLR 2027

Compressed Sight, Selective Insight: Extreme Long Video Language Models That Reward Their Own Gaze

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-video understanding is constrained by a visual-budget problem: a video may span thousands of frames, but only a small fraction can be given to a language model at high spatial resolution. Existing frame-selection methods retain a few frames and discard the rest, while uniform compression preserves the timeline but allocates the same low resolution to every frame. We introduce Compressed Sight, Selective Insight (CSSI), an efficient video language model that keeps every sampled frame in the input: a small set of selected frames receives a grid of spatial detail tokens, and every other frame is represented by one average-pooled token. Which frames receive detail is decided by a query-conditioned selection policy that reads all frames in one linear-time pass and then selects frames one at a time, each choice conditioned on the frames already chosen, so the budget is spent on complementary evidence rather than repeated views of the same scene. The policy is trained with reinforcement learning from a reward computed by the model itself, the margin by which the model separates the correct answer from the strongest incorrect option when reading the resulting input, so training needs no external reward model and no frame-level labels. Across Video-MME, LongVideoBench, MLVU, EgoSchema, and LVBench, CSSI outperforms recent efficiency-oriented video LLMs, with its largest gains on long videos and on object-reasoning questions. CSSI is also the most memory and latency-efficient method at large frame counts, scaling to 16K-frame videos where competing methods run out of memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vide llms, video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TR9UOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CSSI, an efficient long-video understanding framework that represents selected keyframes with spatial detail tokens and compresses other frames into single timeline tokens. It introduces a query-conditioned autoregressive frame selector trained with an answer-margin reward. Experiments demonstrate improvements on several long-video QA benchmarks.

Critical strengths
  1. CSSI achieves consistent improvements over ViLAMP on several long-video QA benchmarks, particularly on longer videos.
  2. The paper provides extensive evaluations, including frame-selection comparisons, needle-in-a-haystack experiments, and efficiency scaling analysis.
Critical weaknesses

1. Limited novelty compared with existing methods.

The mixed-resolution representation emphasized as a main contribution has already been introduced by ViLAMP, which retains detailed tokens for selected frames and compresses unselected frames into single tokens. Moreover, ReFoCUS has already explored autoregressive frame selection trained with an answer-margin reward and policy-gradient optimization. The paper should better clarify its methodological differences and contributions beyond these existing approaches.

2. Insufficient experimental validation.

  • Table 1 lacks comprehensive comparisons with existing keyframe-selection methods on long-video QA benchmarks. Table 2 only provides limited comparisons on Video-MME.
  • An ablation removing the timeline tokens is missing, making their contribution unclear.
  • Although Figure 1 reports latency and memory, a detailed inference-time breakdown and direct efficiency comparison with ViLAMP are missing.
  • The autoregressive selection strategy provides only a 0.5-point improvement over independent ranking in Table 5, raising questions about its effectiveness.

3. Unclear method presentation.

Figure 2 does not clearly illustrate the autoregressive frame-selection process, particularly how previous selections influence subsequent decisions.

Questions for authors

Q1. Novelty relative to ViLAMP and ReFoCUS. What are the precise methodological differences between CSSI and these two existing works? ViLAMP already introduces mixed-resolution video representation, while ReFoCUS proposes autoregressive frame selection with a prediction-margin reward and group-relative policy optimization. Could the authors explain which components are fundamentally new in CSSI and provide controlled experiments demonstrating their contributions?

Q2. Missing comparisons and ablations. Could the authors compare CSSI with established keyframe-selection methods, especially learned or RL-based selectors, across the full benchmarks in Table 1 rather than only the limited categories in Table 2? For the ViLAMP comparison, does the baseline retain its original differential feature merging, or is it replaced by average pooling? Additionally, could the authors report an ablation without timeline tokens, keeping the selected keyframes and their spatial resolution unchanged, to clarify the contribution of the mixed-resolution representation?

Q3. Training data provenance and reward design. CSSI reports approximately 962K candidate QA pairs and 393K filtered training samples, identical to the numbers reported in ReFoCUS. Were these data independently collected and filtered, or was any existing dataset or pretrained selector reused? Additionally, how does the reward in Equation (12) differ substantively from the normalized answer-margin reward in ReFoCUS?

Q4. Inference efficiency. Could the authors report a detailed inference-time breakdown, including visual encoding, selection-policy execution, visual-token construction, and language-model inference? How do the additional costs introduced by the 24-layer Gated DeltaNet-2 selector compare with ViLAMP under the same hardware, input lengths, and visual-token budgets? Including ViLAMP in the efficiency scaling comparison would be particularly informative.

Q5. Autoregressive selection mechanism. Figure 2 does not clearly show how previously selected frames affect subsequent selection decisions. It also suggests a different input ordering from Section 3.3. Could the authors clarify the exact execution sequence? Given the relatively small improvement of autoregressive selection over independent ranking in Table 5 (64.9 vs. 64.4), is this difference statistically significant and consistent across multiple benchmarks?

Reviewer_m1mrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CSSI, an efficient video language model designed for long video understanding. Specifically, CSSI represents every sampled frame with at least one average-pooled token, while a subset of frames receives high-resolution spatial detail tokens. This high-resolution frames are selected by a question-conditioned policy. It is trained using policy gradient with a reward defined by the frozen language model's answer margin on multiple-choice questions. The paper reports evaluation results on several long video benchmarks, demonstrating promising performance compared to the selected methods in this paper.

Critical strengths
  1. The choice to retain every sampled frame coarsely while allocating spatial detail to selected frames is well-motivated. In long video understanding, most videos contain significant temporal redundancy that is irrelevant to the target task.

  2. The reported memory and latency curves in Figure 1 show favorable scaling, and CSSI is the only compared configuration reported to reach 16K frames.

Critical weaknesses
  1. Missing related methods: The paper contrasts itself with prior methods that select a subset of frames and discard the rest, framing the retention of coarse representations for unselected frames as a key contribution (Contribution 1). However, this positioning overlooks the most relevant line of work. For example, (1) QuoTA (AAAI) performs query-oriented frame-level token assignment under a fixed visual-token budget. (2) KVTP (ICCV) uses query relevance to assign different pruning rates to different frames while retaining contextual information from less important frames. (3) Tempo (ECCV) allocates more token to query-critical temporal segments while compressing other regions into compact representations. These approaches share the same core principle as CSSI: non-uniformly allocating visual capacity conditioned on the query while preserving compressed global temporal context. However, I found no discussion or comparison with these methods. They should be discussed explicitly in the related work, and an empirical comparison under matched visual-token budgets (accuracy and efficiency) would be valuable to show CSSI's advantages.

  2. The claimed benefit of sequential selection is to encourage complementary evidence and avoid repeatedly selecting similar frames. This is a central component of the proposed method and is highlighted in Contribution 2. However, Table 5 shows only a 0.5 improvement over top-K selection using the same policy scores.This raises questions about the practical benefit of the sequential feedback mechanism.

Questions for authors
  1. Please refer to W1 and discuss the relationship to the mentioned methods. The cited works are only a small subset of this broader line of research.
  • For example, a compressed view of the whole timeline with selective spatial detail on a few frames (CSSI) is closely related to allocating more tokens to query-critical segments and fewer tokens to less relevant regions while preserving the main storyline (Tempo). I think the binary coarse/detail representation in CSSI may also be less flexible than more adaptive token-allocation schemes, especially for general tasks such as captioning that may benefit from a softer allocation of visual capacity.
  1. Table 2 mainly compares CSSI with lightweight selection methods, while the proposed approach uses a 24-layer selector. It is therefore unclear how much of the gain comes from the proposed selection mechanism itself versus the capacity of the stronger selector backbone. A capacity-matched learned baseline would help clarify this.

  2. It would be useful to compare the proposed answer-margin reward with simpler alternatives, such as binary correctness or correct-answer likelihood, to better understand whether the specific reward design is necessary.

  3. Latency is an important practical metric, but the current analysis (Figure 1) does not provide enough detail on where the runtime is spent. A more detailed latency breakdown would be helpful, for example separating the selector cost, TTFT, and LLM decoding time.

  4. Typo: Line 097 contains "(Figure 1)" twice.

Reviewer_b3iFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a frame-selection method for long-video understanding with LLMs. It uses reinforcement learning to train a query-conditioned, DeltaNet-based causal selection policy. Experiments on several video benchmarks demonstrate its effectiveness, while ablation studies examine its behavior and the contributions of key design choices.

Critical strengths
  1. The DeltaNet-based causal selection policy is an interesting design that offers a promising balance between computational efficiency and accuracy.

  2. The exploration of the trade-off between the number of detailed frames and the number of tokens per frame is informative.

Critical weaknesses
  1. Qwen2 is a relatively outdated backbone. Evaluating the method with more recent model checkpoints would strengthen the evidence for its effectiveness and generalizability.

  2. It is unclear whether the training corpus contains sufficient long-video examples and is well suited to learning a long-video frame-selection policy. More details on the video duration distribution and data composition would help address this concern.

Questions for authors
  1. Beyond initialization from pretrained weights, does the DeltaNet-based selector undergo a supervised fine-tuning (SFT) warm-start stage before RL training? Such a stage may benefit GRPO-style training. Have the authors explored its impact on training stability and final performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.