acceptodds
Withdrawn from ICLR 2027

Video Encoders Built on Image Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The design of a video encoder determines when frames begin to interact and which frame-specific visual evidence remains accessible to the language model. Native video pathways couple neighboring frames during visual encoding, whereas image pathways preserve independently computed frame representations but incur a much larger visual-token cost when all image tokens are forwarded. We ask a basic question: whether a compact video encoder can instead be built on image representations. To answer this question, we separate three operations that are often coupled: per-frame representation, cross-frame token allocation, and temporal interaction. A frozen image encoder first produces frame-specific candidates. A question-aware selector then allocates a fixed token budget across frames using relevance, diversity, and cross-frame correspondence, after which a lightweight learned refiner reads neighboring-frame context and writes residual updates only to the retained anchors. This preserves source positions and keeps the visual output at the fixed budget. Across 13 benchmarks and three vision–language backbones, the resulting pathway matches full-image aggregate performance while using only about 28%–35% of its visual tokens. Specifically, on Qwen3-VL-8B, it achieves a 13-benchmark macro-average of 62.75 with 1,535 visual tokens, compared with 62.58 for the full Image pathway at 4,424 tokens and 59.49 for native Conv3D at 2,212 tokens. On Qwen3-VL-32B, it reaches a 13-benchmark macro-average of 66.28, compared with 66.09 for Image, while providing a 2.16× end-to-end speedup. These results show that compact video encoding does not require early temporal mixing: frame-specific evidence can be preserved first, allocated jointly, and temporally contextualized after selection.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video understanding, visual token compression, image representations, token selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fF3sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to construct a video-encoding pathway with a fixed output-token budget from framewise image representations. A frozen image encoder independently processes each frame. The method then inherits frame-level quotas from an upstream SITE allocator and selects source tokens using question-weighted per-frame determinantal point processes (DPPs) and adjacent-frame correspondence rewards. An optional learned temporal refiner reads neighboring context, including unselected candidates, and writes residual updates to retained anchors while preserving the output-token count, source positions, and intermediate-feature interface. Conditional-KL and fixed-operator analyses motivate the distinction between evidence selection and temporal interaction. Experiments on 13 benchmarks and three Qwen vision–language backbones report aggregate accuracy comparable to the full-image pathway using approximately 28%–35% of its visual tokens.

Critical strengths
  1. Broad and practically useful accuracy–token trade-offs. Experiments cover 13 benchmarks and three backbones and show consistent compression benefits. For example, Qwen3-VL-8B achieves a 13-benchmark macro-average of 62.75 with 1,535 visual tokens, compared with 62.58 using 4,424 tokens for the full-image pathway. On Qwen3-VL-32B, the corresponding scores are 66.28 and 66.09, while end-to-end latency decreases from 911.2 ms to 421.8 ms.
  2. A coherent decomposition, a carefully preserved source interface, and appropriately scoped theoretical claims. Separating framewise representation, cross-frame budget allocation, and temporal interaction clarifies what compression changes. Applying the same selected indices to visual embeddings, MRoPE positions, and DeepStack features preserves interface consistency. The distinction between reading additional context and increasing output-token count is also useful for system design. The analyses explicitly concern conditional prediction distributions and fixed interaction operators; they do not claim a downstream accuracy guarantee, global optimality of greedy selection, or universal superiority of delayed interaction over native video encoding. (pp. 3–6; Appendices F–H)
Critical weaknesses
  1. The central SITE allocator is insufficiently specified and attributed, limiting reproducibility and contribution attribution. Appendix I.3 explains how representative source indices returned by SITE are converted into frame quotas, but does not adequately define how SITE generates its compressed slots and representative indices from the video, question, and budget. Table 7 lists weights for question relevance, motion, saliency, diversity, uncertainty, and other signals, but these settings do not replace complete signal definitions and an allocation algorithm. The allocator’s provenance or corresponding implementation also needs clarification. This omission is consequential: dynamic allocation contributes a 1.62-point macro-average improvement in Table 5 and therefore accounts for a substantial part of the overall gain. The current manuscript does not clearly separate the contribution of the inherited allocator from that of the proposed coordinated selector. (pp. 5, 15, and 27–28).
  2. Incomplete architectural information in Figure 1. Figure 1(a) conveys the processing order, but omits the question input (q), the reduction from (M) candidates to (B) anchors, the context branch through which the refiner reads unselected neighboring tokens, and the (B→B) output interface. Frozen versus trainable modules, preservation of source positions and intermediate features, and the connection to the language model are also not explicitly shown. The cross-frame lines at the top are unexplained and may blur the distinction between correspondences used for selection and feature interaction during encoding.
  3. Inconsistent color semantics. The text and captions use blue, orange, and teal for representation, selection, and temporal interaction, respectively, whereas the temporal-refinement block in Figure 1 is purple. Tokens from different frames also use blue, orange, and green, without a legend distinguishing frame identity from processing stage. Undefined symbols in Figure 2. The two purple question marks in Figure 2(a) may indicate difficulty in isolating frame-specific evidence after early temporal mixing, but their meaning is not defined in the caption. The illustration should also not be interpreted as empirical evidence that Conv3D necessarily loses the relevant information. The red dashed line in Figure 2(b) aligns with the endpoint of the budget-(B) bracket and may represent a budget reference, but its meaning is unexplained and may suggest that the full-image pathway truncates tokens at that location. Limited readability of Figure 1. Module text and several plot labels are too small, while the radar chart contains dense labels and overlapping curves. This makes identifying modules and comparing methods difficult at the normal reading size. Table formatting needs standardization, and a confirmed occlusion defect requires correction. Header line breaks, vertical alignment, and placement of units and direction arrows vary across tables; units for difference columns are also inconsistently displayed. In Table 6, the multirow “Token budget” group label exists in the PDF text layer but is covered by gray shading and is consequently invisible on the rendered page.
Questions for authors
  1. Is SITE an existing method, a public implementation, or a new component of this submission? Please provide its provenance and an algorithm or code sufficient to reconstruct the complete □(((V,q,B)→b_t)) mapping, including how the signals in Table 7 are computed and combined and how compressed slots and representative source indices are generated.
  2. With the same image candidate pool and output budget, how does the original SITE output compare with SITE quotas followed by the proposed coordinated selector? Please clarify any differences in their output interfaces. In Table 5, does the dynamic-allocation ablation change only frame quotas while holding the candidate pool, per-frame DPP rules, and other settings fixed?
  3. Can you report paired per-benchmark refinement gains and their variability across the existing seeds, and provide a matched control on Video-MME or LVBench with the same architecture and training objectives but access only to selected neighboring tokens? This would help establish whether the larger localized gains specifically arise from unselected temporal context rather than general learned residual adaptation.
Reviewer_WtNQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an image-first video encoding framework that separates framewise representation, cross-frame token selection, and temporal refinement. The method combines frozen image features with question-aware token allocation, DPP-based diversity selection, cross-frame correspondence, and a lightweight learned refiner. Experiments across 13 benchmarks and three VLM backbones demonstrate competitive performance with substantially fewer visual tokens and reduced inference latency.

Critical strengths
  1. The experimental evaluation is extensive, covering 13 benchmarks and three model backbones. The proposed approach achieves favorable accuracy–efficiency trade-offs compared with existing token compression methods.

  2. The training-free selection variant achieves most of the reported improvements, suggesting practical applicability without additional backbone training.

Critical weaknesses
  1. The proposed framework primarily combines established techniques, including question-aware selection, DPP-based diversity, cross-frame feature matching, and residual attention. Moreover, Appendix I.3 indicates that the frame-level token budgets are inherited from a frozen SITE allocator. The distinction between the proposed contributions and existing components should be clarified.

  2. According to Table 4, the learned refiner improves the 13-benchmark average by only 0.06–0.11 points across the three backbones, despite introducing approximately 18M trainable parameters. This raises questions about whether the additional training and architectural complexity are justified. Further analysis of which temporal reasoning tasks benefit from refinement would strengthen this component.

Questions for authors
  1. How much of the reported improvement comes from the inherited SITE allocator itself? Could the authors provide a direct comparison against SITE-based selection under the same token budget?

  2. What is the sensitivity of the method to correspondence thresholds and selection hyperparameters? Does the proposed selection strategy generalize to substantially longer videos or different frame-sampling densities?

Reviewer_saAbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an image-first pathway for video-language models. A frozen image encoder produces per-frame candidates; a question-aware selector allocates a fixed global budget across frames and chooses source anchors using relevance, within-frame DPP diversity, and cross-frame correspondence; and an optional lightweight temporal refiner reads nearby unselected tokens and writes residual updates to the retained anchors without increasing the output token count. Across three Qwen backbones and 13 benchmarks, the method reports Avg.(13) scores of 59.43, 62.75, and 66.28 at about 1.5K visual tokens, close to or slightly above the Full Image pathway while using about 28%-35% of its tokens. On Qwen3-VL-32B, end-to-end latency decreases from 911.2 ms to 421.8 ms. These results make a promising case for image-first compression, but the current experiments do not fully isolate the proposed selector from the candidate representation.

Critical strengths
  1. Strong compression evidence across a broad evaluation. The method approximately preserves the Full Image aggregate performance on three backbones and 13 benchmarks while using far fewer visual tokens. The main averages are internally consistent, and the paper also reports tasks on which performance drops rather than presenting only favorable cases.

  2. Useful efficiency and ablation evidence. The paper reports end-to-end latency, TTFT, memory, throughput, token counts, and selection cost. The selection-only variant already reaches 59.37/62.64/66.22, showing that most of the result does not depend on the trained refiner. Table 6 also gives a meaningful failure control: incorrect cross-frame matches reduce Avg.(13) from 59.43 to 57.53, below the no-correspondence result of 58.50.

Critical weaknesses
  1. The main comparison changes both the candidate pool and the selector. The paper does not state whether each compressed baseline starts from Video/Conv3D tokens or per-frame Image tokens, nor whether the baselines preserve the same source positions, MRoPE, and DeepStack interface. All internal ablations use the Image candidate pool. In Table 5, the simplest Image-pool variant, without dynamic allocation or cross-frame coordination, already scores 60.12, above the strongest compressed baseline in the Qwen3-VL-8B main table at 59.86. This does not prove that the candidate pool explains the gain, but it leaves that alternative open. A matched candidate-pool-by-selector comparison is needed to isolate the proposed selector.

  2. The SITE frame allocator is not defined well enough to reproduce the method. SITE supplies the frame budgets used by the later selector, but the paper does not provide its full algorithm or source, the definitions and combination of its scores, representative-index construction, parameter-selection protocol, tie-breaking rules, or the adjacent-frame scene-similarity statistic. This is consequential because dynamic allocation provides the largest single gain in Table 5: +1.62 Avg.(13), compared with +0.90 from cross-frame coordination and +0.11 from the refiner.

Questions for authors
  1. For each main-table compressed baseline, which candidate path is compressed: Video/Conv3D tokens or per-frame Image tokens? Do the baselines use the same 32 frames, source positions, MRoPE, DeepStack streams, and output budget as the proposed method?

  2. Can you provide a complete definition or stable public version of SITE, including all input scores, representative-index construction, integer budget assignment, capacity clipping, tie-breaking, scene similarity, parameter selection, and measured cost?

  3. Under Qwen3-VL-8B with 32 frames, about 1.5K tokens, and the refiner disabled, how do the strongest adaptable baseline and the proposed selector compare on both the Video/Conv3D and Image candidate pools? This matched 2x2 comparison would most affect my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.