- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to construct a video-encoding pathway with a fixed output-token budget from framewise image representations. A frozen image encoder independently processes each frame. The method then inherits frame-level quotas from an upstream SITE allocator and selects source tokens using question-weighted per-frame determinantal point processes (DPPs) and adjacent-frame correspondence rewards. An optional learned temporal refiner reads neighboring context, including unselected candidates, and writes residual updates to retained anchors while preserving the output-token count, source positions, and intermediate-feature interface. Conditional-KL and fixed-operator analyses motivate the distinction between evidence selection and temporal interaction. Experiments on 13 benchmarks and three Qwen vision–language backbones report aggregate accuracy comparable to the full-image pathway using approximately 28%–35% of its visual tokens.
Critical strengths
- Broad and practically useful accuracy–token trade-offs. Experiments cover 13 benchmarks and three backbones and show consistent compression benefits. For example, Qwen3-VL-8B achieves a 13-benchmark macro-average of 62.75 with 1,535 visual tokens, compared with 62.58 using 4,424 tokens for the full-image pathway. On Qwen3-VL-32B, the corresponding scores are 66.28 and 66.09, while end-to-end latency decreases from 911.2 ms to 421.8 ms.
- A coherent decomposition, a carefully preserved source interface, and appropriately scoped theoretical claims. Separating framewise representation, cross-frame budget allocation, and temporal interaction clarifies what compression changes. Applying the same selected indices to visual embeddings, MRoPE positions, and DeepStack features preserves interface consistency. The distinction between reading additional context and increasing output-token count is also useful for system design. The analyses explicitly concern conditional prediction distributions and fixed interaction operators; they do not claim a downstream accuracy guarantee, global optimality of greedy selection, or universal superiority of delayed interaction over native video encoding. (pp. 3–6; Appendices F–H)
Critical weaknesses
- The central SITE allocator is insufficiently specified and attributed, limiting reproducibility and contribution attribution. Appendix I.3 explains how representative source indices returned by SITE are converted into frame quotas, but does not adequately define how SITE generates its compressed slots and representative indices from the video, question, and budget. Table 7 lists weights for question relevance, motion, saliency, diversity, uncertainty, and other signals, but these settings do not replace complete signal definitions and an allocation algorithm. The allocator’s provenance or corresponding implementation also needs clarification. This omission is consequential: dynamic allocation contributes a 1.62-point macro-average improvement in Table 5 and therefore accounts for a substantial part of the overall gain. The current manuscript does not clearly separate the contribution of the inherited allocator from that of the proposed coordinated selector. (pp. 5, 15, and 27–28).
- Incomplete architectural information in Figure 1. Figure 1(a) conveys the processing order, but omits the question input (q), the reduction from (M) candidates to (B) anchors, the context branch through which the refiner reads unselected neighboring tokens, and the (B→B) output interface. Frozen versus trainable modules, preservation of source positions and intermediate features, and the connection to the language model are also not explicitly shown. The cross-frame lines at the top are unexplained and may blur the distinction between correspondences used for selection and feature interaction during encoding.
- Inconsistent color semantics. The text and captions use blue, orange, and teal for representation, selection, and temporal interaction, respectively, whereas the temporal-refinement block in Figure 1 is purple. Tokens from different frames also use blue, orange, and green, without a legend distinguishing frame identity from processing stage. Undefined symbols in Figure 2. The two purple question marks in Figure 2(a) may indicate difficulty in isolating frame-specific evidence after early temporal mixing, but their meaning is not defined in the caption. The illustration should also not be interpreted as empirical evidence that Conv3D necessarily loses the relevant information. The red dashed line in Figure 2(b) aligns with the endpoint of the budget-(B) bracket and may represent a budget reference, but its meaning is unexplained and may suggest that the full-image pathway truncates tokens at that location. Limited readability of Figure 1. Module text and several plot labels are too small, while the radar chart contains dense labels and overlapping curves. This makes identifying modules and comparing methods difficult at the normal reading size. Table formatting needs standardization, and a confirmed occlusion defect requires correction. Header line breaks, vertical alignment, and placement of units and direction arrows vary across tables; units for difference columns are also inconsistently displayed. In Table 6, the multirow “Token budget” group label exists in the PDF text layer but is covered by gray shading and is consequently invisible on the rendered page.
Questions for authors
- Is SITE an existing method, a public implementation, or a new component of this submission? Please provide its provenance and an algorithm or code sufficient to reconstruct the complete □(((V,q,B)→b_t)) mapping, including how the signals in Table 7 are computed and combined and how compressed slots and representative source indices are generated.
- With the same image candidate pool and output budget, how does the original SITE output compare with SITE quotas followed by the proposed coordinated selector? Please clarify any differences in their output interfaces. In Table 5, does the dynamic-allocation ablation change only frame quotas while holding the candidate pool, per-frame DPP rules, and other settings fixed?
- Can you report paired per-benchmark refinement gains and their variability across the existing seeds, and provide a matched control on Video-MME or LVBench with the same architecture and training objectives but access only to selected neighboring tokens? This would help establish whether the larger localized gains specifically arise from unselected temporal context rather than general learned residual adaptation.