acceptodds
Under review as a conference paper at ICLR 2027

Complete the Scene, Cover the Video: Recurrent Token Compression for Video LLMs

Abstract

Visual token compression aims to reduce inference costs in Video Large Language Models while preserving information for video understanding. Existing compressors often build on independent frame-wise selection of important tokens. However, such selection may repeatedly retain content already well represented in earlier outputs while overlooking information not yet preserved, limiting coverage of content unfolding throughout the video. Motivated by representation gaps revealed in our video-level feature coverage analysis, we propose C²V, a training-free, before-LLM method using the preceding frame’s compressed representation as a visual state. It greedily selects anchors that complement the state in representing the current scene. Candidate sets are evaluated by Maximum Mean Discrepancy (MMD) between the attention-weighted scene distribution and an optimal convex mixture of the state and candidate anchor distributions. Mass-preserving aggregation incorporates unselected tokens’ features and probability mass into the anchors, producing LLM input features and the anchor distribution used as the next visual state. We evaluate C²V on five benchmarks spanning short clips to long videos, using Qwen3-VL-8B, InternVideo3-8B, and Qwen2.5-VL-7B at four retention ratios (10–25%). C²V achieves the highest five-benchmark average accuracy among evaluated compressors in all 12 model–retention settings. It also outperforms competing methods in video-level feature coverage at matched token budgets, indicating better collective representation of source features.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video understanding, vision-language models, visual token compression, efficient inference, training-free methods

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bGcjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

C2V is a training-free video token compressor applied before the LLM. It selects current-frame anchors by minimizing MMD between an attention-weighted scene distribution and an optimal mixture of the preceding compressed state and current anchors. Feature aggregation produces the LLM inputs and next-frame state. Experiments cover three backbones, five QA benchmarks, and four retention ratios.

Critical strengths

The formulation is well specified, with a closed-form mixture coefficient and detailed selection, aggregation, and position-handling rules. Component ablations examine the state, mixture weights, and feature fusion.

Critical weaknesses
  1. The coverage analysis does not isolate the contribution of recurrence. Figure 1 evaluates the full pipeline; Table 5 ablates the previous-frame state only through QA accuracy (60.3% versus 59.2%). Report final-output coverage for the existing no-state MMD and fixed-mixture variants, keeping aggregation and token budgets identical.

  2. Long-range redundancy and scalability remain insufficiently tested. The state stores only the preceding frame's compressed features. Test repeated or interleaved scenes and extend the current 128-frame evaluation up to 768 frames, with intermediate settings. Include Qwen3.5-27B and Qwen3.5-35B-A3B to extend beyond 7–8B dense models. Report accuracy, compression overhead, end-to-end latency, and peak memory against no-state selection.

  3. Task coverage is limited to QA. Add temporal grounding and video captioning on suitable benchmarks, reporting timestamp accuracy and caption quality.

  4. The baseline comparison is insufficiently comprehensive. Add a few representative baselines, such as SparseVLM [1], DyCoke [2], VidCom2 [3], HoliTom [4], and the now-public MMG-Vid. Compare accuracy and efficiency under matched inputs and comparable compression budgets, distinguishing before-LLM and hybrid configurations and including compression overhead.

[1] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis A. Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. “SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference.” ICML, 2025.

[2] Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. “DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models.” CVPR, 2025.

[3] Xuyang Liu, Yiyu Wang, Junpeng Ma, and Linfeng Zhang. “Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models.” EMNLP, 2025.

[4] Kele Shao, Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. “HoliTom: Holistic Token Merging for Fast Video Large Language Models.” NeurIPS, 2025.

Questions for authors

See weekness.

3 comments
AuthorsOpenReview ↗

Response to W1: State Conditioning and Final-Output Coverage

Dear Reviewer bGcj,

Thank you for your careful and detailed review. We appreciate your recognition of our well-specified formulation and component ablations, as well as your constructive suggestions on recurrence, scalability, and broader evaluations. Your comments have provided a valuable opportunity to clarify our contributions and strengthen the empirical evidence. We address your concerns with additional experiments and analyses below.

Summary. (1) With identical aggregation and token budgets, state conditioning raises final-output coverage over no-state MMD by 0.9–2.5 pp and over a fixed mixture by 0.5–1.9 pp in all 12 conditions. (2) It reduces cross-visit duplication under interleaving, without constituting long-term memory, and improves accuracy over no-state at every tested input length from 128 to 768 frames (+0.7–1.3 pp) and on Qwen3.5-27B and Qwen3.5-35B-A3B (+0.8–0.9 pp), changing E2E latency by at most 0.1 s and leaving peak memory unchanged. (3) On Charades-STA grounding and YouCook2 captioning, C²V leads the reported compressors on every metric at 10% and 20%, e.g., +2.10 mIoU over MMG-Vid and +6.52 CIDEr over FlashVID† at 10%. (4) Against VidCom2, HoliTom (w/o M), MMG-Vid, and EchoPrune, C²V keeps the highest five-benchmark average (57.6/59.3 vs. at most 56.6/58.6 at 10%/20%), with 26.6 ms of compression overhead, within the 12.0–39.7 ms range of these baselines.

Q1. State conditioning and final-output coverage. You are right that Fig. 1 evaluates the full pipeline rather than isolating recurrence. Following your suggestion, we compared the existing no-state MMD (λ=0\lambda=0), fixed-mixture (λ=0.5\lambda=0.5), and full C²V variants using the same inputs, aggregation rule, and token budgets. We followed the Fig. 1 protocol (Qwen3-VL-8B, VideoMME, 32 sampled frames, 10% retention), measuring coverage from the final aggregated outputs, not the selected anchors alone.

Source weightingVariantRank 32Rank 64Rank 128Rank 256
UnweightedNo-state MMD90.892.393.594.2
Fixed mixture91.593.094.194.9
C²V92.894.295.295.9
Attention-weightedNo-state MMD87.090.393.495.3
Fixed mixture87.691.194.195.7
C²V89.592.294.796.2
Uniqueness-weightedNo-state MMD90.491.992.893.4
Fixed mixture91.192.793.794.5
C²V92.593.894.795.3

MMD selection with aggregation already yields high coverage without a recurrent state. The full pipeline's coverage advantage should therefore not be attributed entirely to recurrence. Nevertheless, fixed mixing improves over no-state in all 12 rank–weighting conditions, and C²V improves over no-state by 0.9–2.5 percentage points and over fixed mixing by 0.5–1.9 points, again in all 12 conditions. With the aggregation rule and budgets unchanged, these results show the additional benefits of using the preceding compressed state and adapting its mixture weight to each candidate set.

AuthorsOpenReview ↗

Response to W2: Long-Range Redundancy, Longer Inputs, and Larger Backbones

Q2. Long-range redundancy, longer inputs, and larger backbones. Following your suggestions, we added interleaved-scene tests, extended evaluation to 768 sampled frames with intermediate settings, and evaluated both Qwen3.5-27B and Qwen3.5-35B-A3B. Comparisons use no-state MMD with the same aggregation, inputs, and output budgets.

1. Interleaved scenes. Using Qwen3-VL-8B features from 900 VideoMME videos (32 sampled frames per source video), we split each video into consecutive segments and interleaved them with other videos: V1,D1,…,Vk,DkV_1,D_1,\ldots,V_k,D_k, with k=2,4,8,16k=2,4,8,16 and unchanged per-frame budgets. Cross-visit duplication is the percentage of later-visit VV output tokens matching an earlier-visit output at cosine similarity ≥0.8\geq0.8.

InterruptionsDuplication reduction vs. no-state, 10% (pp)20% (pp)
11.831.17
31.771.07
71.330.84
150.480.29

All reductions have paired video-level bootstrap 95% CIs excluding zero. The reduction persists under interleaving, although it narrows with frequent interruptions. This indicates reduced cross-visit redundancy in the final outputs, not explicit long-term memory of earlier visits.

2. Scaling to 768 frames. We evaluated Qwen3-VL-8B at 128/256/320/512/768 sampled frames, keeping spatial preprocessing unchanged and the LLM visual-token budget fixed to the uncompressed 32-frame setting (R=32/FR=32/F). Entries below are C²V / no-state accuracy (%); Avg. covers the three benchmarks.

FramesRetentionVideoMMELongVideoBenchLVBenchAvg.
12825%67.9/67.862.1/60.843.1/41.957.7/56.8
25612.5%68.0/68.062.9/61.444.9/44.358.6/57.9
32010%69.0/67.463.1/61.847.3/46.359.8/58.5
5126.25%67.6/66.462.2/61.146.7/45.658.8/57.7
7684.17%66.9/66.162.5/61.246.5/45.658.6/57.6

C²V achieves a higher mean in all five settings, with gains of 0.7–1.3 pp. At 768 frames, its mean is 58.6, vs. 57.6 for no-state and 54.5 for the uncompressed 32-frame baseline.

3. Larger dense and MoE backbones. We evaluated both requested Qwen3.5 models with 32 sampled frames, 10% retention, and non-thinking inference, following the same three-benchmark protocol. Entries are again C²V / no-state.

ModelVideoMMELongVideoBenchLVBenchAvg.
Qwen3.5-27B66.4/66.162.8/61.539.9/38.956.4/55.5
Qwen3.5-35B-A3B63.7/63.159.0/58.140.6/39.554.4/53.6

C²V improves all three benchmarks on both backbones, with mean gains of approximately 0.8–0.9 pp.

4. Compression overhead, latency, and memory. Measurements use RTX A5000 GPUs: two for Qwen3-VL-8B (vision/compression and LLM on separate GPUs), and four for each Qwen3.5 model. All timings are medians over videos, measured within end-to-end runs that include vision encoding, compression, and LLM inference, with generation limited to one token (max_new_tokens=1) and host synchronization. Memory is the sum of per-GPU peak allocations. All pairs are C²V / no-state.

Model / framesCompression overhead (ms)E2E (1 token, s)Peak memory (GiB)
Qwen3-VL-8B / 128167.3/148.94.89/4.8825.54/25.54
Qwen3-VL-8B / 256209.6/181.07.99/7.9527.63/27.63
Qwen3-VL-8B / 320233.0/200.19.56/9.4928.06/28.06
Qwen3-VL-8B / 512294.4/255.714.24/14.2731.23/31.23
Qwen3-VL-8B / 768448.3/354.120.82/20.7235.87/35.87
Qwen3.5-27B / 3227.6/23.81.41/1.4053.34/53.34
Qwen3.5-35B-A3B / 3220.9/17.61.39/1.3767.50/67.50

In a separate measurement with a 16-token output, E2E latency is 3.40/3.39 s for Qwen3.5-27B and 3.06/3.03 s for Qwen3.5-35B-A3B. State conditioning increases compression overhead, but its effect on E2E latency is small in these measurements; peak allocations are unchanged at the reported precision.

These results extend the evidence for state conditioning to interleaved inputs, substantially longer sequences, and both larger backbone types.

AuthorsOpenReview ↗

Response to W3–W4: Temporal Grounding, Captioning, and Additional Baselines

Q3. Beyond QA: temporal grounding and video captioning. We evaluated Qwen3-VL-8B with 32 sampled frames at nominal 10%/20% retention, using common inputs and prompts. Baselines are FlashVID† and FastVID from our original evaluation, plus MMG-Vid and EchoPrune, related methods using past information whose code is now available.

Temporal grounding. On Charades-STA test (3,720 queries, 1,334 videos), R@θ is the percentage with temporal IoU ≥ θ; mIoU is mean IoU. Unparseable start–end intervals receive IoU 0. Scores are percentages.

RMethod[email protected][email protected][email protected]mIoU
100%Uncompressed78.3662.5536.5654.07
10%FlashVID†66.2449.2726.2944.79
FastVID56.0238.1717.9336.70
MMG-Vid68.1249.4126.5945.54
EchoPrune––––
C²V71.5652.3927.3147.64
20%FlashVID†71.9455.7531.6749.59
FastVID64.1447.4223.9042.95
MMG-Vid74.0656.2131.5950.31
EchoPrune––––
C²V74.7056.9432.9650.97

C²V leads the reported compressors on all four metrics at both ratios. At 10%, gains over MMG-Vid are +2.10 pp mIoU (95% CI [1.15,3.03]) and +2.98 pp [email protected] ([1.28,4.67]); CIs use paired video-level bootstrap (2,000 resamples).

“–”: EchoPrune's outputs could rarely be parsed as a start–end interval (8.3%/19.4% at 10%/20%; mostly a single timestamp), so we did not run its full grounding evaluation. Its captioning results are reported below.

Video captioning. On YouCook2 validation (3,178 clips, 414 videos), CIDEr is the primary metric; BERTScore is F1 × 100.

RMethodCIDErMETEORBERTScore
100%Uncompressed53.5114.0629.6
10%FlashVID†38.9511.0725.2
FastVID38.3011.2324.9
MMG-Vid36.3410.7224.9
EchoPrune32.5811.3024.0
C²V45.4712.0027.0
20%FlashVID†45.0612.8027.3
FastVID41.4912.4426.5
MMG-Vid42.4111.9426.8
EchoPrune38.4012.6926.1
C²V50.2113.0128.4

C²V exceeds all four baselines on the three reported metrics at both ratios. Against FlashVID†, the highest-CIDEr baseline here, paired 95% CIs for CIDEr gains are [4.28,8.78] at 10% and [3.28,7.03] at 20%. These results extend the evidence to temporal localization and free-form video description.

Q4. Additional baselines and compression overhead. We added VidCom2, HoliTom, MMG-Vid, and EchoPrune on Qwen3-VL-8B, using the same inputs and comparable LLM-input token budgets. All are evaluated before the LLM. For HoliTom, we use the authors' w/o M configuration, disabling inner-LLM merging; FlashVID† likewise denotes its before-LLM component.

SparseVLM requires a backbone-specific pruning-layer and retention schedule, which we did not assign without validation. DyCoke's original four-frame grouping preserves a full reference frame and cannot match 10%/20% LLM-input retention without modification.

QA accuracy (%); Avg. is the five-benchmark mean (32 frames).

RMethodMVBenchTempCompassVideoMMELongVideoBenchLVBenchAvg.
10%VidCom259.663.956.754.036.854.2
HoliTom (w/o M)65.668.458.555.035.556.6
MMG-Vid64.269.756.856.436.056.6
EchoPrune60.265.958.654.335.454.9
C²V66.270.159.355.536.757.6
20%VidCom265.368.560.655.637.157.4
HoliTom (w/o M)67.271.060.756.437.358.5
MMG-Vid67.272.059.857.336.958.6
EchoPrune63.169.561.456.137.157.5
C²V68.371.561.057.837.859.3

C²V retains the highest five-benchmark average at both ratios, exceeding the strongest added baseline by 1.0/0.7 pp.

Compression overhead at 10% (ms): VidCom2 12.01, HoliTom (w/o M) 39.72, MMG-Vid 26.11, EchoPrune 13.29, C²V 26.58. These use the QA configurations, averaged over 300 VideoMME videos on an RTX A5000, from vision-feature output to LLM input, excluding encoding and LLM computation.

Planned revisions. In the final version, we will add the coverage, long-input, and larger-model comparisons with the no-state variant (Q1, Q2) to a new appendix section, referenced from Secs. 4.3–4.4; the cost measurements to Appendix E; the limitation on re-selection after long intervals to a Limitations section; the captioning results to Sec. 4.4 alongside the coverage analysis and the grounding results to the appendix (Q3); and the additional baselines to the main results (Q4).

Thank you again for your careful review and concrete suggestions. We hope these additional results address your concerns and would be grateful if you would consider them in your final assessment.

Reviewer_ahYPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes C2V, a training-free visual token compression method for Video LLMs. Unlike conventional frame-wise compression approaches, C2V uses the preceding frame’s compressed representation as a visual state to guide current token selection. The method employs MMD-based anchor selection and mass-preserving feature aggregation to improve video-level information coverage. Experiments on five video understanding benchmarks and three Video LLMs demonstrate competitive performance across different token retention ratios.

Critical strengths

The paper is generally well written, and the motivation for reducing redundant token selection across consecutive frames is clearly presented.

Critical weaknesses
  1. The idea of using previously retained information to guide token selection is related to MMG-Vid and EchoPrune, which are also discussed in the paper. Although C2V introduces an MMD-based formulation, what fundamentally distinguishes C2V from these methods?
  2. C2V only uses the preceding frame's compressed representation rather than maintaining a cumulative representation of the entire video. Can this short-term state effectively prevent redundant selections when similar scenes reappear after long temporal intervals?
  3. Although the paper reports compression and prefill latency, the authors are encouraged to additionally report end-to-end inference latency including visual encoding part.
  4. Does C2V maintain its computational efficiency across different video resolutions, particularly at higher resolutions where the cost of pairwise kernel computations increases?
Questions for authors

Please refer to the Critical Weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to W1: Distinction from MMG-Vid and EchoPrune

Dear Reviewer ahYP,

Thank you for your careful review and for recognizing our motivation to reduce redundant token selection. We clarify the distinctions and report additional comparisons below.

Summary. (1) C²V outperforms MMG-Vid and EchoPrune in average QA accuracy in all four tested settings, in all 24 feature-coverage conditions, and in video captioning. (2) A cumulative reference did not improve QA, while its extra compression cost grew with video length (4.6 → 65.1 ms from 32 to 128 frames at 20%); we acknowledge that C²V does not explicitly suppress re-selection after long intervals. (3) Including vision encoding, C²V speeds up TTFT by 2.59×/2.02× and end-to-end latency by 1.74×/1.54× at 10%/25% retention. (4) Raising the pixel budget by 1.5×/2× (tokens per frame: 1.55×/2.17×) increases compression time by only 1.51–1.61×/2.14–2.36×, far below the 2.41×/4.72× growth of pairwise kernel cost.


W1. What fundamentally distinguishes C²V from MMG-Vid and EchoPrune

"Although C2V introduces an MMD-based formulation, what fundamentally distinguishes C2V from these methods?"

We agree that both methods also use past information. The fundamental difference is what C²V measures: each anchor is selected by how much it reduces the part of the current scene left unexplained by the preceding compressed output and the anchors already chosen. MMG-Vid and EchoPrune instead score individual tokens by their dissimilarity to, or predictability from, earlier tokens. MMD is simply the tool that compares the scene, the state, and the anchors in one space.

vs. MMG-Vid. MMG-Vid also aims to complete the current scene ("Background Completion") but measures what is missing differently. Its TG-DPC replaces the density term of DPC, which measures how many current tokens a candidate represents, with the distance to earlier selections, and multiplies it by within-frame separation. Novel and distinct tokens are favored, but how much underrepresented scene content a candidate helps explain does not enter its score. C²V's residual reflects both what is missing and its attention-weighted share of the scene.

vs. EchoPrune. EchoPrune measures redundancy against the raw features of the preceding frame, i.e., what was observed, not what was retained. At 10–25% retention, most of the preceding frame is discarded, so a token can be judged redundant on evidence absent from the output. C²V conditions on the preceding compressed output, so its evaluation of current candidates changes with what was actually kept.

Head-to-head comparison. Official code for both methods was released after our submission; we ported their code to our backbones with default settings and compared them at matched retention.

Feature coverage. Extending Fig. 1 (VideoMME) to both methods, C²V leads in all 24 conditions (2 retentions × 4 ranks × 3 weightings). At 10%, mean attention-weighted coverage is 93.1 vs. 83.4/83.7 (C²V vs. MMG-Vid/EchoPrune).

Video captioning. Table R1. YouCook2 (Qwen3-VL-8B; BERTScore F1 × 100).

RMethodCIDErMETEORBERTScore
10%MMG-Vid36.310.724.9
EchoPrune32.611.324.0
C²V45.512.027.0
20%MMG-Vid42.411.926.8
EchoPrune38.412.726.1
C²V50.213.028.4

C²V also leads on ROUGE-L and BLEU-4, with CIDEr gains of +9.1/+7.8 over the stronger baseline at 10%/20%.

Video QA. Table R2. Accuracy (%); Avg. is over all five benchmarks in the paper.

ModelRMethodMVBenchVideoMMELVBench5-Avg.
Qwen3-VL-8B10%MMG-Vid64.256.836.056.6
EchoPrune60.258.635.454.9
C²V66.259.336.757.6
Qwen3-VL-8B20%MMG-Vid67.259.836.958.6
EchoPrune63.161.437.157.5
C²V68.361.037.859.3
InternVideo3-8B10%MMG-Vid67.856.434.756.0
EchoPrune63.557.735.555.1
C²V69.659.635.857.2
InternVideo3-8B20%MMG-Vid70.659.436.358.2
EchoPrune67.660.636.657.5
C²V71.760.836.358.7

C²V leads all four averages, though not every benchmark.

AuthorsOpenReview ↗

Response to W2: Preceding-State Reference and Long-Interval Reappearance

W2. Can the preceding state prevent redundant re-selection after long intervals?

"Can this short-term state effectively prevent redundant selections when similar scenes reappear after long temporal intervals?"

C²V does not explicitly suppress such re-selection, but the additional redundancy caused by interruptions was small in our tests, and a cumulative reference did not improve QA while its cost grew with video length. Referencing only the preceding output is therefore a deliberate trade-off that keeps the reference size constant regardless of video length. The state is used only for selection; the LLM still receives the outputs of all frames, each at its own temporal position, so re-selected features also mark when a scene recurs.

We compared C²V with a simple cumulative variant that keeps everything else fixed and replaces the reference with the equal-weight mixture of all previous outputs, Ht−1=1t−1∑τ<tQτH_{t-1}=\frac{1}{t-1}\sum_{\tau<t}Q_\tau.

1. Revisit experiment. We interleaved segments of each VideoMME video with segments of other videos (V1,D1,V2,D2,…V_1, D_1, V_2, D_2, \dots; kk = 2/4/8/16 segments). Since C²V's state is replaced at every frame, a single intervening frame already removes the earlier visit from its reference. Duplication is the share of later-visit output tokens nearly identical (cosine ≥ 0.8) to earlier-visit outputs; excess duplication is the increase caused by the interruptions, relative to the same segments played continuously.

Table R3. Qwen3-VL-8B, 10% retention.

kk = 2 / 4 / 8 / 16Value
Excess duplication of C²V (pp)+0.21 / +0.48 / +1.07 / +2.35
Cumulative − C²V: duplication (pp)−1.36 / −1.15 / −1.37 / −1.99
Cumulative − C²V: whole-video coverage (Fig. 1 metric)−0.04 to +0.24

Losing the reference adds little redundancy (0.2–2.4 pp). The cumulative reference lowers duplication by 1–2 pp but leaves whole-video coverage nearly unchanged. Results at 20% follow the same pattern with smaller magnitudes.

2. QA. Table R4. Qwen3-VL-8B, 32 frames.

RetentionReferenceVideoMMELongVideoBenchLVBenchAvg.
10%Preceding (C²V)59.355.536.750.5
Cumulative58.955.236.250.1
20%Preceding (C²V)61.057.837.852.2
Cumulative60.957.837.452.0

3. Cost. C²V references KK state tokens at every frame, whereas the cumulative variant references (t−1)K(t-1)K. Table R5. Compression time (ms; compression only, 900 VideoMME videos).

Sampled framesC²V 10%Cumulative 10%C²V 20%Cumulative 20%
3225.828.833.237.8
6445.257.956.972.9
12880.1111.599.8164.9

Design choice. With no QA gain, nearly unchanged coverage, and a cost gap that grows from 4.6 ms to 65.1 ms (32 → 128 frames, 20%), we keep the preceding-state reference.

Limitation. As you point out, once the state is replaced, a reappearing scene is evaluated anew, and excess duplication grows with the number of interruptions (up to 2.4 pp with 15). We tested only an equal-weight history; more selective long-term memory remains future work.

AuthorsOpenReview ↗

Response to W3-W4: End-to-End Latency and Resolution Scaling

W3. End-to-end latency including visual encoding

"... the authors are encouraged to additionally report end-to-end inference latency including visual encoding part."

We measured the full pipeline (Qwen3-VL-8B, 32 frames, one RTX A5000, batch 1): TTFT = vision encoding + compression + prefill, and E2E = TTFT + 16 output tokens. In elapsed time, C²V speeds up TTFT by 2.59×/2.02× and E2E by 1.74×/1.54× at 10%/25% retention. Table R6 shows the per-stage GPU time (ms).

MethodEnc.Comp.PrefillTTFTDecodeE2E
Uncompressed601.1–1,714.32,315.4498.82,814.2
C²V 10%656.025.8201.9883.7 (2.62×)422.71,306.4 (2.15×)
C²V 25%656.035.6424.81,116.4 (2.07×)436.21,552.6 (1.81×)

C²V's encoding time includes final-layer attention extraction, which VisionZip, FastVID, and FlashVID also require. Compression takes only about 3% of TTFT. Elapsed E2E speedups are lower than the GPU-stage ones because the Hugging Face generation loop takes about the same time per decoding step (~68 ms) in all configurations. Including encoding also preserves the ordering of Table 7's matched-accuracy comparison: TTFT is 985 ms for C²V (15%) vs. 1,106 ms for FastVID (25%) and 1,093 ms for FlashVID† (20%).


W4. Efficiency at higher resolutions

"Does C2V maintain its computational efficiency across different video resolutions, particularly at higher resolutions where the cost of pairwise kernel computations increases?"

Yes. Raising the pixel budget by 1.5× and 2× increases tokens per frame by 1.55× and 2.17×, but compression time by only 1.51–1.61× and 2.14–2.36×, roughly in line with the token count and far below the 2.41× and 4.72× growth of pairwise kernel cost (N2N^2). Table R7. Compression time (ms; median, 300 VideoMME videos).

Pixel budgetTokens/frame10%25%
1× (0.50 MP)46424.735.4
1.5× (0.75 MP)72037.2 (1.51×)57.0 (1.61×)
2× (1.00 MP)100852.9 (2.14×)83.6 (2.36×)

The pairwise work runs in parallel on the GPU. Each frame's kernel matrix is built upfront with dense matrix operations and takes at most 20% of compression time even at 2×. During greedy selection, cached kernel sums let all candidates be scored in parallel with constant work each, so only the number of sequential selection steps grows with the token count. The O(TN2D)O(TN^2D) bound in Appendix E counts arithmetic operations, not latency.


Planned revisions. In the final version, we will add the distinction from MMG-Vid and EchoPrune to Sec. 2.1; the limitation on re-selection after long intervals to a Limitations section; and the end-to-end latency (Table R6) and compression latency across resolutions (Table R7) to Appendix E, alongside the vision-encoding and compression latencies in Table E.1. MMG-Vid and EchoPrune will also be added to the main results and the captioning results in the main text (see our response to Reviewer bGcj).

Thank you again for your careful review. We hope these responses address your concerns and would be grateful if you would take the additional evidence into account in your final assessment.

AuthorsOpenReview ↗

Polite Follow-up on Our Rebuttal

Dear Reviewer,

We sincerely appreciate the time and effort you have devoted to reviewing our paper. We fully understand that this is a particularly busy period for reviewers.

As the discussion period is approaching its end, we would be very grateful if you could find a moment to revisit our response and the additional experimental evidence we provided. We hope these clarifications have helped address your concerns, and we would greatly appreciate any further feedback you may have.

Thank you again for your time and thoughtful consideration.

Reviewer_EB68OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces C²V, a recurrent method for compressing video tokens before they enter the LLM. It selects anchors in the current frame conditioned on the preceding frame's compressed representation, then aggregates current-frame features into these anchors to produce the compressed tokens and the state for the next frame. Experiments report higher average accuracy than the evaluated pre-LLM compression baselines across the tested backbones and retention ratios. The method also improves long-video understanding by processing more sampled frames at a fixed LLM visual-token budget, outperforming the compared compression baselines.

Critical strengths
  1. A systematic formulation of incremental video token selection. The motivation to retain information that complements previously compressed representations is reasonable and practically relevant. Although temporal redundancy is already considered in existing video compression methods, C²V provides an explicit distribution-approximation objective for deciding which current-frame tokens to retain given the preceding compressed state. The combination of this objective, adaptive mixture weighting, and an implementable greedy algorithm provides a useful framework for understanding and addressing the incremental selection problem.

  2. Comprehensive experiments and informative analysis. The evaluation spans multiple backbones, benchmarks, and compression ratios, with component ablations that help explain the contributions of the proposed design. Figure 1 makes the representation gap concrete by comparing feature reconstruction under uniform, attention-based, and uniqueness-based evaluation weights. This analysis improves clarity and complements the downstream accuracy results. Table 2 also demonstrates a useful application: processing more sampled frames while holding the LLM visual-token budget fixed. Together, these experiments and analyses substantially deepen my understanding of the problem and support acceptance.

Critical weaknesses

I view the following as limited concerns about positioning and interpretation; they do not outweigh the strengths above.

  1. The research-gap statement could be more precisely scoped. The second introduction paragraph emphasizes limitations of independent frame-wise representative selection. However, video compression more broadly already includes methods that exploit information across adjacent frames; for example, VideoChat-Flash uses clip-level spatiotemporal compression. The manuscript acknowledges inter-frame merging and makes a more specific distinction in Section 2.1, but bringing that distinction into the introduction would clarify the contribution. In particular, the relevant advance is conditioning anchor-set selection on preceding compressed outputs through an explicit objective. A clearer separation from clip-level compression and temporal redundancy reduction would make the novelty claim easier to assess. VideoChat-Flash is relevant to this positioning discussion even though its trained architecture differs from C²V's training-free setting.

  2. The interpretation of vision-encoder attention as feature importance needs more explanation. Stage 1 assigns probability masses using final-layer vision attention. Such attention can reflect relationships among similar patches, so it is not immediately clear when high received attention indicates useful content and when it mainly reflects repeated visual structure, such as a textured background. The uniform-scene-mass ablation provides empirical support for the chosen weighting, which I appreciate. My remaining concern is the interpretation of this heuristic: the paper could explain more directly why it is appropriate for weighting the reconstruction target and how the approach handles repetitive regions that attract substantial attention. This would clarify the method's assumptions and likely limitations.

Questions for authors

Clarification requests:

  1. Could you sharpen the introduction's research-gap statement by distinguishing independent representative selection, clip-level spatiotemporal compression, and selection conditioned on preceding compressed outputs? In particular, how should readers understand C²V's contribution relative to approaches such as VideoChat-Flash that already exploit inter-frame redundancy?
  2. Beyond the uniform-scene-mass ablation, what evidence or reasoning supports interpreting final-layer vision attention as an appropriate importance weight for the current-frame feature distribution? How should the weighting behave when repeated visual patterns receive substantial attention despite being less relevant to video understanding?

Suggestion for future work: Table 2 is useful evidence of improved accuracy at a fixed visual-token budget, but its 16-frame baseline and scaling to 64 or 128 frames leave open a more ambitious question: can C²V improve on a strong long-context configuration of the same backbone? It would be informative to start from a feasible uncompressed baseline with hundreds of frames, specify the spatial resolution and token budget, and use compression to increase the sampled-frame count by a further factor of four or eight. This would test whether C²V can extend useful temporal coverage and improve accuracy beyond an already strong long-context baseline. I regard this as a future direction, not a requirement for acceptance.

3 comments
AuthorsOpenReview ↗

Response to W1 / Q1: Distinction from VideoChat-Flash and Research Gap

Dear Reviewer EB68,

Thank you for your careful reading and positive assessment of our work. We are grateful that you recognized the value of our systematic formulation of incremental token selection and the breadth of our experiments and analyses. Your comments give us a valuable opportunity to clarify the scope of our contribution and the interpretation of our attention weighting.

W1 / Q1. Distinction from VideoChat-Flash and the research gap.

As you note, exploiting inter-frame redundancy is not itself what distinguishes C²V. The distinction is that C²V decides what to keep in the current frame through an explicit objective conditioned on the preceding frame's compressed output.

Relation to VideoChat-Flash. VideoChat-Flash encodes several frames jointly as a clip and merges similar tokens within each clip using ToMe; video-level compression is then performed inside the LLM through progressive visual dropout. In the official implementation, the pre-LLM merging of each clip is determined by that clip's features alone, and the compressed output of earlier clips is not passed on as a reference for merging the next clip. VideoChat-Flash therefore exploits a wider spatiotemporal context than independent frame-wise compression, but it does not evaluate how well the current selection complements what has already been preserved.

C²V's contribution. C²V evaluates how well the preceding compressed state Qt−1Q_{t-1} and the current anchor set At(S)A_t(S) jointly approximate the current scene distribution PtP_t:

Lt(S)=min⁡λ∈[0,1]MMDκ2(Pt,λQt−1+(1−λ)At(S)).L_t(S)=\min_{\lambda\in[0,1]}\mathrm{MMD}_{\kappa}^2\big(P_t,\lambda Q_{t-1}+(1-\lambda)A_t(S)\big).

Here, Qt−1Q_{t-1} is not the distribution of the previous raw features but that of the actual compressed output after aggregation. C²V thus distinguishes information that was observed from information that survived compression, and conditions the current selection on the latter. Under this formulation, the value of the same candidate depends on the preceding state and on the anchors already chosen. The mixture weight is optimized in closed form for each candidate set (Eq. 5), and anchors are added greedily under the same objective. Rather than merging similar tokens or seeking tokens that differ from the past, C²V selects the additional information needed to explain the current scene together with the preceding compressed state. This links adaptive mixture weighting and greedy selection within a single distribution-approximation problem, which is the methodological contribution of C²V.

AuthorsOpenReview ↗

Response to W2 / Q2: Attention Weighting and Repetitive Patterns

W2 / Q2. Rationale for attention weighting and behavior on repetitive patterns.

Thank you for the suggestion to clarify how attention weights should be interpreted and how they behave on repetitive patterns. We agree that the empirical usefulness of attention should be distinguished from its limits as an indicator of semantic importance.

Why attention weights the reconstruction target. We use the attention each token receives in the final vision layer as a query-agnostic weight for preserving the encoder's representation. In an attention head, oj=∑iajivio_j=\sum_i a_{ji}v_i, so ∑jaji\sum_j a_{ji} is the total aggregation weight assigned to viv_i across query outputs. This motivates received attention as a query-agnostic proxy for how strongly the encoder draws on each token when forming its outputs, rather than an exact measure of semantic importance. Using it as the mass of PtP_t aligns the reconstruction target with how the encoder itself aggregates information. Vision-encoder attention is likewise used by VisionZip, FastVID, and FlashVID. VisionZip (Sec. 4.2, Table 5) reports that masking the 50 tokens with the highest encoder attention before SparseVLM selects 64 tokens lowers TextVQA accuracy from 51.1 to 46.4, indicating that high-attention tokens carry substantial information. Together with the uniform-scene-mass ablation you noted, this supports attention as a useful, if imperfect, weighting.

We agree with your concern. For the same reason, repetitive structures whose patches attend to one another can receive large mass, and high attention alone does not establish that a feature is important for video understanding or needs additional preservation. This is a limitation of using vision attention as a proxy for importance, and C²V shares it: our weighting neither identifies semantically redundant patterns nor recalibrates their attention.

What C²V can mitigate is the repeated selection of such features across frames. Attention determines the mass of PtP_t but does not by itself determine anchor selection. C²V evaluates how well the preceding compressed state and the current anchor set jointly explain PtP_t. Hence, if a high-attention feature is already well represented in the preceding state, there is less need to select it again.

To test this directly, we performed a state-intervention experiment with Qwen3-VL-8B on VideoMME. Let R+R^+ denote repetitive regions that receive more attention mass than their uniform share. Keeping the current features, attention, and token budget fixed, we changed only the preceding state: a state built from tokens of the previous frame that are similar to R+R^+, versus a same-size control state built from other tokens of the same previous frame (804 videos, 2,457 eligible frames at the compressor input).

Anchor share: control state → R+R^+-similar state10% retention20% retention
Assigned to R+R^+15.3% → 11.2%14.2% → 11.8%
Assigned to new content73.7% → 75.6%74.6% → 75.9%

R+R^+: groups of tokens connected by cosine similarity ≥ 0.8 that cover at least 5% of the frame's tokens and receive more attention mass than their uniform share. New content: current tokens whose cosine similarity to every token of the previous frame is below 0.8. All paired differences have 95% video-level bootstrap CIs excluding zero.

Measured jointly over the state and the current output, the current frame's representation (mean nearest-element cosine) did not decrease (+0.0014, 95% CI [0.0012, 0.0016] at 10%), and new content was better represented in the current output (+0.0021). We observed the same tendency in unmodified C²V runs: how well the preceding compressed output represents R+R^+ is negatively correlated with the change in R+R^+ anchors relative to no-state selection (Spearman ρ=−0.28\rho=-0.28 at 10% retention).

These results do not show that attention perfectly reflects semantic relevance. They show that, despite this limitation, C²V can reduce redundant selection of repetitive features that are already preserved and reallocate the budget to new content.

Future work. We also appreciate your suggestion to test whether C²V can improve on a strong long-context baseline with hundreds of frames. We agree that this is a valuable direction for future work.

Planned revisions. In the final version, we will clarify the distinction from clip-level compression (VideoChat-Flash) in the Introduction and Sec. 2.1, and discuss the limitations of vision-encoder attention as an importance weight in a Limitations section.

Thank you again for your positive assessment and constructive feedback. Your comments helped us clarify the contribution and explain the behavior and limits of attention weighting. We would welcome any further questions.

Thanks for the response. The additional state-intervention results helped me better understand C2V.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

Dear Area Chair and Reviewers,

Thank you for your time and consideration of our submission. Following the reviewers’ feedback, we have revised the manuscript to incorporate the following clarifications and additional experimental results.

  1. Distinctions from related methods — Introduction and Sec. 2.1. We clarified the distinction from VideoChat-Flash’s clip-level compression and expanded the discussion of MMG-Vid and EchoPrune, emphasizing how C²V conditions selection on the preceding compressed output and evaluates candidates jointly with the anchors already selected.

  2. Limitations — a new section following the references. We added discussion of the limitations of vision-encoder attention as an importance weight and of using only the preceding compressed output as the selection reference, including possible redundant re-selection when scenes reappear after intervening content.

  3. Longer inputs and larger backbones — Appendix G, referenced from Secs. 4.3–4.4. We added comparisons with no-state MMD for 128–768 sampled frames and for Qwen3.5-27B and Qwen3.5-35B-A3B, with matched inputs, aggregation, and token budgets within each setting.

  4. Encoding-inclusive latency and resolution scaling — Appendix E.2–E.3. We added TTFT and end-to-end latency including vision encoding, together with compression latency across spatial resolutions. These results are referenced from the efficiency analysis in Sec. 4.4.

  5. Computational cost of state conditioning — Appendix E.4. We added compression overhead, end-to-end latency, and peak-memory comparisons with no-state MMD for the longer-input and larger-backbone settings in Appendix G.

We also corrected a minor tabulation error (swapped TempCompass and VideoMME entries) in the Qwen2.5-VL-7B block of Table 1, with no effect on the underlying scores, five-benchmark averages, or main conclusions.

Further updates for the final version. Building on the results reported in our author responses, we will extend our comparisons with the additional baselines to more backbones and retention ratios, evaluating both feature coverage and benchmark performance. We will incorporate these expanded results into the final version. Thank you again for your time and thoughtful consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.