acceptodds
Under review as a conference paper at ICLR 2027

CoGCo: Codec Guided Training-free Video-LLM Token Compressor

14 views

Abstract

Video Large Language Models (Video LLMs) process thousands of visual tokens from sampled frames, making redundant video representations a major source of inference cost. Training-free video token compression must preserve persistent scene context and subtle temporal changes, yet token saliency alone does not reveal which information requires an additional representative. We present CoGCo, which exploits the structure of standard video codecs to separate these complementary responsibilities. Reference-frame tokens establish semantic anchors through attention- and diversity-aware selection, supplemented by reference locations supported by subsequent updates. Prediction-frame selection combines camera-compensated motion vectors, residuals, and within-frame diversity to preserve complementary temporal evidence. It excludes a small fraction of the highest-attention tokens from independent selection, as our analysis suggests that such features primarily capture overall semantic context rather than fine-grained updates. Moreover, we leverage codec frame packets to efficiently allocate the token budget. Across four benchmarks and four backbones, CoGCo achieves higher aggregate accuracy than the SOTA training-free compression baselines at evaluated retention settings of 15%, 20%, and 25%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Large Language Models, Visual Token Compression, Training-Free Inference, Video Codec

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_X1D3OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CoGCo, a training-free video token compression method guided by codec reference-update framework. It selects semantic anchors from I-frames and complementary P-frame updates using attention, feature diversity, motion vectors, and residuals. Packet sizes guide budget allocation, while update-guided reference recovery and global completion improve coverage. Codec-guided aggregation incorporates compatible unselected features into retained representatives without updating model weights. Experiments on four backbones and four video-QA benchmarks report improved average accuracy over existing methods across multiple retention ratios settings.

Critical strengths
  1. The paper is well written and clearly presented. The overall pipeline, individual components, and their connections to video coding are easy to follow, while the overview figure provides useful explanations of the method.

  2. The motivation is clear and intuitive. Persistent appearance and temporal updates serve different roles in video understanding, so using codec information to coordinate their representation provides a reasonable approach to token compression.

  3. The experimental evaluation is reasonably extensive. Four backbones, four benchmarks, multiple compression budgets, component ablations, and hyperparameter sensitivity analyses provide a useful assessment of the method.

Critical weaknesses
  1. The distinction from related temporal-reuse methods needs stronger justification. The reference-update organization is closely related to codec-driven sparsity in OneVision-Encoder [1] and reference-feature reuse in STC [2], although their implementations and operating stages differ. OneVision-Encoder is discussed, but neither approach is experimentally compared. Both address visual encoding costs, which can dominate video inference, whereas CoGCo compresses already encoded features. The authors should clarify the methodological differences and compare their accuracy-efficiency trade-offs under compatible settings.

  2. The improvements over FlashVID [3] are modest. At 15% nominal retention, Table 1 reports average accuracy gains of 0.5, 0.1, and 0.9 percentage points on LLaVA-OneVision, LLaVA-Video, and InternVL3.5, respectively, averaging approximately 0.5 points. Figure 4 also shows only a 2.6% TTFT reduction, from 1320 ms to 1286 ms, at comparable accuracy. These margins require stronger evidence of practical significance and statistical reliability.

  3. The efficiency comparison is insufficiently comprehensive. Figure 4 only compares CoGCo with FlashVID and UniComp [4], leaving its advantage over simpler methods such as VisionZip [5] unclear. The authors should include VisionZip under both matched token budgets and comparable accuracy, with a breakdown of visual encoding, compression, and LLM prefill costs. The analysis also omits peak GPU memory consumption, an important metric for assessing deployment feasibility and scalability. Since codec-sidecar preparation is excluded from the reported TTFT, its preprocessing cost and amortization should also be reported.

  4. The sparse frame settings insufficiently validate codec-guided temporal aggregation. General benchmarks use only 32 or 64 frames, which can leave large temporal gaps in long videos and reduce the predictability between adjacent sampled frames. Since codec metadata is extracted from the prepared sequence, the resulting correspondence and redundancy may differ substantially from densely sampled video. Experiments with 128, 256, and 512 frames on VideoMME-Long and LongVideoBench are needed to establish the design’s effectiveness and scalability.

  5. Task diversity remains limited. All evaluated benchmarks concern video question answering (VideoQA), leaving applicability to temporal grounding and video captioning unverified. These tasks require precise temporal localization and broader event coverage, respectively, and would provide stronger evidence that compression preserves information beyond QA-specific cues.

  6. Model-scale generalization is insufficiently evaluated. All tested backbones contain fewer than 10B parameters. Experiments on the MoE model Qwen3.5-35B-A3B and the larger dense model Qwen3.5-27B would help establish whether the method remains effective on larger, practically relevant models that are still accessible to research laboratories.

[1] Feilong Tang et al. “OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence.” arXiv preprint arXiv:2602.08683, 2026.

[2] Yiyu Wang, Xuyang Liu, Xiyan Gui, Xinying Lin, Boxue Yang, Chenfei Liao, Tailai Chen, and Linfeng Zhang. “Accelerating Streaming Video Large Language Models via Hierarchical Token Compression.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.

[3] Ziyang Fan, Keyu Chen, Ruilong Xing, Yulin Li, Li Jiang, and Zhuotao Tian. “FlashVID: Efficient Video Large Language Models via Training-Free Tree-Based Spatiotemporal Token Merging.” In Proceedings of the International Conference on Learning Representations (ICLR), 2026.

[4] Chao Yuan, Shimin Chen, Minliang Lin, Limeng Qiao, Guanglu Wan, and Lin Ma. “UniComp: Rethinking Video Compression Through Informational Uniqueness.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.

[5] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. “VisionZip: Longer Is Better but Not Necessary in Vision Language Models.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.

Questions for authors
  1. Can the authors provide comparisons with OneVision-Encoder and STC-Cacher under compatible settings, including separate measurements of visual encoding and LLM prefill costs?

  2. Can the authors add efficiency comparisons with VisionZip at matched token budgets and comparable accuracy, and separately report codec preprocessing costs?

  3. Can the authors evaluate more input frames on long-video benchmarks, reporting accuracy, TTFT, and peak memory?

  4. Can the authors provide additional experiments on more video tasks beyond VideoQA, such as temporal grounding and video captioning?

  5. Can the authors evaluate larger models, such as Qwen3.5-35B-A3B and Qwen3.5-27B, to demonstrate generalization across model scales and architectures?

6 comments
AuthorsOpenReview ↗

Response to Reviewer X1D3 (1)

We thank the reviewer for recognizing the clarity, motivation, and breadth of our evaluation. To address your questions, we clarify our methodological distinctions from pre-encoder/streaming approaches alongside empirical controls, demonstrate upper-bound gap recovery and TTFT/memory savings, and provide new experiments extending CoGCo to denser frame inputs, the larger Qwen3.5-35B-A3B model, and non-QA tasks.

R1 Distinction from OneVision2 Encoder and STC

We thank the reviewer for highlighting these works and noting that their implementations and operating stages differ from CoGCo. Specifically, that OneVision-Encoder [3] [15] relies on codec-aligned pre-training to drop patches before/during visual encoding and STC [4] performs feature-level streaming cache reuse, whereas CoGCo is a strictly training-free, post-encoder framework. Beyond operating stage and training-free compatibility, CoGCo differs methodologically in three concrete ways: (1)Backward Reference Recovery: Unlike forward sparsification or forward feature caching, CoGCo traces P-frame prediction chains backward to recover overlooked I-frame anchor regions that carry structural context. (2)Motion-Compensated Residual Filtering: Rather than relying solely on raw block sparsity or embedding similarity, CoGCo separates global camera motion from local object dynamics, suppressing camera-induced false updates. (3)Packet-Guided Allocation and Cross-Frame Aggregation: Instead of discarding unselected P-frame tokens, CoGCo uses bitstream packet sizes to allocate budgets and aggregates compatible P-frame update features into retained anchor tokens along motion-verified correspondences. To provide an empirical reference under comparable token budgets (Table R5), we compare OneVision2 and STC Pruner results against CoGCo on VideoMME and MVBench. While OneVision2 requires 65% retention on VideoMME to reach 58.3%, CoGCo achieves 59.5% at 25% retention without modifying or retraining the vision encoder.

Table R5 Codec-gating controls and STC results

Method/variantRet. VideoMME / MVBenchVideoMMEMVBench
OneVision265%/23%58.351.0
STC Pruner25% / —57.1—
CoGCo25%/25%59.558.0

R2 Practical value and reliability of the gains

We thank the reviewer for asking to clarify the practical significance and reliability of these gains. In training-free video token compression, strong baselines already operate within 1.3–4.2 points of the full-token ceiling, so sub-point improvements recover a substantial share of the remaining accuracy gap. Specifically, CoGCo recovers up to 42% of the remaining gap between FlashVID and the upper bound on InternVL3.5-8B, 30% on Qwen3-VL-8B, and 31% on LLaVA-Video, while achieving >100% recovery across all budgets on LLaVA-OV. Translated into compute savings at matched accuracy, CoGCo at 15% retention reaches 57.0% on LLaVA-OV v.s. FlashVID's 56.7% at 25%—eliminating 40% of visual tokens, accelerating LLM prefill by 46.9%, reducing total model TTFT by 17.0% (1,321 to 1,096 ms). Finally, these gains hold consistently across all 12 backbone–budget, and widen in more demanding regimes: reaching +0.5 to +1.1 points on the larger Qwen3.5-35B-A3B model (Table R10), alongside non-QA tasks (Tables R9a–R9b).

R3 VisionZip efficiency and GPU memory

We evaluate VisionZip on LLaVA-OV, Table R6 compares accuracy and latency at matched token budgets and comparable accuracy.

Table R6. Accuracy and latency breakdown on LLaVA-OV (ms).

MethodRatioAvg. Acc. (%)EncoderComp.LLM prefillTTFT
CoGCo15%576471373121,096
VisionZip15%5359752320971
VisionZip35%56598539021,555

At the matched 15% budget, CoGCo improves average accuracy by 3.7 pp over VisionZip, with 12.9% higher TTFT. At comparable accuracy, CoGCo at 15% exceeding VisionZip at 35%, while reducing TTFT by 29.5%. These results establish a more favorable accuracy–latency trade-off at comparable predictive performance. We further measure peak GPU memory across visual encoding, compression, and LLM prefill for CoGCo and FlashVID.

Table R7. Peak GPU memory by stage (GB).

RatioMethodEncoderCompressionLLM prefill
15%CoGCo18.515.515.4
15%FlashVID18.515.715.5
25%CoGCo18.515.416.8
25%FlashVID18.515.516.9

CoGCo's compression-stage peak and overall measured peak matches FlashVID. At 15% retention, CoGCo requires 15.4 GB during LLM prefill, compared with 16.9 GB for FlashVID at 25%, while achieving comparable average accuracy. Finally, we measure codec-sidecar preparation separately from model inference. H.264 re-encoding takes 1,453 ms per video, v.s. 1,383 ms for direct decoding, an additional 70 ms. The prepared codec metadata is reusable across questions, while the online compression operations remain included in the reported model TTFT.

AuthorsOpenReview ↗

Response to Reviewer X1D3 (2)

R4 Sparse sampling and larger frame counts

CoGCo does not assume strong redundancy between frames. Instead, codec motion vectors and residuals propose candidate updates, while feature compatibility filters out unreliable associations. Table R8 reports our extended evaluation at 128 and 256 frames across LLaVA-OneVision and InternVL3.5.
Table R8. Accuracy comparison at 128 and 256 input frames on VideoMME-Long and LongVideoBench.

FramesRatioMethodVideoMME-Long OVVideoMME-Long InternVL3.5LongVideoBench OVLongVideoBench InternVL3.5Average
12825%FlashVID49.452.858.461.155.4
12825%CoGCo51.055.957.960.956.4
12820%FlashVID51.052.457.560.855.4
12820%CoGCo49.856.358.860.156.3
12815%FlashVID49.551.758.059.754.7
12815%CoGCo50.954.957.859.955.9
25625%FlashVID50.653.257.762.055.9
25625%CoGCo50.356.457.761.656.5
25620%FlashVID50.753.857.161.655.8
25620%CoGCo50.955.958.561.756.8
25615%FlashVID50.351.657.862.055.4
25615%CoGCo50.356.258.460.956.5

CoGCo outperforms FlashVID across all six frame-count and retention averages. Notably, the largest average gains occur at the tightest 15% budget, and on VideoMME-Long where InternVL3.5 improves by +3.1 to +4.6 points. Increasing density to 128/256 frames reduces sampling stride by 4×–8×, directly shifting motion vector and residual distributions to stress-test temporal generalization. Scaling further to 512 frames introduces quadratic pre-compression encoding and LLM prefill memory bottlenecks that exceed our GPU envelope for unpruned baselines and fall outside standard training-free evaluation protocols; we leave 512-frame streaming scaling as future work.

R5 Temporal grounding beyond VideoQA

We agree that evaluation beyond VideoQA should test whether compression preserves temporal evidence for a different output task. We add a temporal-grounding comparison on Charades-TimeLens with Qwen3-VL-8B because LLaVA-OneVision and LLaVA-Video do not support grounding task [13].

Table R9a Temporal grounding on Charades TimeLens with Qwen3-VL

SettingmIoU[email protected][email protected][email protected]
Full tokens47.7368.6052.4226.85
FlashVID 25%46.1465.6949.9926.32
CoGCo 25%46.7866.9651.5026.73
FlashVID 20%45.4665.6948.7124.89
CoGCo 20%46.0566.0750.1026.32
FlashVID 15%44.6164.5947.7524.32
CoGCo 15%45.4465.4549.8425.99

The improvement is consistent across localization overlap and all three recall thresholds. CoGCo raises mIoU from 46.14 to 46.78 and [email protected] from 49.99 to 51.50.
Second, we evaluate detailed video captioning on VideoDetailCaption across LLaVA-OneVision and InternVL3.5:

Table R9b. Video captioning scores on VideoDetailCaption

MethodRatioLLaVA-OneVision-7BInternVL3.5-8BAverage
Vanilla100%3.363.293.33
FlashVID25%3.082.963.02
CoGCo25%3.022.993.01
FlashVID20%3.052.932.99
CoGCo20%3.063.003.03
FlashVID15%2.972.902.94
CoGCo15%2.982.922.95

CoGCo outperforms FlashVID across all three retention ratios on InternVL3.5, confirming that it preserves fine-grained visual details.

R6 Larger dense and MoE models

We thank the reviewer for suggesting both larger architectures. To evaluate scalability, we prioritized Qwen3.5-35B-A3B (Table R10). Although Qwen3.5-35B-A3B activates 3B parameters, its 35B capacity and fine-grained expert routing make it sensitive to token distortion. Under same protocol, CoGCo achieves consistent gains across all budgets. By contrast, evaluating the 27B dense model requires substantially higher compute and memory bandwidth than was feasible to complete across all benchmarks during the rebuttal window; we are actively securing additional compute resources and will gladly include the full Qwen3.5-27B dense results in the camera-ready version.

Table R10: Evaluation on larger MoE model Qwen3.5-35B-A3B across four benchmarks

RatioMethodVideoMMEMVBenchMotionBenchLongVideoBenchAverage
25%CoGCo64.669.359.861.063.7
25%FlashVID64.569.259.559.663.2
20%CoGCo64.568.259.560.263.1
20%FlashVID63.569.059.558.062.5
15%CoGCo63.567.859.359.662.6
15%FlashVID63.166.958.857.361.5
AuthorsOpenReview ↗

References

[3] Feilong Tang et al. OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence. arXiv:2602.08683v3, 2026. https://arxiv.org/abs/2602.08683v3

[4] Yiyu Wang et al. Accelerating Streaming Video Large Language Models via Hierarchical Token Compression. arXiv:2512.00891, 2025. https://arxiv.org/abs/2512.00891

[13] Minseok Kang et al. OTT-Vid: Optimal Transport Temporal Token Compression for Video Large Language Models. arXiv:2605.11803v1, 2026. https://arxiv.org/abs/2605.11803v1

[15] Xiang An et al. LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence. arXiv:2605.25979, 2026. https://arxiv.org/abs/2605.25979

Reviewer_iQc4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoGCo, a training-free visual token compression framework for Video LLMs that leverages video codec information to distinguish persistent semantic context from temporal updates.

Critical strengths
  1. The distinction between persistent semantic context in I-frames and temporal updates in P-frames is intuitive and well motivated.
  2. The experimental evaluation is comprehensive across different datasets and compression rates.
Critical weaknesses
  1. The method involves a relatively complex pipeline with numerous operations, yet the accuracy improvements over existing baselines seem (to me) relatively modest in some settings. Could the authors provide a more detailed quantitative comparison of the computational overhead introduced by CoGCo, particularly compared with FlashVID, to help assess whether these improvements justify the additional complexity?
  2. Meanwhile, appendix shows that different hyperparameter configurations are used for each dataset.This raises my concerns about the generalizability of CoGCo. Could the authors clarify whether the proposed method can maintain its performance advantages on unseen datasets without dataset-specific hyperparameter tuning?
Questions for authors

same as weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer iQc4

We thank you for recognizing the clarity of our presentation, intuition of separating persistent context from temporal updates, and breadth of our experiments. Below, we provide a stage-wise runtime and memory profile showing our method yields a 17.0% TTFT reduction and 1.5 GB memory savings, and demonstrate that a single shared configuration per backbone preserves our gains across four benchmarks without per-dataset tuning.

R1 Computational overhead and benefit

We agree that the accuracy benefit should translate into a computational return after compression. We carefully examined the pipeline and eliminated several redundant implementation overheads: (1)Unnecessary CPU–GPU synchronization: Replaced repeated per-candidate GPU operations and synchronization with a single transfer of small similarity matrices, followed by CPU selection. (2)Redundant similarity computation: Reused previously computed normalized features and similarities instead of repeatedly recomputing them during candidate selection and merging. (3)Repeated metadata processing: Reused parent mappings across temporal support and aggregation, and batched metadata access to avoid repeated per-frame synchronization. These implementation-level optimizations preserve the algorithmic design and selection criteria while reducing CoGCo's compression overhead.

Table R1. Updated runtime and accuracy comparison. Runtime in milliseconds.

MethodRatioAvg. Acc. (%)EncoderCompressionLLM prefillTTFT
CoGCo15%57.06471373121,096
FlashVID25%56.7647865881,321
UniComp25%56.46482445291,401

At comparable aggregate accuracy, CoGCo achieves 17.0% lower TTFT than FlashVID and 21.8% lower than UniComp. The compression breakdown further quantifies the cost of individual components: budget allocation & preparation (7 ms), camera compensation (14 ms), evidence & temporal support (20 ms), I-frame attention selection (24 ms), P-frame candidate generation (8 ms), P-frame selection (16 ms), remaining I-frame selection (19 ms), and merge aggregation (33 ms). These operations contribute to the information-preservation benefits demonstrated by the ablations in Table 3. At comparable accuracy, CoGCo reduces prefill peak memory by 1.5 GiB compared with FlashVID. At the same token budget, CoGCo also exhibits lower compression-stage memory usage, while both methods share same encoder peak of 18.5 GiB.

R2 Shared parameters across datasets

We agree that practical deployment should not require a separate configuration for every dataset. In the original submission, Table 8 reported dataset-specific parameter tuples to reflect fine-grained adaptation to varying benchmark characteristics, following standard evaluation practices in prior training-free compression studies [2, 10, 11, 12]. To directly test whether CoGCo's performance depends on per-dataset settings, we re-evaluated all three backbones using one fixed default configuration per backbone across all four datasets and all three retention ratios (Table R4). The single shared configuration preserves the advantage over both FlashVID and UniComp across all nine backbone–budget averages, with a mean absolute change of only 0.09 percentage points relative to Table 8. This confirms that a single default preset per backbone generalizes robustly across diverse benchmarks and budgets, while Table 8 simply illustrates the minor upper-bound headroom from task-aware calibration. The fixed tuples are 1.25/2.00/0.850/0.875 for LLaVA-OneVision, 1.50/2.25/0.900/0.850 for LLaVA-Video, and 1.50/2.50/0.875/0.850 for InternVL3.5, in the parameter order of Table 8. The other shared settings remain unchanged.

Table R4 Four-benchmark average with one configuration per backbone

BackboneMethod25%20%15%
LLaVA-OVFlashVID56.756.656.5
LLaVA-OVUniComp56.456.155.6
LLaVA-OVCoGCo shared57.2 (+0.02)57.0 (−0.23)57.0 (−0.04)
LLaVA-VideoFlashVID58.558.358.3
LLaVA-VideoUniComp57.857.356.8
LLaVA-VideoCoGCo shared58.9 (−0.01)58.6 (−0.08)58.4 (−0.05)
InternVL3.5FlashVID60.559.559.4
InternVL3.5UniComp60.660.159.1
InternVL3.5CoGCo shared61.8 (−0.04)61.2 (+0.14)60.1 (−0.21)

As suggested by Reviewer X1D3, we further evaluated CoGCo on Qwen3.5-35B-A3B, a larger MoE backbone, achieving average improvements over FlashVID of +0.5, +0.6, and +1.1 points at 25%, 20%, and 15% retention, respectively. We also transferred the existing Qwen3-VL-8B-Instruct configuration to Charades-TimeLens without task-specific tuning. At 25% retention, CoGCo achieves 46.78 mIoU versus 46.14 for FlashVID. Together, these results demonstrate generalization across model architectures and tasks, complementing our shared-configuration experiments across four QA benchmarks.

AuthorsOpenReview ↗

References

[2] Hesong Wang et al. EarlyTom: Early Token Compression Completes Fast Video Understanding. arXiv:2605.30010, 2026. https://arxiv.org/abs/2605.30010

[10] Zhihang Lin et al. Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference. arXiv:2405.05803, 2024. https://arxiv.org/abs/2405.05803

[11] Xiao Wang et al. ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understanding. arXiv:2412.20504, 2024. https://arxiv.org/abs/2412.20504

[12] Jeongseok Hyun et al. Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs. arXiv:2507.07990, 2025. https://arxiv.org/abs/2507.07990

Thank you again for your thoughtful review and constructive feedback. As the discussion period is drawing to a close, we wanted to briefly check in and see if our responses and additional results have satisfactorily addressed your concerns.

Please let us know if you have any remaining questions or if there is anything else we can clarify before the deadline. We sincerely appreciate your time and consideration.

Reviewer_vbg7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoGCo introduces a training-free video token compression framework that uses codec-derived motion, residuals, and reference relationships to guide token selection and aggregation.

Critical strengths
  • The use of codec information is well motivated. Motion and residual signals complement attention and feature similarity when identifying temporal updates. The performance drop after removing update-guided candidates provides initial support for this design.

  • The evaluation covers multiple model architectures and benchmarks. CoGCo achieves consistent improvements in aggregate accuracy across the reported backbone–budget combinations.

Critical weaknesses
  • The reported latency does not establish an end-to-end efficiency advantage. TTFT excludes video decoding, frame preparation, and codec-sidecar generation, while the pipeline additionally requires re-encoding the prepared frames. Given the approximately 2.6% latency improvement over FlashVID in Figure 4, the additional preprocessing cost should be quantified, particularly for single-query videos.

  • The benefits of the additional algorithmic complexity are not sufficiently justified. Camera-motion estimation, iterative diversity selection, reference recovery, and global aggregation introduce computation beyond token reduction itself. Several settings show only 0.1–0.4 percentage-point average gains over FlashVID, motivating component-level accuracy–latency ablations and comparisons against simpler alternatives.

  • The baseline coverage is insufficient to establish superiority over more recent relevant approaches. ForestPrune offers a directly relevant spatiotemporal compression comparison, while EarlyTom would help assess whether CoGCo remains competitive when vision-encoder computation is also reduced.

Questions for authors

See weakness above.

9 comments
AuthorsOpenReview ↗

Response to Reviewer vbg7

We thank the reviewer for the positive feedback on our codec-guided motivation and multi-backbone evaluation, as well as for focusing on practical deployment efficiency. In this response, we quantify codec preparation overhead (showing it adds only 70 ms and is amortized or avoided), report an optimized compressor that reduces end-to-end model TTFT by 17.0% at comparable accuracy, and add direct comparisons against ForestPrune and EarlyTom.

R1 Efficiency and codec preparation

We agree that codec preparation should be quantified. Re-encoding takes 1453 ms, compared with 1383 ms for direct decoding for no-codec approaches, adding only 70 ms (5.1%). Importantly, codec metadata is prepared only once per video and can be reused across multi-round questions and answers. Thus, this overhead is further amortized.

Our work focuses on post-encoder token compression and separating input preparation from model inference is also consistent with prior work such as DMC-Net [9]; EMA [8]; LLaVA-OneVision-2 [15]; and STC [4].

In practical deployments, re-encoding can be avoided. CoGCo extracts motion vectors, residuals, and reference relationships from H.264 bitstreams. Existing H.264 videos can be processed directly using the original I and P frames, while live-video encoders can generate the required structure during initial encoding. Therefore, no additional re-encoding is needed for such inputs, even for single-round inference.

R2 Computational return of the additional components

We agree that the original latency improvement was modest and that the additional components should justify their computational cost. We therefore carefully examined the inference pipeline and eliminated several redundant implementation overheads:

Unnecessary CPU–GPU synchronization: Replaced repeated per-candidate GPU operations and synchronization with a single transfer of small similarity matrices, followed by CPU-side selection.

Redundant similarity computation: Reused previously computed normalized features and similarities instead of repeatedly recomputing them during candidate selection and merging.

Repeated metadata processing: Reused parent mappings across temporal support and aggregation, and batched metadata access to avoid repeated per-frame synchronization.

These optimizations preserve the algorithmic design and selection criteria while reducing CoGCo's compression overhead.

Table R1. Updated runtime and accuracy comparison. Runtime in milliseconds.

MethodBudgetAvg. Acc. (%)EncoderCompressionLLM prefillTTFT
CoGCo15%57.06471373121,096
FlashVID25%56.7647865881,321
UniComp25%56.46482445291,401

At comparable aggregate accuracy, CoGCo achieves 17.0% lower TTFT than FlashVID and 21.8% lower TTFT than UniComp.

The compression breakdown further quantifies the cost of individual components: budget allocation and preparation (7 ms), camera compensation (14 ms), evidence and temporal support (20 ms), I-frame attention selection (24 ms), P-frame candidate generation (8 ms), P-frame selection (16 ms), remaining I-frame selection (19 ms), and merge aggregation (33 ms). These operations contribute to the information-preservation benefits demonstrated by the controlled ablations in Table 3.

R3 ForestPrune and EarlyTom

We thank the reviewer for suggesting these additional baselines. Table R3 includes the LLaVA-OneVision results of ForestPrune and EarlyTom on three benchmarks. CoGCo consistently outperforms both methods at the comparable token budgets, demonstrating its effectiveness across VideoMME, MVBench, and LongVideoBench. Notably, CoGCo achieves these improvements without modifying the pretrained vision encoder.

Table R3 Published LLaVA-OneVision results on overlapping benchmarks

MethodRatioVideoMMEMVBenchLongVideoBenchAverage
ForestPrune30%57.757.356.357.1
ForestPrune20%56.857.656.256.9
EarlyTom25%58.557.456.357.4
EarlyTom20%58.057.855.657.1
EarlyTom15%56.957.554.456.3
CoGCo25%59.558.057.958.5
CoGCo20%58.858.258.058.3
CoGCo15%58.957.758.158.2
AuthorsOpenReview ↗

References

[1] Shaobo Ju et al. ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modeling. arXiv:2603.22911, 2026. https://arxiv.org/abs/2603.22911

[2] Hesong Wang et al. EarlyTom: Early Token Compression Completes Fast Video Understanding. arXiv:2605.30010, 2026. https://arxiv.org/abs/2605.30010

[4] Yiyu Wang et al. Accelerating Streaming Video Large Language Models via Hierarchical Token Compression. arXiv:2512.00891, 2025. https://arxiv.org/abs/2512.00891

[8] Zijia Zhao et al. Efficient Motion-Aware Video MLLM. arXiv:2503.13016, 2025. https://arxiv.org/abs/2503.13016

[9] Zheng Shou et al. DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition. arXiv:1901.03460, 2019. https://arxiv.org/abs/1901.03460

[15] Xiang An et al. LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence. arXiv:2605.25979, 2026. https://arxiv.org/abs/2605.25979

Thank you again for your thoughtful review and constructive feedback. As the discussion period is drawing to a close, we wanted to briefly check in and see if our responses and additional results have satisfactorily addressed your concerns.

Please let us know if you have any remaining questions or if there is anything else we can clarify before the deadline. We sincerely appreciate your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common response

We thank the reviewers for their constructive feedback and recognition of our codec-guided reference–update motivation and multi-backbone evaluation, and we thank the Area Chair for overseeing the review process. Across the reviews, the central questions focus on (1) end-to-end efficiency and preprocessing overhead, (2) hyperparameter generalization, and (3) scalability across denser frames, larger models, and non-QA tasks.

1. Summary of Key Experimental & Analytical Responses

  • Practical Latency, Memory, and Preprocessing Return: Eliminating redundant similarity computations and CPU–GPU synchronization reduces CoGCo's compression overhead to 137 ms. At comparable accuracy on LLaVA-OneVision, CoGCo cuts LLM prefill time by 46.9%, reduces end-to-end model TTFT by 17.0%, and saves 1.5 GiB of peak prefill memory. We also quantify benchmark re-encoding separately (1,453 ms vs. 1,383 ms direct decoding, a +70 ms one-time cost amortized across queries) and show that compatible H.264 streams avoid re-encoding altogether.

  • Tuning-Free Generalization & Expanded Baselines: Using a single fixed configuration per backbone across all four benchmarks and three budgets changes accuracy by only 0.09% while outperforming FlashVID and UniComp in all backbone–budget averages. We further add comparisons against ForestPrune, EarlyTom, VisionZip, and pre-encoder codec methods (OneVision-Encoder / STC), confirming superior accuracy–latency trade-offs without modifying the pretrained vision encoder.

  • Scaling to Denser Inputs, Larger Models, and New Tasks: Extending evaluation to 128 and 256 frames on VideoMME-Long and LongVideoBench yields consistent macro-average gains over FlashVID across all six frame–budget settings. On the larger Qwen3.5-35B-A3B MoE backbone, CoGCo improves average accuracy by +0.5 to +1.1. Finally, new evaluations on Charades-TimeLens (temporal grounding) and VideoDetailCaption (detailed captioning) confirm that CoGCo preserves fine-grained temporal evidence well beyond VideoQA.

2. Summary of Revisions in the Updated Manuscript (Highlighted in Blue)

Main Paper:

  • Abstract, Introduction, and Conclusion: Expanded the described evaluation scope to incorporate the Qwen3.5-35B-A3B MoE backbone and Charades-TimeLens temporal grounding benchmark.

  • Experiments (Section 4): Integrated full benchmark evaluations on the larger Qwen3.5-35B-A3B MoE backbone (Table 3) and added Charades-TimeLens (Table 4) fine-grained temporal grounding results, updated the TTFT comparison (Figure 4) reflecting our runtime optimizations, along with a pointer to Appendix E.

Appendix:

  • Additional Generalization Experiments (Appendix E): Added cross-dataset evaluations using a single shared parameter preset per backbone (Appendix E.1), open-ended dense video captioning results on VideoDetailCaption (Appendix E.2), and extended 128- and 256-frame scaling evaluations on VideoMME-Long and LongVideoBench (Appendix E.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.