acceptodds
Under review as a conference paper at ICLR 2027

CoMo: Efficient Video Transformer via Codec Residual and Motion-Field Geometry

Abstract

Transformers have demonstrated remarkable capabilities in video understanding, but their computational cost grows rapidly with the large number of spatiotemporal tokens. Existing token reduction approaches commonly rely on learned selection modules, intermediate model features, or comparisons between spatially co-located patches, introducing additional overhead, requiring redundant tokens to be partially processed, or failing to recognize redundancy under spatial displacement. Video codecs, in contrast, have already characterized temporal redundancy through motion compensation and residual coding. In this paper, we systematically investigate codec-derived residuals and motion vectors as lightweight signals for video token reduction. We find that residuals provide an effective measure of information not explained by motion-compensated prediction, whereas raw motion-vector magnitude captures absolute displacement but overlooks spatial variations in the motion field. Based on these findings, we introduce , a training-free and model-agnostic framework that combines codec residuals with motion-field geometry to adaptively remove motion-compensated redundancy before the vision encoder. Across action recognition and video language understanding benchmarks, CoMo consistently improves the tradeoff between performance and efficiency, reducing VLM computation by up to 53.2% and accelerating inference by up to while retaining accuracy comparable to dense inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Transformer, Token Pruning, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WH5pOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

To help reduce the cost of processing visual information, the authors propose CoMo. It extracts residual and motion vectors from the video codec to remove redundancy before the vision towers. The experimental results demonstrate that CoMo can achieve a good accuracy-efficiency balance.

Critical strengths
  1. The presentation of this paper is clear. This can help me quickly understand the problem they want to solve and makes their idea easy to follow.
  2. The topic is critical. The VLMs are powerful, but using them to process long videos is always costly. The efforts of trying to solve this problem are good.
Critical weaknesses

I have a few concerns, and some of them are critical:

  1. From the novelty perspective, there is some existing work that presents similar ideas, but the authors only mentioned Mage-VL. For example, MeToM from CVPR 2026 has a similar idea. There are also a few unpublished works on arXiv presenting a similar idea. They should be part of the comparison or at least in the literature discussion. The authors should clearly explain why CoMo is unique and novel.
  2. One of the biggest concerns is: is this pruning before the vision encoder safe? Vision encoders extract patch features based on all context patches. Although the authors haven't provided code yet, I specifically did a small test and found that the feature of the same patch changed after I randomly removed some patches. For the output meaning, it is also a bit random: sometimes it's similar in meaning to the full pass, but sometimes it's broken. And if we further consider the vision encoders of different VLMs, they process patches in different ways. For example, Qwen VL's temporal step is 2. How will CoMo handle this situation?
  3. Another interesting point I would like to discuss is: from Table 5, it seems like Random is a more balanced method, which has close to top-tier performance and zero cost to think about and implement. This may also raise another question: is the success from pruning due to a carefully designed pruning method or just from the natural redundancy in VLMs?
Questions for authors

Please check my comments on the critical weaknesses.

1 comment

We thank the reviewer for the time and effort in reviewing our paper.

"1. Novelty and related work (MeToM and arXiv works)"

We thank the reviewer for pointing out these relevant works. We acknowledge that our related-work discussion is missing some recent works and have expanded it to include MeToM and OneVision-Encoder, clarifying their relationship to CoMo. However, our novelty claim is not the use of codec metadata itself, but a systematic analysis of which codec signals support effective token selection. MeToM uses residuals to guide early merging, GoP packet sizes to allocate temporal budgets, and multi-layer attention for further merging inside the LLM. CoMo instead studies residual and motion signals separately, demonstrates the value of local motion geometry over raw motion magnitude, and combines the signals through a lightweight pre-encoder selection rule without relying on internal model features or attention. Our fusion analysis further shows that a shared pair of thresholds achieves performance comparable to the stronger individual signal across datasets, avoiding per-task signal selection or retuning. We also compare CoMo with an OneVision-Encoder-style selector (combining raw motion magnitude and residual energy) under the same training-free setting, where CoMo achieves higher accuracy while using fewer TFLOPs.

MethodTFLOPs ↓MotionBench ↑
OV-Encoder-style32.2157.27
CoMo (ours)31.2357.94

Tab. W1: Comparison with an OV-Encoder-style selector (raw MV magnitude + residual) on MotionBench.

"2. Safety of pruning before the vision encoder"

We sincerely thank the reviewer for this insightful comment and for testing pre-encoder patch removal. To evaluate the impact of pre-encoder pruning, we compare the retained tokens' ViT output features and the LLM's final hidden states against their unpruned (dense) counterparts, alongside the accuracy gap between pre- and post-ViT pruning and the prediction flip rate (Tab. W1). Because CoMo only removes patches that are already predictable from reference frames (low residual and low motion deformation), it preserves the highest feature cosine similarity in both the vision encoder and the LLM, yielding the smallest pre-vs-post accuracy drop and the lowest Correct→Wrong flip rate among all methods.

MethodEnc. cos ↑LLM cos ↑Post-ViTPre-ViTΔ Acc. ↑Correct→Wrong ↓
Random0.7400.993358.3157.49−0.825.4%
RLT0.8350.994458.5957.59−1.005.2%
CoMo (ours)0.8520.995358.4657.94−0.523.6%

Tab. W2: Pre-encoder pruning analysis on MotionBench (Qwen3-VL-8B). Enc. / LLM cos: feature cosine similarity to dense inference for retained ViT tokens and the final LLM answer state; Post- / Pre-ViT: accuracy when applying the same mask after or before the ViT (ΔAcc.=Pre−Post\Delta\text{Acc.} = \text{Pre} - \text{Post}); Correct→Wrong: flip rate from correct dense predictions.

"3. Compatibility with different vision encoders"

CoMo adapts to different vision encoders by aligning its pruning mask with each encoder's native spatiotemporal token unit (Dt×Dh×DwD_t \times D_h \times D_w) before the ViT forward pass. Specifically, as defined in Eqs. (5) and (7), for encoders that group frames and patches (such as Qwen-VL's 2-frame tubelets and 2×22\times 2 spatial merge units), CoMo applies spatial mean pooling over the spatial footprint Ωi\Omega_i and temporal max pooling over the temporal window Ti\mathcal{T}_i, so that each token unit is kept or dropped as a whole. Each retained token then preserves its original 3D positional coordinate (t,h,w)(t, h, w) from the full grid, allowing CoMo to run unchanged across VideoMAE, Qwen2.5-VL, Qwen3-VL, and GLM-4.1V without modifying any internal ViT layers. (can be refer from main paper Line 254 - 256 & 284 - 288)

"4. Random baseline: method design or natural redundancy?"

We thank the reviewer for this observation. In Tab. 5, the Random baseline used a fixed retention ratio that consumed more TFLOPs than CoMo's adaptive threshold selection. When we match Random to CoMo's exact per-video token budget (Tab. W2), CoMo consistently outperforms Random across both benchmarks while automatically adapting the token count to each video's motion complexity, confirming that the gains come from our codec-guided selection design rather than natural VLM redundancy alone.

MethodMotionBenchFPSBench (Avg@4)
Random (per-video budget matched to CoMo)56.60 ± 0.4429.62 ± 0.54
CoMo (ours)57.9430.70

Tab. W3: Comparison with Random under the same per-video token budget on Qwen3-VL-8B (mean ±\pm std over 5 random seeds).

Reviewer_JdTHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a training-free token pruning method that leverages residuals and motion vectors extracted during video decoding to score and prune redundant tokens before the vision encoder, reducing the computational cost of video understanding.

Critical strengths
  • By pruning redundant video tokens before they enter the vision encoder, the method directly reduces the computation required by the encoder itself.
  • Training-free and model-agnostic. The method leverages codec residuals and motion vectors without requiring additional training, making it potentially easy to integrate into existing models.
  • Clear presentation. The paper is well-structured and easy to follow.
Critical weaknesses
  • The motivation for local area change is unclear. Table 2 shows relatively small differences between different motion descriptors. Divergence appears to be a simpler alternative with comparable or better performance, making the additional local area change design seem unnecessary.

  • Limited gains over existing token selection methods. As shown in Table 3, the proposed method does not appear to offer a clear improvement over previous video token selection methods [1, 2] in terms of the accuracy-computation trade-off.

  • The benefit of combining two signals is not clearly demonstrated. Table 4 does not show a consistent advantage from combining residuals and motion vectors. The main benefit seems to be avoiding the need to select a signal for different tasks, rather than improving the overall accuracy-efficiency trade-off.

  • Limited novelty and unclear practical gains. Using codec residuals and motion vectors before the vision encoder is not entirely new, as similar ideas have been explored in [3]. The main contribution here is applying this design to training-free token selection. However, the experimental results do not demonstrate clear gains in the accuracy-computation trade-off, leaving the practical value of this adaptation insufficiently established.

[1] Ding S, Zhao P, Zhang X, et al. Prune spatio-temporal tokens by semantic-aware temporal accumulation. ICCV, 2023.

[2] Hao X, Li G, Gowda S N, et al. Principles of visual tokens for efficient video understanding. ICCV, 2025.

[3] Tang F, An X, Yan Y, et al. OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence. arXiv:2602.08683, 2026.

Questions for authors
  • What is the specific benefit of local area change over using divergence alone, given the results in Table 2?

  • How does the proposed method compare with FastVID and FlashVID across different token budgets? Does pruning before the vision encoder provide a clear advantage over these methods?

  • Compared with [3], what are the key advantages of the proposed approach to leveraging residuals and motion vectors for token selection?

1 comment

We thank the reviewer for the time and effort in reviewing our paper.

"1. Motivation of local area change"

We agree with the reviewer that the gap between local area change and divergence is small. Our primary finding is that spatial derivatives of the motion field substantially outperform raw motion-vector magnitude (Tab. 3). Within the derivative family, while divergence simply sums horizontal and vertical velocity gradients independently, local area change measures the true 2D area expansion or contraction of a patch by also accounting for rotation and shear (Appendix B, Eq. 9). As shown in Tab. J1, when evaluated as a standalone geometry-only selector on VLM benchmarks, local area change yields a modest but consistent advantage over divergence.

Motion descriptor (geometry only)MotionBenchFPSBench (Avg@4)
Divergence56.5730.45
Local area change56.9430.70

Tab. J1: Geometry-only comparison between local area change and divergence on Qwen3-VL-8B.

"2a. Gains over prior video token selection methods in action recognition."

We thank the reviewer for pointing out these works and will include them in the related work. We would like to clarify that our action-recognition experiments (Tab. 3) serve as a controlled analysis of individual codec signals rather than as a comparison of complete methods. CoMo differs from both works in its setting. LITE [1] train a token selector for each dataset, supervised by Grad-CAM scores computed with ground-truth labels, whereas CoMo requires no training and derives its signals directly from the video bitstream. STA [2] prunes tokens progressively inside the encoder based on intermediate features, so the early layers still process all tokens, whereas CoMo removes tokens before the encoder. Finally, both methods are designed for action-recognition backbones, whereas CoMo applies unchanged to off-the-shelf VLMs (Tab. 5).

"2b. Comparison with FastVID / FlashVID across token budgets"

We conduct a compute-matched comparison against FastVID and FlashVID on MotionBench (Tab. J2). At a comparable computational budget, CoMo achieves higher accuracy while still using fewer TFLOPs, demonstrating the advantage of pruning redundant tokens before the vision encoder rather than after it.

MethodTFLOPs ↓MotionBench ↑
FastVID32.8957.69
FlashVID32.9857.62
CoMo (ours)31.2357.94

Tab. J2: Compute-matched comparison with FastVID and FlashVID on MotionBench (Qwen3-VL-8B).

"3. Benefit of combining two signals"

As shown in Tab. 4, neither signal consistently dominates across datasets. Fusion achieves performance comparable to the stronger individual signal using a single pair of thresholds across datasets, without per-task signal selection or retuning. This is a practical benefit: choosing the stronger signal in advance would require task-specific evaluation, whereas CoMo provides a unified selection rule that can be applied directly. Its value therefore lies in consistent cross-task performance and reduced tuning requirements, rather than a large accuracy gain over the best individual signal on each dataset.

"4. Novelty relative to OneVision-Encoder"

We thank the reviewer for highlighting OneVision-Encoder [3]. CoMo differs from [3] in two key respects: (1) CoMo is training-free and model-agnostic, plugging directly into off-the-shelf VLMs without retraining a dedicated visual backbone as in [3]; and (2) while [3] combines raw motion magnitude and residual energy, we systematically analyze each codec signal and replace raw magnitude with local motion geometry. As shown in Tab. J3, CoMo outperforms an OV-Encoder-style selector (raw MV magnitude + residual) on MotionBench while using fewer TFLOPs.

MethodTFLOPs ↓MotionBench ↑
OV-Encoder-style32.2157.27
CoMo (ours)31.2357.94

Tab. J3: Comparison with an OV-Encoder-style selector (raw MV magnitude + residual) on MotionBench.

[1] Ding S, Zhao P, Zhang X, et al. Prune spatio-temporal tokens by semantic-aware temporal accumulation. ICCV, 2023.

[2] Hao X, Li G, Gowda S N, et al. Principles of visual tokens for efficient video understanding. ICCV, 2025.

[3] Tang F, An X, Yan Y, et al. OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence. arXiv:2602.08683, 2026.

Reviewer_FMoKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoMo, a training-free approach to efficient video understanding that selects informative visual tokens before the vision encoder. It leverages two signals available from video decoding: codec prediction residuals and motion-field geometry, measured through a local area-change descriptor. Tokens are retained when either signal exceeds its threshold. Experiments across action recognition and video-language benchmarks demonstrate substantial computational savings while largely preserving the performance of pretrained models.

Critical strengths
  1. CoMo provides a practical and training-free solution for accelerating pretrained video models. Pre-encoder pruning reduces both computational cost and inference latency across multiple architectures.
  2. The method makes effective use of readily available codec information. In particular, the experiments show that spatial motion-field variations are more informative than raw motion-vector magnitude for token selection.
Critical weaknesses
  1. The proposed local area-change descriptor lacks a convincing justification over simpler alternatives. In Table 2, it improves Top-1 accuracy over divergence by only 0.02 percentage points, despite the clear gains of motion-field derivatives over raw motion-vector magnitude. Moreover, Table 4 shows virtually no additional benefit from combining residual and geometry signals. These results fail to establish the necessity of the specific geometric formulation or signal fusion.
  2. Several controlled Top-K comparisons in Table 3 show only marginal accuracy improvements of 0.01–0.30 percentage points, without confidence intervals or paired significance tests to establish their reliability.It is also somewhat surprising that the Random baseline remains highly competitive in Table 5.
Questions for authors
  1. Would simpler descriptors, such as divergence, achieve comparable results when used in the complete CoMo framework? What additional benefit does the proposed signal fusion provide over the stronger individual signal?
  2. Could the authors provide statistical evidence supporting the small accuracy improvements, along with comparisons against simple baselines at matched computational budgets?
2 comments

We thank the reviewer for the time and effort in reviewing our paper.

"1. Justification of local area change over divergence"

We agree with the reviewer that local area change and divergence achieve very similar accuracy. Geometrically, local area change is det⁡(I+J)−1=tr⁡(J)+det⁡(J)\det(\mathbf{I} + \mathbf{J}) - 1 = \operatorname{tr}(\mathbf{J}) + \det(\mathbf{J}), whereas divergence is ∣tr⁡(J)∣|\operatorname{tr}(\mathbf{J})|; for frame-normalized motion fields, the second-order term det⁡(J)\det(\mathbf{J}) is small, so the two descriptors behave similarly at mild pruning thresholds. As shown in Tab. F1, both descriptors perform comparably at mild thresholds, while local area change holds a better performance in (τr,τa)=(1.8,0.6)(\tau_r, \tau_a) = (1.8, 0.6). We will add this experiment in our revised version.

(τ_r, τ_a)Local area changeDivergence
(0.6, 0.2)57.9457.96
(1.2, 0.4)56.2256.15
(1.8, 0.6)55.4555.20

Tab. F1: Local area change v.s. Divergence in complete CoMo setting.

"2. Benefit of signal fusion"

We thank the reviewer for raising this point. As shown in Tab. 4 in our main paper, neither single signal consistently dominates across tasks (e.g., residual is stronger on appearance-heavy benchmarks, whereas motion geometry is stronger on fine-grained temporal benchmarks such as FPSBench and SSv2). The primary benefit of fusion is cross-task robustness without per-dataset signal selection or retuning: using a single fixed pair of thresholds, CoMo matches the stronger individual signal on every benchmark. In practice, knowing which single signal is better in advance would require task-specific validation, whereas fusion provides a unified, training-free selection rule that works reliably out of the box.

"3. Statistical significance of small improvements"

We thank the reviewer for this suggestion and have re-run Tab. 3 with paired 95% bootstrap confidence intervals across the validation videos. As shown in Tab. R3 below, codec residual (b) provides a marginal gain over RGB difference (a) on K400 and SSv2 (performing comparably without pixel-level frame differencing), whereas motion-field derivatives (d, e) achieve statistically significant improvements over raw motion-vector magnitude (c) across all settings, especially on motion-sensitive SSv2.

IDMethodK400 50%K400 60%SSv2 50%SSv2 60%
–Dense inference79.8379.8370.5370.53
(a)RGB difference (RLT)78.8579.1369.4769.90
(b)Codec residual78.92 (+0.07 ± 0.26)79.24 (+0.11 ± 0.23)69.52 (+0.05 ± 0.29)69.99 (+0.03 ± 0.25)
(c)Motion-vector magnitude78.6479.0668.0569.02
(d)Motion Jacobian norm78.84 (+0.20 ± 0.18)79.35 (+0.29 ± 0.20)69.65 (+1.51 ± 0.31)70.08 (+1.06 ± 0.27)
(e)Local area change78.99 (+0.35 ± 0.23)79.35 (+0.29 ± 0.20)69.60 (+1.56 ± 0.30)70.07 (+1.05 ± 0.27)

Tab. R3: Tab. 3 re-run with per-sample predictions. Values in parentheses are paired gains over (a) for (b) and over (c) for (d)–(e), ± the half-width of the 95% bootstrap confidence interval over videos.

"4. Competitiveness of the Random baseline"

In Tab. 5, the Random baseline used a fixed retention ratio that consumed more TFLOPs than CoMo's adaptive threshold selection. When we match Random to CoMo's exact per-video token budget (Tab. F2), CoMo consistently outperforms Random across both benchmarks, confirming that codec-guided selection brings clear gains beyond natural VLM redundancy.

MethodMotionBenchFPSBench (Avg@4)
Random (per-video budget matched to CoMo)56.60 ± 0.4429.62 ± 0.54
CoMo (ours)57.9430.70

Tab. F2: Comparison with Random under the same per-video token budget on Qwen3-VL-8B (mean ±\pm std over 5 random seeds).

Thank you for the additional experiments and clarifications. The statistical analysis and the comparison with Random baseline are helpful and address some of my concerns.

However, I am not fully convinced by the brief analysis of signal fusion without additional experimental evidence. While I appreciate the effort to compare area change with divergence, I still find the added benefit of area change rather limited.

Overall, while some of the additions are helpful, the limited support for the two design choices has made me less convinced of the strength of the paper’s contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.