- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
To help reduce the cost of processing visual information, the authors propose CoMo. It extracts residual and motion vectors from the video codec to remove redundancy before the vision towers. The experimental results demonstrate that CoMo can achieve a good accuracy-efficiency balance.
Critical strengths
- The presentation of this paper is clear. This can help me quickly understand the problem they want to solve and makes their idea easy to follow.
- The topic is critical. The VLMs are powerful, but using them to process long videos is always costly. The efforts of trying to solve this problem are good.
Critical weaknesses
I have a few concerns, and some of them are critical:
- From the novelty perspective, there is some existing work that presents similar ideas, but the authors only mentioned Mage-VL. For example, MeToM from CVPR 2026 has a similar idea. There are also a few unpublished works on arXiv presenting a similar idea. They should be part of the comparison or at least in the literature discussion. The authors should clearly explain why CoMo is unique and novel.
- One of the biggest concerns is: is this pruning before the vision encoder safe? Vision encoders extract patch features based on all context patches. Although the authors haven't provided code yet, I specifically did a small test and found that the feature of the same patch changed after I randomly removed some patches. For the output meaning, it is also a bit random: sometimes it's similar in meaning to the full pass, but sometimes it's broken. And if we further consider the vision encoders of different VLMs, they process patches in different ways. For example, Qwen VL's temporal step is 2. How will CoMo handle this situation?
- Another interesting point I would like to discuss is: from Table 5, it seems like Random is a more balanced method, which has close to top-tier performance and zero cost to think about and implement. This may also raise another question: is the success from pruning due to a carefully designed pruning method or just from the natural redundancy in VLMs?
Questions for authors
Please check my comments on the critical weaknesses.
We thank the reviewer for the time and effort in reviewing our paper.
"1. Novelty and related work (MeToM and arXiv works)"
We thank the reviewer for pointing out these relevant works. We acknowledge that our related-work discussion is missing some recent works and have expanded it to include MeToM and OneVision-Encoder, clarifying their relationship to CoMo. However, our novelty claim is not the use of codec metadata itself, but a systematic analysis of which codec signals support effective token selection. MeToM uses residuals to guide early merging, GoP packet sizes to allocate temporal budgets, and multi-layer attention for further merging inside the LLM. CoMo instead studies residual and motion signals separately, demonstrates the value of local motion geometry over raw motion magnitude, and combines the signals through a lightweight pre-encoder selection rule without relying on internal model features or attention. Our fusion analysis further shows that a shared pair of thresholds achieves performance comparable to the stronger individual signal across datasets, avoiding per-task signal selection or retuning. We also compare CoMo with an OneVision-Encoder-style selector (combining raw motion magnitude and residual energy) under the same training-free setting, where CoMo achieves higher accuracy while using fewer TFLOPs.
Tab. W1: Comparison with an OV-Encoder-style selector (raw MV magnitude + residual) on MotionBench.
"2. Safety of pruning before the vision encoder"
We sincerely thank the reviewer for this insightful comment and for testing pre-encoder patch removal. To evaluate the impact of pre-encoder pruning, we compare the retained tokens' ViT output features and the LLM's final hidden states against their unpruned (dense) counterparts, alongside the accuracy gap between pre- and post-ViT pruning and the prediction flip rate (Tab. W1). Because CoMo only removes patches that are already predictable from reference frames (low residual and low motion deformation), it preserves the highest feature cosine similarity in both the vision encoder and the LLM, yielding the smallest pre-vs-post accuracy drop and the lowest Correct→Wrong flip rate among all methods.
Tab. W2: Pre-encoder pruning analysis on MotionBench (Qwen3-VL-8B). Enc. / LLM cos: feature cosine similarity to dense inference for retained ViT tokens and the final LLM answer state; Post- / Pre-ViT: accuracy when applying the same mask after or before the ViT (ΔAcc.=Pre−Post); Correct→Wrong: flip rate from correct dense predictions.
"3. Compatibility with different vision encoders"
CoMo adapts to different vision encoders by aligning its pruning mask with each encoder's native spatiotemporal token unit (Dt×Dh×Dw) before the ViT forward pass. Specifically, as defined in Eqs. (5) and (7), for encoders that group frames and patches (such as Qwen-VL's 2-frame tubelets and 2×2 spatial merge units), CoMo applies spatial mean pooling over the spatial footprint Ωi and temporal max pooling over the temporal window Ti, so that each token unit is kept or dropped as a whole. Each retained token then preserves its original 3D positional coordinate (t,h,w) from the full grid, allowing CoMo to run unchanged across VideoMAE, Qwen2.5-VL, Qwen3-VL, and GLM-4.1V without modifying any internal ViT layers. (can be refer from main paper Line 254 - 256 & 284 - 288)
"4. Random baseline: method design or natural redundancy?"
We thank the reviewer for this observation. In Tab. 5, the Random baseline used a fixed retention ratio that consumed more TFLOPs than CoMo's adaptive threshold selection. When we match Random to CoMo's exact per-video token budget (Tab. W2), CoMo consistently outperforms Random across both benchmarks while automatically adapting the token count to each video's motion complexity, confirming that the gains come from our codec-guided selection design rather than natural VLM redundancy alone.
Tab. W3: Comparison with Random under the same per-video token budget on Qwen3-VL-8B (mean ± std over 5 random seeds).