- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes IPB, a training-free method for reducing the number of video frames and visual tokens processed by a vision-language model. It uses codec information, including packet sizes and motion vectors, to select frames, and combines motion and texture cues to select patches within those frames. These decisions are made on the CPU before GPU inference, allowing selection to overlap with inference across requests. The main objective is to reduce the computational cost of video understanding without substantially compromising accuracy.
Critical strengths
The paper addresses a practical issue: frame selection and token pruning can themselves become expensive under concurrent serving. Moving these decisions to the CPU is a sensible approach, particularly when GPU-based selection competes with the model for resources. The method requires no additional training or a separate learned selector. This makes it appealing as an inference optimization. The Qwen implementation also removes patches before visual encoding, which allows the approach to save computation in the vision encoder. The accuracy–compute results are encouraging. Across several benchmarks, IPB reduces computation while retaining competitive accuracy. The evaluation across multiple backbones and the component ablations provide useful evidence that the approach has practical potential.
Critical weaknesses
The paper does not demonstrate consistent performance gains over the baseline methods. The substantial serving speedups are measured against BOLT+EVS, which incurs considerable selection overhead. Compared with Uniform+EVS, throughput is nearly identical and TTFT is slightly worse in the highlighted configuration. The results therefore support the benefit of avoiding expensive selection more clearly than they establish an advantage for IPB over simpler alternatives. It remains unclear when the codec prior helps reliably. Frame-selection gains over uniform sampling are often small and vary across tasks. Motion and coding complexity do not necessarily reflect the information needed to answer a particular question, and the paper could better characterize the cases where these signals are useful or misleading. There is also some uncertainty about robustness to encoding settings. In one tested configuration, 299 of 300 videos fall back to uniform sampling. Although accuracy is preserved, this does not demonstrate that the proposed selection mechanism remains effective under that configuration.
Questions for authors
I do not have extra question for the author
3 comments
Thank you for the detailed reply and additional experiments in the discussion. However my main concern about robustness against general video scenes remain. I would maintain the original score.
We thank the reviewer for the response and the time spent on our paper.
- Task dependence. Codec priors help when evidence is concentrated in a few segments and can hurt tasks that need continuous coverage (
Table 5). We acknowledge this as a limitation of IPB. - Robustness to encoding. In our experiments, fallback never occurred on the original Video-MME videos (0/600), and IPB never fell below uniform sampling across the encoding settings of
Table 7.
We appreciate the reviewer's feedback, which helped clarify both the strengths and the limits of IPB.
Response to Reviewer jodd
Response to Reviewer jodd
[W1]
Does IPB gain consistently over simpler baselines such as Uniform+EVS?→ Yes: −32.8% TTFT, +36.0% throughput, and +2.1 pp accuracy.The 38.0× TTFT gain is relative to BOLT+EVS. Against Uniform+EVS, IPB gains in three ways: accuracy at lower compute, serving latency and throughput on real requests, and encoder cost.
Accuracy at lower compute. At the same LLM token budget (rp = 0.375,
Table 2), IPB improves Video-MME accuracy over Uniform+EVS by 2.1 pp with 30.7% fewer total FLOPs.Serving on real requests. We serve both methods on Qwen3-VL-4B, the backbone of
Table 2, with real Video-MME videos of mixed length (rp = 0.375, the same LLM visual-token budget).At C = 64, IPB reduces mean TTFT by 32.8% and raises throughput by 36.0% over Uniform+EVS.
Revised manuscript:
Table 3.Encoder cost. Uniform+EVS prunes after the vision encoder, so the encoder still processes every patch. IPB removes patches before the encoder.
IPB thus processes one quarter of the patches in the vision encoder at the same LLM token budget.
[W2]
When does the codec prior help, and when does it mislead?→ It helps event localization and hurts tasks that need continuous coverage.When codec priors help and when they do not.
IPB allocates more frames to segments with high coding cost and motion, and keeps sparser coverage elsewhere. This helps when the answer depends on locating specific events. It hurts when the answer requires following the whole clip without gaps.
Table 5(Qwen3-VL-4B, K=64) shows this split:The gains on event ordering and temporal perception suggest that codec-guided sampling can help when the evidence is concentrated in a few segments. The drops on Object Shuffle and counting tasks are consistent with a different requirement: the answer may depend on observing several swaps or repetitions across the clip. Coding cost and motion can highlight visual changes, but they do not tell us which changes matter to the question.
Revised manuscript:
Table 5.[W3]
Does selection remain effective under short-GOP encodings?→ Yes: a relaxed GOP rule removes the fallback (0/300, 75.2%).Encoding robustness.
The fallback in
Table 7comes from the minimum GOP duration in segment detection. IPB scores only GOPs of at least 2 s(Appendix A.2.1). This reflects typical encodings, whose GOPs are usually longer than 2 s and which insert extra I-frames at scene cuts(Sec. 3.1). Shorter GOPs often arise from abrupt cuts or flashes, where the I-frame dominates the packet size and the score does not reflect temporal change(Sec. 3.2).On the original Video-MME videos, this rule never causes fallback:
Fallback occurs only when the encoder fixes every GOP below 2 s, as with GOP=30. For such videos, we lower the threshold to min(2s, 0.5×median GOP). The corresponding row of
Table 7becomes:In the revision, we will adopt this rule in
Appendix A.2.1.Revised manuscript:
Table 7and Appendix A.2.1.