acceptodds
Under review as a conference paper at ICLR 2027

Codec-derived Information Priors for Video Token Sparsification in Vision Language Model Inference

Abstract

We propose a codec-guided video token sparsification framework for high-throughput large vision-language model serving. Existing methods either require full visual encoding for frame selection or depend on intermediate features and attentions for token pruning, placing additional model-side computation on the GPU critical path and increasing serving overhead under concurrent requests. To this end, we leverage compressed-domain information exposed by video codecs to jointly perform frame selection and token mask generation before visual encoding. Specifically, frame selection is guided by coding cost and motion magnitude extracted from the bitstream, while token masks are generated from spatial priors based on motion and texture complexity. By resolving visual input selection on the CPU before visual encoding, IPB decouples sparsification from downstream GPU inference and enables CPU–GPU overlap across requests. At high concurrency, IPB achieves 3.2 the aggregate throughput and 38.0 lower TTFT than a combined frame-selection and token-pruning baseline, while maintaining favorable accuracy–efficiency trade-offs across four video benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
codec-guided sparsification, video token pruning, frame selection, patch selection, vision-language models, compressed-domain video, motion vectors, efficient video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_joddOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes IPB, a training-free method for reducing the number of video frames and visual tokens processed by a vision-language model. It uses codec information, including packet sizes and motion vectors, to select frames, and combines motion and texture cues to select patches within those frames. These decisions are made on the CPU before GPU inference, allowing selection to overlap with inference across requests. The main objective is to reduce the computational cost of video understanding without substantially compromising accuracy.

Critical strengths

The paper addresses a practical issue: frame selection and token pruning can themselves become expensive under concurrent serving. Moving these decisions to the CPU is a sensible approach, particularly when GPU-based selection competes with the model for resources. The method requires no additional training or a separate learned selector. This makes it appealing as an inference optimization. The Qwen implementation also removes patches before visual encoding, which allows the approach to save computation in the vision encoder. The accuracy–compute results are encouraging. Across several benchmarks, IPB reduces computation while retaining competitive accuracy. The evaluation across multiple backbones and the component ablations provide useful evidence that the approach has practical potential.

Critical weaknesses

The paper does not demonstrate consistent performance gains over the baseline methods. The substantial serving speedups are measured against BOLT+EVS, which incurs considerable selection overhead. Compared with Uniform+EVS, throughput is nearly identical and TTFT is slightly worse in the highlighted configuration. The results therefore support the benefit of avoiding expensive selection more clearly than they establish an advantage for IPB over simpler alternatives. It remains unclear when the codec prior helps reliably. Frame-selection gains over uniform sampling are often small and vary across tasks. Motion and coding complexity do not necessarily reflect the information needed to answer a particular question, and the paper could better characterize the cases where these signals are useful or misleading. There is also some uncertainty about robustness to encoding settings. In one tested configuration, 299 of 300 videos fall back to uniform sampling. Although accuracy is preserved, this does not demonstrate that the proposed selection mechanism remains effective under that configuration.

Questions for authors

I do not have extra question for the author

3 comments
AuthorsOpenReview ↗

Response to Reviewer jodd

Response to Reviewer jodd

Overall Response: We sincerely thank the reviewer for the thoughtful review, and for recognizing that moving selection to the CPU is a sensible approach to a practical serving issue, that IPB requires no training or learned selector, and that its accuracy–compute results are encouraging. IPB is designed for concurrent serving: selection must finish on the CPU, without VLM features or attention, and codec signals are informative signals available under this constraint. Below, we show where IPB gains over Uniform+EVS, characterize when the codec prior helps, and explain the fallback under short GOPs. Changes in the revised manuscript are marked in blue.


[W1] Does IPB gain consistently over simpler baselines such as Uniform+EVS? → Yes: −32.8% TTFT, +36.0% throughput, and +2.1 pp accuracy.

The 38.0× TTFT gain is relative to BOLT+EVS. Against Uniform+EVS, IPB gains in three ways: accuracy at lower compute, serving latency and throughput on real requests, and encoder cost.

Accuracy at lower compute. At the same LLM token budget (rpr_p = 0.375, Table 2), IPB improves Video-MME accuracy over Uniform+EVS by 2.1 pp with 30.7% fewer total FLOPs.

Serving on real requests. We serve both methods on Qwen3-VL-4B, the backbone of Table 2, with real Video-MME videos of mixed length (rpr_p = 0.375, the same LLM visual-token budget).

CMethodMean TTFT (s)p95 TTFT (s)p99 TTFT (s)Output tok/s
64Uniform+EVS3.7510.8011.56581.5
64IPB2.527.197.83790.9
128Uniform+EVS15.4523.4625.49580.2
128IPB11.0916.5117.96788.3

At C = 64, IPB reduces mean TTFT by 32.8% and raises throughput by 36.0% over Uniform+EVS.

Revised manuscript: Table 3.

Encoder cost. Uniform+EVS prunes after the vision encoder, so the encoder still processes every patch. IPB removes patches before the encoder.

IPB thus processes one quarter of the patches in the vision encoder at the same LLM token budget.


[W2] When does the codec prior help, and when does it mislead? → It helps event localization and hurts tasks that need continuous coverage.

When codec priors help and when they do not.

IPB allocates more frames to segments with high coding cost and motion, and keeps sparser coverage elsewhere. This helps when the answer depends on locating specific events. It hurts when the answer requires following the whole clip without gaps. Table 5 (Qwen3-VL-4B, K=64) shows this split:

Task typeTaskUniformIPBΔ (pp)
Locating eventsEvent before/after Event62.867.0+4.3
Temporal Perception72.778.2+5.5
Temporal Reasoning44.647.5+2.8
Continuous coverageObject Shuffle40.035.5−4.5
Moving Count73.069.0−4.0
Action Count60.059.0−1.0

The gains on event ordering and temporal perception suggest that codec-guided sampling can help when the evidence is concentrated in a few segments. The drops on Object Shuffle and counting tasks are consistent with a different requirement: the answer may depend on observing several swaps or repetitions across the clip. Coding cost and motion can highlight visual changes, but they do not tell us which changes matter to the question.

Revised manuscript: Table 5.


[W3] Does selection remain effective under short-GOP encodings? → Yes: a relaxed GOP rule removes the fallback (0/300, 75.2%).

Encoding robustness.

The fallback in Table 7 comes from the minimum GOP duration in segment detection. IPB scores only GOPs of at least 2 s (Appendix A.2.1). This reflects typical encodings, whose GOPs are usually longer than 2 s and which insert extra I-frames at scene cuts (Sec. 3.1). Shorter GOPs often arise from abrupt cuts or flashes, where the I-frame dominates the packet size and the score does not reflect temporal change (Sec. 3.2).

On the original Video-MME videos, this rule never causes fallback:

Video-MME subsetVideosMedian GOP (s)Fallback
Short3003.10
Medium3003.50

Fallback occurs only when the encoder fixes every GOP below 2 s, as with GOP=30. For such videos, we lower the threshold to min⁡(2 s, 0.5×median GOP)\min(2\,\text{s},\ 0.5 \times \text{median GOP}). The corresponding row of Table 7 becomes:

EncodingCRFGOPUniformIPBΔFallback
H.264 (2 s rule)233073.673.60.0299/300
H.264 (relaxed rule)233073.675.2+1.60/300

In the revision, we will adopt this rule in Appendix A.2.1.

Revised manuscript: Table 7 and Appendix A.2.1.

Thank you for the detailed reply and additional experiments in the discussion. However my main concern about robustness against general video scenes remain. I would maintain the original score.

We thank the reviewer for the response and the time spent on our paper.

  • Task dependence. Codec priors help when evidence is concentrated in a few segments and can hurt tasks that need continuous coverage (Table 5). We acknowledge this as a limitation of IPB.
  • Robustness to encoding. In our experiments, fallback never occurred on the original Video-MME videos (0/600), and IPB never fell below uniform sampling across the encoding settings of Table 7.

We appreciate the reviewer's feedback, which helped clarify both the strengths and the limits of IPB.

Reviewer_yx7NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Information Priors from Bitstream (IPB) uses video-codec information to generate frame selections and spatial patch masks on the CPU. Temporal selection combines GOP coding cost and motion activity with density smoothing, segment detection, and budget allocation. Spatial selection combines motion vectors with Sobel texture from decoded images. The goal is to reduce selection work on the GPU critical path and overlap CPU selection with GPU inference across requests. The Qwen path supports pruning before the vision Transformer; the appendix separately describes gathering after spatial pooling for LLaVA-OV, so these implementations should be distinguished. Experiments include concurrent serving, accuracy–compute comparisons on four benchmarks, and frame-selection evaluations across backbones. In the Qwen3-VL-8B stress test that repeatedly uses the same 30-minute video, the highest-concurrency configuration achieves approximately 3.2× aggregate throughput and 38.0× lower TTFT relative to BOLT+EVS.

Critical strengths

Moving selection off the GPU critical path has practical deployment value. IPB requires neither an additional semantic selection model nor training a codec-specific encoder. Table 1 and the full concurrency sweeps measure throughput and TTFT directly rather than inferring speed from FLOPs. For Qwen3-VL-8B at retention 0.25 and concurrency 64, IPB reports 6.45 s TTFT and 89.6 tok/s, versus 245.07 s and 27.9 tok/s for BOLT+EVS. Its serving cost is also close to Uniform+EVS, indicating that adaptive input selection can avoid expensive model-side scoring in this configuration.

Combining inexpensive signals yields a competitive accuracy–compute trade-off. At retention 0.375 on Qwen3-VL-4B, Table 2 reports gains of approximately 2.1, 2.2, and 2.7 points over Uniform+EVS on Video-MME, LongVideoBench, and MVBench, respectively, with approximately 31% lower reported compute. EgoSchema-subset decreases by 0.4 points, making this a useful trade-off rather than uniform dominance. The single-signal comparisons in Table 5 also support combining coding cost with motion gating. Task-level results, encoding sweeps, and implementation listings add useful detail.

Critical weaknesses

The headline serving gains are not yet tightly connected to quality on realistic workloads under the same configuration. The Qwen serving test uses the same synthetic 30-minute video, fixed input/output lengths, and an 8B model, whereas the central accuracy–FLOPs table uses the 4B model. The current results therefore do not directly establish an accuracy–latency curve for heterogeneous real-video requests. Against Uniform+EVS, IPB actually has slightly higher TTFT at concurrency 64 and retention 0.25 (6.45 versus 5.97 s), with essentially equal throughput; the 38× improvement should remain specific to BOLT+EVS. Appendix Table 11 additionally compares reducing whole frames with IPB against retaining all frames and merging tokens with AoT. These comparisons are useful, but broader deployment claims would benefit from jointly measuring quality and latency on the same model and real request set, with CPU resources and caching explicitly documented.

The encoding sweep does not fully separate useful codec priors from fallback protection. Under H.264 with GOP=30 in Table 6, 299 of 300 videos fall back to uniform sampling because of the two-second GOP eligibility threshold. Matching uniform sampling there mainly demonstrates fallback protection, rather than discriminative codec priors under that encoding structure. The authors acknowledge this limitation, but it still constrains the generalization evidence. Temporal selection also uses heuristic CV-dependent thresholds and duration rules; the existing one-at-a-time ablations do not replace reporting the active-selector rate, quality, and cost across encoding structures.

Questions for authors

Could IPB, Uniform+EVS, and BOLT+EVS be evaluated on the same backbone using real requests with varied video lengths, content, and encodings, reporting accuracy, mean and tail TTFT, and throughput? Please specify CPU thread counts, whether parsing/decoding is timed, reuse of same-video caches, and the measured benefit of CPU–GPU overlap.

Could fallback rates be reported for each encoding configuration, with results separated between active IPB selection and fallback videos? How do accuracy and cost change for short GOPs when the two-second eligibility threshold is reduced or removed?

For Table 2, what are the realized ViT and LLM token counts after two-frame mask OR pooling and complete spatial-merge-block alignment? At matched realized budgets, could uniform frames plus IPB patches, IPB frames plus EVS, and full IPB be compared to isolate temporal and spatial contributions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer yx7N

Response to Reviewer yx7N

Overall Response: We sincerely thank the reviewer for the careful and detailed review, which clearly distinguishes backbone, pruning stage, workload, and effective budget, and for recognizing the deployment value of moving selection off the GPU critical path and the competitive accuracy–compute trade-off of combining inexpensive codec signals. IPB is designed for concurrent serving: selection must finish on the CPU, without VLM features or attention, and codec signals are informative signals available under this constraint. Below, we report accuracy and serving on the same backbone as Table 2, separate active selection from fallback under short GOPs, and give realized token counts with a comparison of frame and patch contributions. Changes in the revised manuscript are marked in blue.


[W1/Q1] Same backbone, real requests: accuracy, tail TTFT, throughput, CPU and cache settings, and overlap? → On Qwen3-VL-4B, IPB has the lowest TTFT and highest throughput.

The 38.0× TTFT gain in Table 1 is relative to BOLT+EVS. Below, we compare all three methods on the backbone of Table 2.

Accuracy, mean and tail TTFT, throughput. We serve all three methods on Qwen3-VL-4B, the backbone of Table 2, with real Video-MME videos of mixed length (rpr_p = 0.375, the same LLM visual-token budget for all methods).

CMethodVideo-MME Acc. (%) (Table 2)Mean TTFT (s)p95 TTFT (s)p99 TTFT (s)Output tok/s
64Uniform+EVS59.853.7510.8011.56581.5
64BOLT+EVS63.3018.9224.9025.13295.8
64IPB62.002.527.197.83790.9
128Uniform+EVS59.8515.4523.4625.49580.2
128BOLT+EVS63.3033.7363.4566.80295.5
128IPB62.0011.0916.5117.96788.3

IPB achieves the lowest mean and tail TTFT and the highest throughput at both concurrency levels, with accuracy 2.15 pp above Uniform+EVS and 1.30 pp below BOLT+EVS.

Revised manuscript: Table 3.

CPU threads, timed stages, and caches. All methods run under the same configuration.

ItemSetting
CPU threads8 media-loading workers; 4 threads each for PyTorch and OpenCV
Parsing and decodingMeasured separately from TTFT
Same-video cachesOff

TTFT is measured from the submission of the prepared input to the model, so the comparison reflects GPU serving cost under the same token budget.

Measured benefit of CPU–GPU overlap. We run IPB with and without overlap on 64 requests in 4 fixed batches of 16 (Qwen3-VL-8B), with identical generated tokens and batch schedules.

OverlapCPU–GPU overlap (s)Mean TTFT (s)Output tok/s
OFF034.22137.2
ON8.7427.78174.9

Overlap reduces mean TTFT by 18.8% and raises throughput by 27.4%.

Revised manuscript: Sec. 4 and Table 14 (Appendix B).


[W2/Q2] Fallback rates per encoding, and what changes when the 2 s threshold is relaxed? → Fallback occurs only at GOP=30; a relaxed rule removes it (0/300, 75.2%).

The fallback in Table 7 comes from the minimum GOP duration in segment detection. IPB scores only GOPs of at least 2 s (Appendix A.2.1). This reflects typical encodings, whose GOPs are usually longer than 2 s and which insert extra I-frames at scene cuts (Sec. 3.1). Shorter GOPs often arise from abrupt cuts or flashes, where the I-frame dominates the packet size and the score does not reflect temporal change (Sec. 3.2).

On the original Video-MME videos, this rule never causes fallback:

Video-MME subsetVideosMedian GOP (s)Fallback
Short3003.10
Medium3003.50

Fallback occurs only when the encoder fixes every GOP below 2 s, as with GOP=30. For such videos, we lower the threshold to min⁡(2 s, 0.5×median GOP)\min(2\,\text{s},\ 0.5 \times \text{median GOP}). The corresponding row of Table 7 becomes:

EncodingCRFGOPUniformIPBΔFallback
H.264 (2 s rule)233073.673.60.0299/300
H.264 (relaxed rule)233073.675.2+1.60/300

No fallback occurs under the other encoding settings in Table 7. The relaxed rule does not change the selection cost.

In the revision, we will adopt this rule in Appendix A.2.1.

Revised manuscript: Table 7 and Appendix A.2.1.

AuthorsOpenReview ↗

Response to Reviewer yx7N - 2


[Q3] Realized token counts, and frame vs. patch contributions at matched budgets? → Realized retention equals nominal; IPB patches beat random by 2.54 pp.

Realized token counts. After OR-pooling the two frame masks of each tubelet and aligning to complete 2×2 merge blocks, IPB keeps qtq_t blocks in tubelet tt, which gives 4∑tqt4\sum_t q_t ViT tokens and ∑tqt\sum_t q_t LLM tokens.

Settings Qwen3-VL-4B, Video-MME, K = 64, rpr_p = 0.375.

Token count per videoRange over Video-MMEExample video
ViT tokens, all patches15,360–48,38446,592
ViT tokens after IPB5,760–18,14417,472
LLM visual tokens1,440–4,5364,368

The realized retention equals the nominal 0.375 in both ViT and LLM tokens, so pooling and alignment do not inflate the budget.

Temporal and spatial contributions. We compare frame and patch selection at rpr_p = 0.375 on Qwen3-VL-4B, Video-MME.

Frame selectionPatch pruningVideo-MME Acc. (%)Source
UniformRandom57.46Matched pre-ViT control
UniformEVS (post-ViT)59.85Table 2
UniformIPB60.00Matched pre-ViT control
IPBIPB62.00Table 2

On uniform frames, IPB patches outperform random patches by 2.54 pp and match post-ViT EVS; adding IPB frame selection reaches 62.00%.

Revised manuscript: Table 8.

Reviewer_NMEnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies training-free visual token pruning for efficient VideoLLMs. It proposes Information Priors from Bitstream (IPB), a codec-guided video token sparsification framework that jointly performs temporal frame selection and spatial patch masking. Specifically, IPB leverages codec-derived coding cost and motion activity for frame selection, and motion and texture cues for patch masking. By performing visual input selection on the CPU before visual encoding, IPB avoids additional GPU-side selection overhead and enables CPU–GPU overlap under concurrent requests. Experiments show 3.2× higher aggregate throughput and 38.0× lower TTFT, while maintaining competitive accuracy across 4 benchmarks.

Critical strengths
  1. The paper addresses a practical bottleneck in high-concurrency VideoLLM serving. It moves visual input selection outside the GPU critical path to avoid additional model-side computation.

  2. The proposed framework jointly performs temporal frame selection and spatial patch masking before visual encoding. It is training-free and does not require pretrained selection models.

  3. The experiments cover multiple VLM backbones and benchmarks. The paper reports accuracy–compute trade-offs, as well as TTFT and aggregate output-token throughput across different concurrency levels and retention ratios.

Critical weaknesses
  1. The novelty of the proposed framework is not sufficiently clear compared to existing codec-guided methods. MeToM[1] already leverages GOP packet sizes and codec residuals for training-free visual token compression. CodecSight[2] also uses codec metadata for patch pruning before visual encoding. The paper should better clarify its differences from these methods and include relevant comparisons.

  2. The reported serving improvements mainly come from comparisons with GPU-intensive baselines. In Table 1, IPB achieves nearly identical throughput and slightly higher TTFT than Uniform+EVS across most settings. For example, at C=64 and rp=0.25, IPB achieves 89.6 tok/s and 6.45 s TTFT, compared to 89.8 tok/s and 5.97 s for Uniform+EVS. This makes it unclear whether the proposed codec-guided selection provides additional serving benefits beyond avoiding GPU-side selection overhead.

  3. The serving evaluation is limited to repeated requests using the same synthetic 30-minute video. It does not evaluate heterogeneous video inputs or varying request arrival patterns. Moreover, a detailed breakdown of CPU preprocessing cost and CPU–GPU overlap is missing. These experiments are important to support the claimed benefits under realistic concurrent serving workloads.

  4. The ablation studies mainly focus on frame-selection components and hyperparameters. The contributions of motion and texture cues in patch masking are not separately evaluated. Comparisons with motion-only, texture-only, and random patch selection are missing. The paper also lacks a joint ablation of temporal frame selection and spatial patch masking, making it difficult to assess the contribution of each component.

  5. The robustness of codec-derived priors requires further investigation. Table 6 shows that 299 of 300 videos fall back to uniform sampling under H.264 with GOP=30, indicating sensitivity to encoding configurations. Moreover, the query-agnostic selection can miss query-relevant visual evidence. Table 12 shows noticeable accuracy drops on some tasks, such as Object Shuffle and Action Count. More analysis is needed to understand when codec-derived priors are reliable for video understanding.

  6. The comparisons with existing video token compression methods are limited. The authors justify BOLT+EVS as a composed baseline due to the lack of standardized joint frame-selection and token-pruning baselines. However, several relevant training-free methods, such as FastVID[3], MeToM[1], Dycoke[4], and Holitom[5], are not included. More comparisons under matched visual-token budgets and serving settings are needed to establish the effectiveness of the proposed framework.

[1] Wu, Zhuojie, et al. "MeToM: Metadata-Guided Token Merging for Efficient Video LLMs." CVPR 2026.

[2] Zou, Yulin, et al. "CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference." arXiv 2026.

[3] Shen, Leqi, et al. "Fastvid: Dynamic density pruning for fast video large language models." NeurIPS 2025.

[4] Tao, Keda, et al. "Dycoke: Dynamic compression of tokens for fast video large language models." CVPR 2025.

[5] Shao, Kele, et al. "Holitom: Holistic token merging for fast video large language models." NeurIPS 2025.

Questions for authors

Plz refer to the weaknesses.

10 comments
AuthorsOpenReview ↗

Response to Reviewer NMEn

Response to Reviewer NMEn

Overall Response: We sincerely thank the reviewer for the thorough and detailed review, and for recognizing that IPB addresses a practical bottleneck in high-concurrency VideoLLM serving and is evaluated across multiple backbones, benchmarks, and serving settings. Below, we clarify the differences from MeToM and CodecSight, quantify the gains over Uniform+EVS, evaluate heterogeneous inputs, random arrivals, CPU cost, and overlap, ablate motion and texture, characterize when codec priors help, and compare with FastVID, DyCoke, and HoliTom in a common serving runtime. Changes in the revised manuscript are marked in blue.


[W1] How does IPB differ from MeToM and CodecSight? → Selection runs on the CPU, without model features or model-internal state.

We agree that MeToM [A] uses GOP packet sizes and codec residuals, and that CodecSight [B] prunes patches before visual encoding using codec metadata. Using codec signals is therefore not what distinguishes IPB. The distinction is where selection runs, which follows from the serving objective behind IPB:

Goal. Serve concurrent video requests without adding selection work to the GPU.

Constraint. Selection must therefore finish on the CPU before inference. It cannot use VLM features or attention, the signals that GPU-side selectors rely on.

Why codec signals. Without model features, the CPU needs a signal that is both informative and cheap. Scoring pixels requires decoding every candidate frame: BOLT+EVS spends 50 s per request on CPU decoding and preprocessing. The encoder has already measured where prediction fails, and stores this as coding cost and motion vectors in the bitstream. IPB reuses these signals instead of recomputing them.

Evidence. At the same pre-encoder token budget, codec-guided patch selection outperforms random selection by 2.54 pp. Codec-guided frame selection matches the CLIP-based BOLT on Video-MME (59.8% vs. 59.9%, Table 4) with no GPU work.

With this objective in mind, we compare what each method retains, what signals it uses, and where it makes those decisions.

MeToM [A]

  • Merges low-residual patches before the ViT blocks.
  • Allocates tokens per frame by GOP packet size.
  • Further merges tokens by feature similarity and LLM attention.

CodecSight [B]

  • Decodes and preprocesses video on the GPU.
  • Uses motion to prune P-frame patches; keeps I-frames intact.
  • Uses GOP boundaries to guide KV refresh and reuse across windows.

IPB

  • Selects frames by coding cost and motion.
  • Masks patches by motion and texture.
  • Completes both on the CPU.

These design choices lead to the differences summarized in the table below.

MethodSelection completed on the CPUNo VLM features or attention for selectionDrops whole framesNo residual extractionNo model-internal state (features, attention, KV cache)
GPU-side selectors and pruners (BOLT+EVS, AoT, FastVID, DyCoke, HoliTom)✗✗△–✗
MeToM✗✗✗✗✗
CodecSight✗✓✗✓✗
IPB✓✓✓✓✓
△: BOLT selects frames; the pruning methods keep all sampled frames. –: not applicable.

Why these differences matter in practice.

  • GPU work per request. Two of MeToM's three stages decide which tokens to merge from visual features or LLM attention (Sec. 3.2.3–3.2.4 of [A]). This work therefore runs on the GPU for every request, and grows with concurrency. IPB makes all selection decisions before the GPU.

  • Encoder input. MeToM encodes every sampled frame. IPB removes unselected frames before the vision encoder.

  • Evaluation setting. MeToM reports single-request latency. IPB targets throughput and TTFT under concurrent requests.

In a common serving runtime at the same LLM token budget, IPB reaches 8.64 s mean TTFT at C=64, versus 21.8–22.1 s for DyCoke and HoliTom, two of MeToM's main baselines. Methods that reduce tokens after the vision encoder must hold the features of all frames on the GPU, which limits concurrency:

Video batch size BBFastVIDDyCokeHoliTomIPB
13.56 GiB3.56 GiB2.89 GiB0.26 GiB
27.11 GiB7.11 GiB5.77 GiB0.52 GiB
414.22 GiB14.22 GiB11.55 GiB1.04 GiB
828.44 GiB28.44 GiB23.10 GiB2.07 GiB
Largest BB without OOM101012≥ 64

Neither MeToM [A] nor CodecSight [B] has released code, so we did not reproduce them. We therefore claim no direct accuracy or speed ranking against either method.

Revised manuscript: Figure 1(c) and Sec. 1.

[A] Wu, Zhuojie, et al. "MeToM: Metadata-Guided Token Merging for Efficient Video LLMs." CVPR 2026.

[B] Zou, Yulin, et al. "CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference." arXiv 2026.

AuthorsOpenReview ↗

Response to Reviewer NMEn -2


[W2] Any serving benefit beyond avoiding GPU-side selection? → Yes: 75% fewer encoder patches at the same LLM token budget.

Figure 4 compares the vision encoder cost of IPB and EVS in peak activation memory, wall-clock time, and FLOPs. We quantify this below under a serving configuration.

Uniform+EVS and IPB give the LLM the same number of visual tokens, so their LLM cost is the same. They differ in where the reduction happens. EVS prunes after the vision encoder, so the encoder still processes every patch. IPB removes patches before the encoder.

Setting. Qwen3-VL-8B, 3 real videos, K=64K=64, rpr_p = 0.25, batch size 8, 3 runs per video.

MetricUniform+EVSIPBReduction
ViT input patches per video12,5443,13675.0%
LLM visual tokens per video7847840%
Vision matrix FLOPs per batch95.803 TFLOPs23.403 TFLOPs75.6%
Peak GPU memory allocated above baseline3.459 GiB1.381 GiB60.1%
Peak allocated GPU memory excluding weights and KV cache5.766 GiB3.688 GiB36.0%
Total peak allocated GPU memory30.094 GiB28.016 GiB6.9%

IPB thus matches Uniform+EVS in serving throughput (Table 1) while using one quarter of the encoder compute.


[W3] Heterogeneous inputs, arrival patterns, CPU cost, and overlap? → Faster completion on mixed videos, equal sustainable load, 17× less CPU work than BOLT+EVS, and +27.4% throughput from overlap.

Heterogeneous video inputs.

On mixed real videos, IPB completes requests faster and with higher throughput than Uniform+EVS.

Setting. Qwen3-VL-8B, 12 encoded clips from 3 real videos: 30 / 120 / 600 s, H.264 (GOP 1 / 2 / 6 s) and H.265 (GOP 2 s). C=64, 192 requests per method, 128 output tokens. Uniform+EVS and IPB use the same 1,152 LLM visual tokens per input.

MetricUniform+EVSIPBΔ
Mean end-to-end latency per request (s)17.3414.90−14.1%
First token → completion (s)11.725.36−54.3%
Mean TTFT (s)5.629.54+69.8%
Time to complete all 192 requests (s)53.0050.14−5.4%
Output throughput (tok/s)463.7490.2+5.7%

IPB admits requests later but completes them faster, so each request finishes sooner overall.

Compared with BOLT+EVS on the same workload, IPB reduces mean TTFT from 20.46 s to 9.54 s and raises throughput from 202.5 to 490.2 tok/s.


Request arrival patterns.

We send requests as a Poisson process at 6 average arrival rates. For each rate, we check whether at least 99% of requests meet the SLO (TTFT ≤ 5 s and end-to-end latency ≤ 30 s), and report the highest tested rate that passes.

Setting. LLaVA-OV-7B, 3 seeds, 8,220 requests in total. For each seed, both methods receive exactly the same arrival times.

MethodHighest tested arrival rate meeting the SLO
Uniform3 RPS
IPB3 RPS

IPB therefore sustains the same load as Uniform under random arrivals. CPU-side selection adds no penalty to the sustainable request rate.


CPU preprocessing cost.

We profile the CPU stages of each request under concurrent serving.

Setting. Qwen3-VL-8B, one H100, real 30-minute video, C=64, 64 requests per method. Both methods use 2,912 LLM visual tokens.

CPU stage (s per request)BOLT+EVSIPB
Frame selection< 0.011.55
Video decoding33.411.24
Preprocessing16.770.18
Total50.182.97

IPB spends 1.55 s on CPU selection and avoids 47 s of decoding and preprocessing, because it decodes only the frames it selects.


CPU–GPU overlap.

We compare IPB with and without overlap in an otherwise identical pipeline. To isolate the effect of overlap from scheduling, we run fixed batches instead of continuous batching.

Setting. Qwen3-VL-8B, one H100, 64 requests in 4 consecutive batches of 16. Generated tokens and batch schedules are identical between ON and OFF.

ON: Starts preparing the next batch while the GPU runs the current one.

OFF: Starts preparing the next batch after the current one finishes generation.

OverlapCPU–GPU overlap (s)Mean TTFT (s)Output tok/s
OFF034.22137.2
ON8.7427.78174.9

Overlap reduces mean TTFT by 18.8% and raises throughput by 27.4%.

Revised manuscript: Sec. 4 and Table 14 (Appendix B).

AuthorsOpenReview ↗

Response to Reviewer NMEn - 3


[W4] Random, motion-only, texture-only, and joint ablations? → Each cue beats random; combining them is best.

As requested, we ablate the motion and texture cues in patch selection and evaluate their combination with IPB frame selection.

Setting: Qwen3-VL-4B, Uniform 64 frames, patch retention 0.75, Video-MME 2,700 questions.

Frame selectionPatch selectionAccuracy (%)
UniformRandom (3-seed mean)62.59
UniformMotion-only62.89
UniformTexture-only62.96
UniformMotion + Texture63.19
IPBMotion + Texture (Full IPB)63.33

Combining motion and texture yields higher accuracy than either cue alone, and adding IPB frame selection gives the best result (63.33%).


[W5] Is the codec prior robust to encoding, and when does it miss evidence? → A relaxed GOP rule removes the fallback; priors help event localization and hurt continuous-coverage tasks.

Encoding robustness.

The fallback in Table 7 comes from the minimum GOP duration in segment detection. IPB scores only GOPs of at least 2 s (Appendix A.2.1). This reflects typical encodings, whose GOPs are usually longer than 2 s and which insert extra I-frames at scene cuts (Sec. 3.1). Shorter GOPs often arise from abrupt cuts or flashes, where the I-frame dominates the packet size and the score does not reflect temporal change (Sec. 3.2).

On the original Video-MME videos, this rule never causes fallback:

Video-MME subsetVideosMedian GOP (s)Fallback
Short3003.10
Medium3003.50

Fallback occurs only when the encoder fixes every GOP below 2 s, as with GOP=30. For such videos, we lower the threshold to min⁡(2 s, 0.5×median GOP)\min(2\,\text{s},\ 0.5 \times \text{median GOP}). The corresponding row of Table 7 becomes:

EncodingCRFGOPUniformIPBΔFallback
H.264 (2 s rule)233073.673.60.0299/300
H.264 (relaxed rule)233073.675.2+1.60/300

In the revision, we will adopt this rule in Appendix A.2.1.

Revised manuscript: Table 7 and Appendix A.2.1.

When codec priors help and when they do not.

IPB allocates more frames to segments with high coding cost and motion, and keeps sparser coverage elsewhere. This helps when the answer depends on locating specific events. It hurts when the answer requires following the whole clip without gaps. Table 5 (Qwen3-VL-4B, K=64) shows this split:

Task typeTaskUniformIPBΔ (pp)
Locating eventsEvent before/after Event62.867.0+4.3
Temporal Perception72.778.2+5.5
Temporal Reasoning44.647.5+2.8
Continuous coverageObject Shuffle40.035.5−4.5
Moving Count73.069.0−4.0
Action Count60.059.0−1.0

Object Shuffle asks for an object's final position after repeated swaps, and counting tasks require seeing every repetition. Both depend on temporal continuity rather than on a few informative segments. We will add this characterization to the Limitations section.

Revised manuscript: Table 5.


Reviewer_RnhSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents IPB, a training-free approach to reducing visual computation in video VLM inference. It uses GOP-level coding cost and motion activity to select frames, then combines motion vectors with Sobel-based texture scores to select patches. The selection is carried out on the CPU rather than through an additional learned GPU-side selector. The authors evaluate IPB across several video benchmarks and model backbones, and report accuracy, computational cost, and serving performance under different concurrency levels. The results show reduced vision-encoder costs and generally competitive accuracy, along with latency and throughput improvements over BOLT+EVS and AoT in the evaluated settings.

Critical strengths

1.The framework addresses a practical cost that token-reduction papers can overlook.IPB jointly selects frames and patches without an additional learned selection model. On the Qwen3-VL path, pruning takes place before visual encoding, and Figure 4 reports reductions in vision-encoder FLOPs, walltime, and activation memory. This is a meaningful benefit beyond reducing the number of tokens seen by the language model.

  1. The evaluation goes beyond a single accuracy or FLOPs comparison. Table 2 covers four benchmarks and several retention ratios; at 50% patch retention, IPB improves accuracy on three of the four benchmarks relative to Uniform+EVS while reducing total pipeline FLOPs by about 21%. Table 1 also evaluates concurrent serving. In addition to the large gains over BOLT+EVS, IPB improves latency and throughput over AoT on LLaVA-OV-7B at approximately matched token budgets.
Critical weaknesses

The added value over closely related codec-guided methods is not yet fully clear. MeToM (CVPR 2026) also uses GOP-level information and codec residuals to guide early video-token reduction. IPB is not the same method: it explicitly selects frames, uses motion and texture for patch masking, and is designed around CPU-side selection and concurrent serving. The paper cites related codec-guided work, but I would like a clearer account of why these particular differences matter in practice. A matched comparison with MeToM would be useful, though a more detailed methodological and systems comparison could also help.

  1. The experiments do not fully separate the benefits of the selection rules from the benefits of early pruning and the serving pipeline.** Table 5 and the appendix provide useful evidence for the temporal scoring choices, but there is no matched random pre-ViT patch-selection baseline or clear factorial test of IPB frame selection versus IPB patch selection. This matters for judging whether the proposed spatial importance scores help beyond simply pruning patches early. On the systems side, the 38.0× TTFT result is measured against BOLT+EVS, which carries substantial selection overhead. At 25% retention and 64 concurrent requests, IPB is close to Uniform+EVS. The AoT comparisons provide additional evidence for the serving gains, but the contribution of CPU–GPU overlap remains unclear.
Questions for authors

1.What practical limitation of MeToM does the explicit frame-selection and patch-masking design address? Is there a comparable accuracy–cost or serving result under a similar token budget? A more detailed explanation of the difference would also be helpful if a direct reproduction is not feasible.

  1. Have the authors compared the proposed patch scores with random pre-ViT pruning at the same patch budget and pruning stage? Are frame selection and patch selection evaluated independently and in combination? This would help distinguish the value of the codec-informed scores from the savings of early token reduction.

  2. Is CPU–GPU overlap measured against an otherwise identical IPB pipeline without overlap? A breakdown of preprocessing, vision encoding, queuing, and inference time would help explain what drives the concurrency gains and how the BOLT+EVS and AoT comparisons should be interpreted.

2 comments
AuthorsOpenReview ↗

Response to Reviewer RnhS

Response to Reviewer RnhS

Overall Response: We sincerely thank the reviewer for the careful and detailed review, and for recognizing that IPB addresses a practical cost that token-reduction work can overlook, reduces vision-encoder cost by pruning before visual encoding, and is evaluated beyond accuracy, including concurrent serving. IPB is designed for concurrent serving: selection must finish on the CPU, without VLM features or attention, and codec signals are informative signals available under this constraint. Below, we clarify how this differs from MeToM and CodecSight, show that IPB's patch scores outperform random pre-ViT pruning at the same budget (+2.54 pp), and measure CPU–GPU overlap in an otherwise identical pipeline together with a per-stage breakdown against BOLT+EVS and AoT. Changes in the revised manuscript are marked in blue.


[W1/Q1] What practical limitation of MeToM does IPB address? → Selection finishes on the CPU, without model features or GPU work.

We agree that prior work such as [A,B] also uses codec information to reduce video tokens. The distinction starts with the serving objective that motivates IPB:

Goal. Serve concurrent video requests without adding selection work to the GPU.

Constraint. Selection must therefore finish on the CPU before inference. It cannot use VLM features or attention, the signals that GPU-side selectors rely on.

Why codec signals. Without model features, the CPU needs a signal that is both informative and cheap. Scoring pixels requires decoding every candidate frame: BOLT+EVS spends 50 s per request on CPU decoding and preprocessing. The encoder has already measured where prediction fails, and stores this as coding cost and motion vectors in the bitstream. IPB reuses these signals instead of recomputing them.

Evidence. At the same pre-encoder token budget, codec-guided patch selection outperforms random selection by 2.54 pp. Codec-guided frame selection matches the CLIP-based BOLT on Video-MME (59.8% vs. 59.9%, Table 4) with no GPU work.

With this objective in mind, we compare what each method retains, what signals it uses, and where it makes those decisions.

MeToM [A]

  • Merges low-residual patches before the ViT blocks.
  • Allocates tokens per frame by GOP packet size.
  • Further merges tokens by feature similarity and LLM attention.

CodecSight [B]

  • Decodes and preprocesses video on the GPU.
  • Uses motion to prune P-frame patches; keeps I-frames intact.
  • Uses GOP boundaries to guide KV refresh and reuse across windows.

IPB

  • Selects frames by coding cost and motion.
  • Masks patches by motion and texture.
  • Completes both on the CPU.

These design choices lead to the differences summarized in the table below.

MethodSelection completed on the CPUNo VLM features or attention for selectionDrops whole framesNo residual extractionNo model-internal state (features, attention, KV cache)
GPU-side selectors and pruners (BOLT+EVS, AoT, FastVID, DyCoke, HoliTom)✗✗△–✗
MeToM✗✗✗✗✗
CodecSight✗✓✗✓✗
IPB✓✓✓✓✓
△: BOLT selects frames; the pruning methods keep all sampled frames. –: not applicable.

Why these differences matter in practice.

  • GPU work per request. Two of MeToM's three stages decide which tokens to merge from visual features or LLM attention (Sec. 3.2.3–3.2.4 of [A]). This work therefore runs on the GPU for every request, and grows with concurrency. IPB makes all selection decisions before the GPU.

  • Encoder input. MeToM encodes every sampled frame. IPB removes unselected frames before the vision encoder.

  • Evaluation setting. MeToM reports single-request latency. IPB targets throughput and TTFT under concurrent requests.

In a common serving runtime at the same LLM token budget, IPB reaches 8.64 s mean TTFT at C=64, versus 21.8–22.1 s for DyCoke and HoliTom, two of MeToM's main baselines. Methods that reduce tokens after the vision encoder must hold the features of all frames on the GPU, which limits concurrency:

Video batch size BBFastVIDDyCokeHoliTomIPB
13.56 GiB3.56 GiB2.89 GiB0.26 GiB
27.11 GiB7.11 GiB5.77 GiB0.52 GiB
414.22 GiB14.22 GiB11.55 GiB1.04 GiB
828.44 GiB28.44 GiB23.10 GiB2.07 GiB
Largest BB without OOM101012≥ 64

Neither MeToM [A] nor CodecSight [B] has released code, so we did not reproduce them. We therefore claim no direct accuracy or speed ranking against either method.

Revised manuscript: Figure 1(c) and Sec. 1.

[A] Wu, Zhuojie, et al. "MeToM: Metadata-Guided Token Merging for Efficient Video LLMs." CVPR 2026.

[B] Zou, Yulin, et al. "CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference." arXiv 2026.

AuthorsOpenReview ↗

Response to Reviewer RnhS - 2


[W2/Q2] Do codec scores help beyond early pruning, and do frame and patch selection each contribute? → Yes: +2.54 pp over random pre-ViT pruning at the same budget.

To isolate the effect of IPB’s spatial scores, we compare them with random pre-ViT pruning while fixing the uniformly sampled frames and the number of retained merge blocks in each temporal tubelet.

Setting. Qwen3-VL-4B, Video-MME, 64 frames, rpr_p = 0.375.

Spatial selection on the same uniform framesAccuracy (%)
Random seed 057.41
Random seed 156.63
Random seed 258.33
Random mean57.46
IPB patches60.00

At the same pre-ViT token budget, IPB patch selection improves accuracy from 57.46% to 60.00% (+2.54 percentage points) on the same uniformly sampled frames.

Frame selection alone Table 4 of the revised manuscript separately compares Uniform and IPB frame selection without patch pruning, using the same Qwen3-VL-4B backbone and 64-frame budget.

Frame selectionVideo-MME overall (%)Short (%)Medium (%)Long (%)
Uniform65.176.164.854.3
IPB65.577.864.054.7

The table below summarizes the results for frame selection, patch selection, and their combination.

Frame selectionPatch pruningPatch retention (rpr_p)Video-MME Acc. (%)Source
UniformRandom0.37557.46Matched pre-ViT control
UniformEVS (post-ViT)0.37559.85Table 2
UniformIPB0.37560.00Matched pre-ViT control
IPBIPB0.37562.00Table 2
UniformNone1.00065.1Table 4
IPBNone1.00065.5Table 4

Revised manuscript: Table 8.


[W2/Q3] Is overlap measured against an otherwise identical pipeline, and where does serving time go? → Yes: overlap cuts TTFT by 18.8%; BOLT+EVS and AoT lose time to selection and GPU queuing.

Overlap against an otherwise identical pipeline. Yes: we run IPB with and without overlap on 64 requests in 4 fixed batches of 16 (Qwen3-VL-8B), with identical generated tokens and batch schedules.

OverlapCPU–GPU overlap (s)Mean TTFT (s)Output tok/s
OFF034.22137.2
ON8.7427.78174.9

Overlap reduces mean TTFT by 18.8% and raises throughput by 27.4%.

Revised manuscript: Sec. 4 and Table 14 (Appendix B).

Breakdown against BOLT+EVS. We profile each request at C=64C = 64.

Setting Qwen3-VL-8B , Real 30-minute video, 2912 LLM visual tokens.

Stage (s per request)BOLT+EVSIPB
CPU: frame selection< 0.011.55
CPU: video decoding33.411.24
CPU: preprocessing16.770.18
GPU: frame selection (CLIP)1.090
GPU: vision encoder and EVS0.130.03
GPU: LLM prefill0.090.08
Queuing: input ready → first GPU kernel0.160.04
Request start → input ready237.8814.52
TTFT238.3114.68

BOLT+EVS must decode and preprocess every candidate frame before scoring it, and this CPU work and the waiting for it account for almost all of its TTFT; on the GPU, CLIP scoring is 83% of its work before the first token.

Breakdown against AoT. We profile each request at C=16C = 16.

Setting LLaVA-OV-7B ,48 requests per method, 1764 LLM visual tokens.

Stage (s per request)AoTIPB
CPU: metadata and frame selection0.312.66
CPU: video decoding1.510.46
CPU: preprocessing0.170.04
GPU: vision encoder0.190.02
GPU: token merging0.140
GPU: LLM prefill0.370.04
Queuing: input ready → first GPU kernel3.550.03
Request start → input ready5.974.64
TTFT9.914.73

AoT occupies the GPU 12× longer per request (0.70 s vs. 0.06 s), so requests queue before the GPU, while IPB requests start on the GPU almost immediately.


Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All reviewers and Area chairs

We sincerely thank all reviewers for their careful and constructive reviews, and for recognizing the practical value of moving visual input selection off the GPU critical path. Below, we summarize the main concerns shared across reviews and how we address them. Detailed results are in each individual response. Changes in the revised manuscript are marked in blue.

1. Novelty relative to codec-guided methods (RnhS, NMEn)

A novelty concern was raised relative to MeToM [A] and CodecSight [B], which also use codec signals for video token reduction. We agree that using codec signals alone does not distinguish IPB. The distinction is where selection runs, which follows from IPB's serving objective:

  • Goal: Serve concurrent video requests without adding selection work to the GPU.

  • Problem: GPU-side reduction does not scale.

    • Frame selectors score every candidate frame with a vision model, so their cost grows with video length (Figure 1b).
    • Most token pruners [C,D,E] hold the features or KV cache, so each request takes VRAM.
  • Constraint: Selection must finish on the CPU before inference, so it cannot use VLM features or attention.

  • Why codec signals: Under this constraint, the CPU needs a signal that is informative and cheap. The encoder has already measured where prediction fails and stores this in the bitstream; IPB reuses it instead of recomputing it.

Selection on the CPUNo VLM features or attentionDrops whole framesNo model-internal state
MeToM✗✗✗✗
CodecSight✗✓✗✗
IPB✓✓✓✓

Under this constraint, the codec signals are informative: IPB patch selection outperforms random pre-encoder pruning by 2.54 pp at the same budget, and IPB frame selection matches the CLIP-based BOLT on Video-MME (59.8% vs. 59.9%) with no GPU work. IPB adds 0.26 GiB per video and runs 64 concurrent videos without running out of memory.

Neither MeToM [A] nor CodecSight [B] has released code, so we did not reproduce them. We therefore claim no direct accuracy or speed ranking against either method.

Revised manuscript: Figure 1(c), Sec. 1, and Table 8.


2. Gains over Uniform+EVS and the scope of 38× (all reviewers)

All reviewers noted that the 38.0× TTFT gain is relative to BOLT+EVS, and that IPB was close to Uniform+EVS in Table 1.

Why BOLT+EVS. IPB selects both frames and patches before inference, so a matched baseline requires both. No standard method (with open source code) selects frames and prunes tokens jointly, so we compose the representative method for each stage. We additionally compare each stage separately: frame selection against BOLT and FOCUS (Table 4), and token reduction against AoT, FastVID, DyCoke, and HoliTom at the same LLM token budget.

Comparison with Uniform+EVS. On the backbone of Table 2 (Qwen3-VL-4B) with real Video-MME requests at the same LLM token budget:

MethodVideo-MME Acc. (%) (Table 2)Mean TTFT (s), C=64Output tok/s, C=64
Uniform+EVS59.853.75581.5
BOLT+EVS63.3018.92295.8
IPB62.002.52790.9

IPB reduces mean TTFT by 32.8% and raises throughput by 36.0% over Uniform+EVS, with 2.15 pp higher accuracy. Because IPB prunes before the vision encoder, it processes one quarter of the patches at the same LLM token budget.

Revised manuscript: Table 3.


[A] Wu, Zhuojie, et al. "MeToM: Metadata-Guided Token Merging for Efficient Video LLMs." CVPR 2026.

[B] Zou, Yulin, et al. "CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference." arXiv 2026.

[C] Shen, Leqi, et al. "Fastvid: Dynamic density pruning for fast video large language models." NeurIPS 2025.

[D] Tao, Keda, et al. "Dycoke: Dynamic compression of tokens for fast video large language models." CVPR 2025.

[E] Shao, Kele, et al. "Holitom: Holistic token merging for fast video large language models." NeurIPS 2025.

AuthorsOpenReview ↗

General Response

3. Where the serving gains come from (RnhS, NMEn, yx7N)

Reviewers asked to separate the effect of CPU–GPU overlap and to break down serving time.

  • Overlap: In an otherwise identical IPB pipeline, overlap reduces mean TTFT by 18.8% and raises throughput by 27.4%.

  • BOLT+EVS: Model-based selection must decode and preprocess every candidate frame: 50 s of CPU work per request versus 3 s for IPB. On the GPU, CLIP scoring accounts for 83% of its work before the first token.

  • AoT: AoT encodes all frames and merges tokens on the GPU, occupying the GPU 12× longer per request, so requests queue before the GPU.

Revised manuscript: Sec. 4 and Table 14 (Appendix B).


4. Robustness to encoding settings (NMEn, yx7N, jodd)

Reviewers noted that 299 of 300 videos fall back to uniform sampling with H.264 GOP=30. This fallback comes from a heuristic 2 s minimum GOP rule. On the original Video-MME videos, the rule never causes fallback (0/600). For encodings that fix every GOP below 2 s, a relaxed threshold removes the fallback (299/300 → 0/300) and raises accuracy from 73.6% to 75.2%.

Revised manuscript: Table 7 and Appendix A.2.1.


5. When codec priors help (NMEn, jodd)

Codec priors help on tasks that require locating specific events (e.g., Temporal Perception +5.5 pp) and hurt on tasks that require following the whole clip without gaps (e.g., Object Shuffle −4.5 pp). We characterize this split in the individual responses.

Manuscript: Table 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.