acceptodds
Withdrawn from ICLR 2027

ContextTom: Training-Free Video VLM Acceleration with Early Spatial Context

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video vision-language models process thousands of visual tokens before producing an answer. Reducing these tokens early saves computation, but also determines which evidence reaches the language model. We introduce ContextTom, a training-free framework that separates the timing of temporal and spatial compression. Its central idea is to compress time before vision encoding while preserving spatial coverage through early multimodal processing. A fixed-budget temporal partition produces representative medoid anchors, supplemented with bounded local detail from neighboring observations. The anchors retain their complete spatial grids through a short decoder prefix, allowing question tokens to access the visual evidence and completing Qwen3-VL\textquotesingle s multi-level visual injections. ContextTom then selects spatial representatives using query attention and agreement across visual feature levels, and continues directly from their computed states. This schedule reduces vision computation and later decoder workload while reusing the prefix. On Qwen3-VL-8B, ContextTom achieves 76.22% accuracy on Video-MME short with a mean time to first token (TTFT) of 123.5 ms, yielding a speedup over full-input inference. It matches the full-input accuracy of 72.20% on EgoSchema with a speedup and achieves a speedup on LongVideoBench at 59.24% accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-language model, Training-free acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_v8iqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ContextTom, a training-free method for accelerating video vision-language model inference. It performs early temporal compression using lightweight visual descriptors, preserves complete spatial features through the early DeepStack prefix in Qwen3-VL, and subsequently applies query-guided spatial selection with multi-level visual similarity. The method reuses retained hidden states to reduce later decoder computation. Experiments are reported on Video-MME, EgoSchema, LongVideoBench, and MVBench across several model backbones.

Critical strengths
  1. A reasonable compression strategy. Combining early temporal reduction with delayed spatial selection is a meaningful design direction for reducing inference cost while preserving early multimodal context.

  2. Broad experimental coverage. The paper evaluates multiple video benchmarks and model backbones and includes component ablations and latency profiling.

Critical weaknesses
  1. Insufficient experimental comparability and reproducibility. Table 1 reports 76.22% accuracy and 123.5 ms TTFT on Video-MME Short, whereas the development experiment in Appendix D reports 76.89% accuracy and 472.52 ms TTFT. Although the paper acknowledges differences in execution paths and budgets, it does not adequately map these differences to the reported results. Moreover, the paper states that baseline budgets, dates, and evaluation sets may differ while still making strong cross-method comparisons. The reported evidence is therefore insufficient to fully verify the central efficiency claims.

  2. Insufficient validation of the technical contribution and an internal inconsistency. The component ablations are concentrated on Qwen3-VL-8B and Video-MME Short, leaving the independent contribution over existing compression methods insufficiently established. The adaptation of the DeepStack-oriented design to InternVL3.5 and LLaVA-OneVision is also unclear. Furthermore, the AI-use statement refers to a frame-wise score-calibration property and its proof, but I could not locate a corresponding explicit result in the main text or appendices. This discrepancy requires clarification.

Questions for authors
  1. What exactly is the frame-wise score-calibration property mentioned in the AI-use statement, and where are its formal definition and proof?

  2. What explains the differences in accuracy and TTFT between Table 1 and Appendix D? Please specify the evaluation subset, subtitle setting, execution backend, and timing protocol.

  3. Were all baselines evaluated with consistent model versions, input sampling, resolution, evaluation sets, and timing conventions? What specific differences are covered by the table notes?

  4. How is ContextTom implemented for InternVL3.5 and LLaVA-OneVision? Which components depend on Qwen3-VL-specific DeepStack behavior, and how is cross-backbone comparability maintained?

  5. Can the authors provide controlled comparisons under matched budgets and execution paths to isolate the contribution of each component? What are the method's limitations under rapid motion, occlusion, and abrupt scene changes?

Reviewer_dq7XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ContextTom, a training-free inference acceleration framework for video VLMs. It first compresses the temporal axis before vision encoding using fixed-budget segmentation and medoid anchors, with bounded local supplementation. It then keeps full spatial grids through a short decoder prefix aligned with Qwen3-VL’s DeepStack injections. After that, it prunes spatial tokens using query attention and multi-level visual similarity, and continues decoding from the retained prefix states. Empirically, the method reports strong TTFT reductions on Qwen3-VL and competitive results on several video benchmarks.

Critical strengths
  1. The problem framing is clear: compression timing matters when early multimodal layers inject visual features and when questions need spatial access before pruning.
  2. The empirical study is reasonably thorough, covering multiple benchmarks and backbones, with timing, component, and budget ablations.
  3. On Qwen3-VL, ContextTom often achieves the lowest TTFT among reported baselines, with favorable accuracy–latency trade-offs in some settings, including matching full-input EgoSchema accuracy.
Critical weaknesses
  1. Limited elegance and novelty. ContextTom is a long pipeline of hand-designed heuristics rather than a single clean compression principle: descriptor-based temporal segmentation, medoid anchors, residual fusion, a fixed full-spatial decoder prefix, query scoring, adjacent-pair pruning, multi-level similarity, coverage floors, and prefix-state continuation. Relative to prior training-free temporal reduction and early-decoder pruning, the main idea is a Qwen3-VL-oriented schedule, not a broadly new operator.
  2. The spatial stage may be over-credited. With mild default retention, most speedup likely comes from early temporal reduction. Ablations suggest temporal-only is already close, so the elaborate spatial module adds limited marginal value.
  3. TTFT may overstate practical gains. Preprocessing can dominate wall-clock time, so large TTFT speedups may not translate to end-to-end latency. End-to-end and throughput results under realistic serving would strengthen the claims.
Questions for authors
  1. Without DeepStack (or with a different injection depth), how should pp be chosen? Please provide a protocol that is not Qwen3-VL-specific, and ablate prefix-then-prune vs. pre-decoder spatial compression under that protocol on InternVL/LLaVA.
  2. Under matched token budgets, how much of the speedup/accuracy remains if the spatial stage is replaced by a simple baseline (e.g., attention top- kk or similarity merge) after the same temporal anchors?
  3. Please report end-to-end latency (including decode/preprocess) and throughput; also clarify the discrepancy between development and Table 1 short-subset accuracies, and confirm matched budgets for all baselines.
Reviewer_VGHLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ContextTom is a training-free framework for accelerating video VLMs by separating temporal and spatial compression. It selects temporal medoid anchors before vision-transformer processing, supplements them with bounded local features, and preserves complete spatial grids through an early decoder prefix. Spatial representatives are then selected using query importance and multi-level visual similarity, and their computed states continue through the remaining decoder layers. Experiments cover four video benchmarks and multiple model families, reporting accuracy and time to first token (TTFT).

Critical strengths
  1. The framework offers a coherent approach to reducing both vision-encoder and decoder computation. Temporal anchoring reduces the grids processed by vision-transformer blocks, while spatial selection shortens later decoder sequences. Preserving full spatial grids through DeepStack injections and continuing from computed prefix states provides a sensible architecture-aware implementation that avoids restarting decoder processing. The scoring, positional bookkeeping, and cache handling are clearly described.
  2. The evaluation demonstrates promising acceleration across multiple benchmarks and model families. Experiments cover four video benchmarks and Qwen3-VL, InternVL3.5, and LLaVA-OneVision. Reported results include a 2.77× model-side TTFT speedup on Video-MME short and matching full-input accuracy on EgoSchema with a 2.82× speedup. Component ablations further support the usefulness of query guidance and multi-level visual similarity.
Critical weaknesses
  1. The novelty and independent benefit of the proposed compression schedule are insufficiently established. EarlyTom already combines early temporal compression with spatial selection, while HoliTom and FrameFusion combine related temporal, similarity, and importance mechanisms. ContextTom’s main distinction is scheduling spatial selection after DeepStack injections, but the timing experiment yields only a 0.60-point accuracy improvement without separately controlling injection completion, scoring changes, and state reuse. Temporal-only processing already achieves 75.78% at 130.7 ms, versus 76.22% at 123.5 ms for the complete method. Potentially different evaluation sets, budgets, and execution optimizations further weaken attribution of the comparative gains.
  2. The selection rules rely on insufficiently validated assumptions about evidence preservation. Question-agnostic RGB descriptors can discard brief or subtle semantic events before query-aware selection becomes available, while same-cell supplementation without motion alignment can mix incompatible observations. Spatial similarity and average attention likewise do not establish that one token can replace another, particularly because the fixed removal budget has no absolute-similarity safeguard. Completing DeepStack injections preserves visual additions but does not ensure that evidence from removed cells has transferred sufficiently to retained states. The existing average-accuracy ablations do not adequately examine these risks on brief events, motion, or spatial-relation questions.
Questions for authors
  1. Can you isolate the effects of pruning depth, DeepStack completion, query scoring, and prefix-state reuse under otherwise matched conditions, and report paired uncertainty for the 0.60-point timing benefit?
  2. How does temporal anchoring perform on questions involving brief events or subtle object-state changes? Are there cases where same-cell supplementation introduces motion-misaligned content?
  3. How often does the fixed spatial budget remove pairs with low cross-level similarity? Would an absolute-similarity threshold improve evidence preservation, and how would it affect the achieved compression ratio?
  4. Can you provide accuracy–latency comparisons on identical evaluation examples with controlled budgets and equivalent execution optimizations? Please also explain the configuration and execution differences between the 472.52 ms development TTFT and the 123.5 ms main result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.