acceptodds
Under review as a conference paper at ICLR 2027

Learning When to Look: Adaptive Visual Computation for Video MLLMs

Abstract

Video multimodal large language models (MLLMs) process long visual sequences, yet fixed token budgets ignore variation in the evidence needed by different video–question pairs. We introduce Route2K, which organizes visual tokens into nested prefixes and uses a low-rank predictor initialized from one real seed state to choose a route without repeated full-decoder decisions. The route sets early decoder width before question-conditioned late pruning. On a frozen 1,000-question NExT-QA validation subset, Route2K reaches 71.7% accuracy at 1,599 executed visual token-layers, versus 71.4% at 2,324 for a measured FastVID operating point; narrow execution reaches 69.0%. On MVBench and TempCompass, it executes fewer visual token-layers than selected FastVID points with reported accuracy within one percentage point. Validation ablations support route-conditioned entry widths. These results show the potential of sample-conditioned routing to improve the accuracy–computation balance of video question answering.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Video Understanding, Efficient Inference, Visual Token Compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YjdYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Route2K for sample-dependent visual computation in video question answering. It constructs a deterministic nested visual-token path, runs the frozen Video-MLLM once on an eight-token seed, and uses STOP-D, a learned low-rank transition predictor, to simulate changes in answer states as visual evidence is expanded. A Jensen-Shannon divergence threshold chooses a route. Samples that stop at the seed reuse its answer; other samples run an independent final decoder pass with a route-dependent early-layer width, followed by pruning to eight visual tokens. Thus, the deployed method uses at most two real decoder passes and does not reuse the seed KV cache.

On a repeatedly used, frozen 1,000-question NExT-QA validation subset, Route2K achieves 71.7% accuracy at 1,599 executed visual token-layers, versus 71.4% at 2,324 for one FastVID point and 69.4% at 1,633 for another. The paper also reports MVBench and TempCompass results, an execution ablation on 300 validation questions, and latency measurements on 100 questions. The proposal is interesting, and its accounting limitations are disclosed unusually clearly. However, the learned routing component is insufficiently specified and isolated experimentally, while the current comparisons do not establish a robust practical advantage.

Critical strengths

Relevant problem and plausible approach. Determining how much visual evidence a video-question pair needs is a useful complement to ranking tokens under a fixed budget. Predicting transitions from one real seed state could avoid the repeated decoder calls that would make naive progressive acquisition expensive.

A concrete execution design. The paper carefully distinguishes route depth, early-layer entry width, final active width, and actual decoder calls. The nested path is described in considerable detail, including temporal coverage, expansion, spatial suppression, and preservation of positional coordinates. Algorithm 1 makes the two-pass execution structure understandable.

Encouraging evidence for the execution design. On NExT-QA, route-conditioned execution improves over the narrow predecessor by 2.7 percentage points, with 49 corrections and 22 degradations. In the 300-question validation ablation, expanding the early context improves over narrow execution, and pruning too early substantially reduces accuracy. These results support the importance of allowing question interaction before aggressive late pruning.

Transparent reporting. The paper charges the seed pass, separates token-layer counts from latency, acknowledges different answer readouts on MVBench, describes the separate TempCompass calibration population, and states that NExT-QA was repeatedly used during development. Appendix A reports actual measurements, including the fact that Route2K remains slower than FastVID. These disclosures improve the credibility and interpretability of the evidence, even though they do not resolve the underlying limitations.

Critical weaknesses

W1. The central learned component is neither sufficiently specified nor directly validated.

Sections 3.3-3.4 describe STOP-D through a generic function D_theta and mention coefficient reconstruction, answer-distribution reconstruction, and routing supervision. The supplied PDF does not specify the training losses, weights, routing targets, predictor architecture, question-conditioned pooling, exact context features, or procedure used to fit the rank-eight basis. It also does not precisely identify the extracted answer/question states, give the predictor's training-set size and optimization settings, or report the numerical threshold, reference loss, and tolerance used in Eq. (2). A reproducibility statement that records exist does not replace these details or an accessible implementation.

More importantly, Table 3 freezes STOP-D and the routes while changing entry-width and pruning rules. It tests execution choices, not whether learned state-transition prediction is needed. Relevant controls include a direct budget predictor using the same available features, seed-confidence stopping, and permuted or randomized routes that preserve the budget distribution and approximate execution cost. Transition prediction errors, recursive error accumulation, false-stop rates, and sensitivity to basis rank would establish whether the proposed mechanism works as intended. The current results cannot separate intelligent sample allocation from the benefits of wider early processing followed by late pruning.

W2. The fixed-width ablation includes avoidable seed computation, weakening the matched-cost claim.

Table 3 compares Route2K at 1,606.4 executed token-layers with fixed M=80 at 1,600.0, describing the costs as nearly equal. However, the fixed-width method's final branch costs 1,376.0; the remaining 224.0 comes from the seed pass. An independent fixed-width final pass does not need this routing pass. Under that deployable implementation, Route2K uses approximately 16.7% more visual token-layers than the fixed-M=80 comparator, while gaining 2.33 accuracy points on validation. This is a different trade-off from an equal-cost improvement.

The authors explicitly acknowledge this issue for standalone fixed-width execution, and the main-text Always-Wide comparison appropriately uses the one-pass cost of 2,272. I therefore do not regard the tables as an arithmetic error. The missing experiment is a seed-free fixed-width frontier using the same token order and pruning schedule, with widths chosen to match Route2K's actual cost. For example, under the reported 16-layer wide/12-layer narrow schedule, a fixed entry width around 94-96 tokens would approximately match 1,606 token-layers without a seed pass. Its accuracy is unknown and must be measured, rather than inferred.

The key adaptive-versus-fixed validation result also rests on ten corrections and three degradations. Treating questions as independent, an exact two-sided McNemar calculation gives p approximately 0.092. This reviewer-derived calculation is only a diagnostic: questions can share videos, so a video-clustered analysis is preferable. It does not establish no effect, but it illustrates the uncertainty of the claimed 2.33-point advantage.

W3. Cross-benchmark evidence is not sufficiently controlled to establish generalization of the claimed trade-off.

On MVBench, Route2K uses option-logit scoring while the baselines use generated answers and a parser. Consequently, 51.575% versus 51.475% cannot isolate a method effect. The paper acknowledges this, but the abstract and conclusion still use cross-benchmark near-parity to support the overall allocation claim. Both methods should be evaluated under the same answer protocol.

The TempCompass Target100 comparison uses the same 3,533 evaluation questions and is more informative: Route2K scores 59.921% versus FastVID's 60.826% at lower token-layer cost. However, a point difference of 0.905 percentage points is not statistical evidence of non-inferiority within a one-point tolerance. Paired uncertainty is needed. The global-threshold and Target100 rows cannot demonstrate a calibration improvement because they change both threshold and evaluation population; the paper correctly states this. The label-free, video-disjoint calibration is not itself evidence of label leakage.

The main NExT-QA result comes from a 1,000-question subset explicitly acknowledged as repeatedly used during development. This does not establish contamination, but an untouched evaluation population is needed to assess generalization after freezing all choices. Report video-level split identifiers or manifests, an uncompressed accuracy reference, and confidence intervals that account for multiple questions per video. The current study also covers only Qwen2.5-VL-7B, one frame protocol, and multiple-choice answer scoring, limiting broader claims about video MLLMs.

W4. The stopping signal is not clearly connected to the answer-producing execution.

Eq. (9) stops when the predicted answer distribution changes little for the next scheduled increment. Even with a perfect one-step predictor, a small immediate change need not imply that later increments are unimportant. This is particularly relevant because Section 4.2 documents non-monotonic prediction/correctness behavior. That exploratory sweep does not show that STOP-D fails, but it motivates measuring false early stops and comparing cheap multi-step lookahead with the proposed first-small-change rule.

There is a second mismatch: STOP-D predicts states along budgets K_t, whereas continued samples execute M(K)=min(2K,136) and then apply late pruning. The PDF does not establish that the supervised transition states represent this final execution or that their local JS changes predict its accuracy/cost trade-off. Eq. (2) is a useful objective formulation, not a demonstrated guarantee. Please specify the actual reference, tolerance, and threshold-selection procedure, and show the resulting quality-cost curve and held-out constraint satisfaction for the deployed execution. A formal guarantee is not necessary if these empirical checks are convincing.

W5. The practical efficiency advantage remains unestablished.

Appendix A.2 does measure runtime, and it would be incorrect to say that the paper only reports token counts. Nevertheless, Route2K's model-side TTFT is 446.126 ms versus 389.929 ms for FastVID, approximately 14.4% slower; its post-ViT time is approximately 1.96 times FastVID's. Route2K is faster than the uncompressed model and the measured FlashVID point, but Table 4 does not report the corresponding accuracies on that timing subset or establish a quality-matched frontier. The 3.8% TTFT difference from FlashVID also needs uncertainty given the small subset and non-interleaved measurements.

Executed visual token-layers are a useful partial accounting measure, but they exclude the full effects of repeated text processing, routing, vision encoding, and hardware behavior. Routing itself takes 25.0 ms of the reported 81.3-ms post-ViT time, and all visual encoding is still performed up front. The next decisive experiment is a joint accuracy-latency comparison on the same examples, with seed-free fixed policies and baseline budget sweeps, reporting both calls and runtime variation. A token-layer improvement can still be a worthwhile research contribution; it simply does not currently establish a deployment advantage.

Questions for authors

What are STOP-D's exact architecture, losses, state definitions, basis-fitting procedure, training data, and optimization settings? What numerical reference loss, tolerance, and thresholds produced the reported policies?

Does STOP-D outperform direct budget prediction, seed-confidence stopping, and route permutations at comparable execution cost? How accurate are its recursively predicted transitions and stop decisions?

What is the accuracy-cost frontier of fixed-width policies after removing their unnecessary seed pass? Does adaptive routing outperform a seed-free fixed width around 94-96 under the Table 3 schedule?

Do the cross-benchmark conclusions hold under identical answer readouts, video-disjoint evaluation populations, and paired uncertainty estimates?

How well does stopping on local predicted JS control the quality of the actual expanded-and-pruned final pass, and is there an accuracy-matched end-to-end latency improvement?

Reviewer_eurAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Route2K for adaptive visual computation in video MLLMs. Based on the observation that different video–question pairs require different visual token budgets, this paper proposes a nested evidence path (which frames should be used) and a low-rank state-transition predictor to decide when to stop adding evidence. The selected route controls the visual token budget in early decoder layers before attention-based pruning. Experiments on NExT-QA, MVBench, and TempCompass show reduced visual token-layer computation with competitive accuracy at selected operating points.

Critical strengths

[+] The paper addresses a relevant limitation of fixed-budget token compression by allocating visual computation to individual video–question pairs.

[+] The low-rank transition predictor offers an interesting way to estimate the effect of additional evidence without repeated full-decoder passes.

Critical weaknesses

[-] The practical benefit over FlashVID remains unclear. Route2K reduces TTFT by only 3.8% (446.1 vs. 463.6 ms), while NExT-QA accuracy drops from 73.1% to 71.7%. Accuracy–latency curves across token budgets, particularly comparisons at matched accuracy, would help clarify whether Route2K offers a better trade-off than FlashVID.

[-] The main NExT-QA results use only a 1000-question validation subset, and the dataset was repeatedly used during method development. This limits confidence in generalization. Evaluation on the full validation set and an independent held-out test set would strengthen the conclusions.

[-] The presentation needs improvement. The abstract introduces several unexplained terms, including "real seed state" "early decoder width" and "narrow execution" The token-layer metric should also be clearly distinguished from measured latency, so readers can assess the practical efficiency gains.

[-] Answer-distribution stability does not necessarily indicate sufficient visual evidence. With only 32 uniformly sampled frames, key events may still missing, while the next token increment may contain little relevant information. For samples where the policy stops, evaluating further token expansion and denser frame sampling would help determine whether the stopping rule overlooks useful evidence.

Questions for authors

See the weaknesses above. The main concern is the reliance on a 1,000-question NExT-QA validation subset after repeated use of the dataset during development.

Reviewer_fGYqOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Route2K for sample-adaptive visual computation in video MLLMs. Starting from a real decoder state obtained with eight visual tokens, a low-rank transition predictor estimates answer-distribution changes along a nested token path. The selected route controls the early decoder width, followed by late pruning to eight visual tokens. Experiments evaluate accuracy and computation on three video-QA benchmarks.

Critical strengths
  1. Well-motivated problem. The paper distinguishes selecting informative tokens from determining how much visual evidence each question requires. Using a real seed state to guide inexpensive transition prediction provides an interesting approach to adaptive visual computation.
  2. Transparent computation accounting. The paper includes seed-pass costs and distinguishes decoder entry width, final retained width, visual token-layers, and measured latency. It also acknowledges important evaluation limitations.
Critical weaknesses
  1. Limited baseline coverage. External comparisons are restricted to FastVID and FlashVID, primarily at selected operating points. Representative alternatives such as VisionZip [1] and PruneVid [2] are missing. EarlyTom [3] would also help assess total inference efficiency, including vision encoding. Accuracy-latency curves under identical input and answer-readout protocols are needed to establish a convincing comparative advantage.
  2. Insufficient cross-model validation. All experiments use Qwen2.5-VL-7B-Instruct, despite the method's dependence on backbone-specific hidden states, a fitted low-rank basis, and pruning depth. Applicability to newer and larger models remains unestablished. A recent backbone such as Qwen3.6-27B, together with a recent model of similar size to the original backbone, would help assess generalization and distinguish architectural changes from scaling effects.
  3. Inadequate isolation of the routing mechanism. Table 3 varies execution settings while keeping routes fixed. Comparisons with direct budget prediction, seed-confidence routing, and shuffled routes are missing. The fixed-width control also includes an unnecessary seed pass; a standalone width around 94 or 95 would approximately match Route2K's reported visual-token-layer cost but is not evaluated. Recursive prediction error and premature stopping are also unquantified.
  4. Incomplete evaluation and reproducibility. MVBench uses different answer-readout protocols across methods, and the principal NExT-QA result comes from a development-exposed validation subset. Latency points are not quality matched, and Route2K is slower than FastVID at the measured settings. Missing predictor, training, and calibration specifications further limit reproducibility.

[1] Yang S, Chen Y, Tian Z, et al. VisionZip: Longer is Better but Not Necessary in Vision Language Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025: 19792-19802. Paper

[2] Huang X, Zhou H, Han K. PruneVid: Visual Token Pruning for Efficient Video Large Language Models. Findings of the Association for Computational Linguistics: ACL 2025, 2025: 19959-19973. Paper

[3] Wang H, Jin X, Lu L, et al. EarlyTom: Early Token Compression Completes Fast Video Understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026: 40559-40568. Paper

Questions for authors
  1. Can you broaden the external comparisons and report accuracy-latency curves under a common protocol, including an uncompressed reference and a genuinely compute-matched single-pass fixed-width baseline?
  2. Do the gains persist on a recent backbone such as Qwen3.6-27B and a recent model of comparable size to the original backbone, with predictors and pruning settings adapted under comparable training budgets?
  3. Does STOP-D outperform direct budget prediction, seed-confidence routing, and shuffled routes at matched cost? How accurately does it predict future states, and how often does it stop prematurely?
  4. Can you provide the missing implementation details, align the MVBench answer-readout protocol, and report quality-matched latency? How do accuracy and routing overhead change with longer visual sequences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.