- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Route2K for sample-dependent visual computation in video question answering. It constructs a deterministic nested visual-token path, runs the frozen Video-MLLM once on an eight-token seed, and uses STOP-D, a learned low-rank transition predictor, to simulate changes in answer states as visual evidence is expanded. A Jensen-Shannon divergence threshold chooses a route. Samples that stop at the seed reuse its answer; other samples run an independent final decoder pass with a route-dependent early-layer width, followed by pruning to eight visual tokens. Thus, the deployed method uses at most two real decoder passes and does not reuse the seed KV cache.
On a repeatedly used, frozen 1,000-question NExT-QA validation subset, Route2K achieves 71.7% accuracy at 1,599 executed visual token-layers, versus 71.4% at 2,324 for one FastVID point and 69.4% at 1,633 for another. The paper also reports MVBench and TempCompass results, an execution ablation on 300 validation questions, and latency measurements on 100 questions. The proposal is interesting, and its accounting limitations are disclosed unusually clearly. However, the learned routing component is insufficiently specified and isolated experimentally, while the current comparisons do not establish a robust practical advantage.
Critical strengths
Relevant problem and plausible approach. Determining how much visual evidence a video-question pair needs is a useful complement to ranking tokens under a fixed budget. Predicting transitions from one real seed state could avoid the repeated decoder calls that would make naive progressive acquisition expensive.
A concrete execution design. The paper carefully distinguishes route depth, early-layer entry width, final active width, and actual decoder calls. The nested path is described in considerable detail, including temporal coverage, expansion, spatial suppression, and preservation of positional coordinates. Algorithm 1 makes the two-pass execution structure understandable.
Encouraging evidence for the execution design. On NExT-QA, route-conditioned execution improves over the narrow predecessor by 2.7 percentage points, with 49 corrections and 22 degradations. In the 300-question validation ablation, expanding the early context improves over narrow execution, and pruning too early substantially reduces accuracy. These results support the importance of allowing question interaction before aggressive late pruning.
Transparent reporting. The paper charges the seed pass, separates token-layer counts from latency, acknowledges different answer readouts on MVBench, describes the separate TempCompass calibration population, and states that NExT-QA was repeatedly used during development. Appendix A reports actual measurements, including the fact that Route2K remains slower than FastVID. These disclosures improve the credibility and interpretability of the evidence, even though they do not resolve the underlying limitations.
Critical weaknesses
W1. The central learned component is neither sufficiently specified nor directly validated.
Sections 3.3-3.4 describe STOP-D through a generic function D_theta and mention coefficient reconstruction, answer-distribution reconstruction, and routing supervision. The supplied PDF does not specify the training losses, weights, routing targets, predictor architecture, question-conditioned pooling, exact context features, or procedure used to fit the rank-eight basis. It also does not precisely identify the extracted answer/question states, give the predictor's training-set size and optimization settings, or report the numerical threshold, reference loss, and tolerance used in Eq. (2). A reproducibility statement that records exist does not replace these details or an accessible implementation.
More importantly, Table 3 freezes STOP-D and the routes while changing entry-width and pruning rules. It tests execution choices, not whether learned state-transition prediction is needed. Relevant controls include a direct budget predictor using the same available features, seed-confidence stopping, and permuted or randomized routes that preserve the budget distribution and approximate execution cost. Transition prediction errors, recursive error accumulation, false-stop rates, and sensitivity to basis rank would establish whether the proposed mechanism works as intended. The current results cannot separate intelligent sample allocation from the benefits of wider early processing followed by late pruning.
W2. The fixed-width ablation includes avoidable seed computation, weakening the matched-cost claim.
Table 3 compares Route2K at 1,606.4 executed token-layers with fixed M=80 at 1,600.0, describing the costs as nearly equal. However, the fixed-width method's final branch costs 1,376.0; the remaining 224.0 comes from the seed pass. An independent fixed-width final pass does not need this routing pass. Under that deployable implementation, Route2K uses approximately 16.7% more visual token-layers than the fixed-M=80 comparator, while gaining 2.33 accuracy points on validation. This is a different trade-off from an equal-cost improvement.
The authors explicitly acknowledge this issue for standalone fixed-width execution, and the main-text Always-Wide comparison appropriately uses the one-pass cost of 2,272. I therefore do not regard the tables as an arithmetic error. The missing experiment is a seed-free fixed-width frontier using the same token order and pruning schedule, with widths chosen to match Route2K's actual cost. For example, under the reported 16-layer wide/12-layer narrow schedule, a fixed entry width around 94-96 tokens would approximately match 1,606 token-layers without a seed pass. Its accuracy is unknown and must be measured, rather than inferred.
The key adaptive-versus-fixed validation result also rests on ten corrections and three degradations. Treating questions as independent, an exact two-sided McNemar calculation gives p approximately 0.092. This reviewer-derived calculation is only a diagnostic: questions can share videos, so a video-clustered analysis is preferable. It does not establish no effect, but it illustrates the uncertainty of the claimed 2.33-point advantage.
W3. Cross-benchmark evidence is not sufficiently controlled to establish generalization of the claimed trade-off.
On MVBench, Route2K uses option-logit scoring while the baselines use generated answers and a parser. Consequently, 51.575% versus 51.475% cannot isolate a method effect. The paper acknowledges this, but the abstract and conclusion still use cross-benchmark near-parity to support the overall allocation claim. Both methods should be evaluated under the same answer protocol.
The TempCompass Target100 comparison uses the same 3,533 evaluation questions and is more informative: Route2K scores 59.921% versus FastVID's 60.826% at lower token-layer cost. However, a point difference of 0.905 percentage points is not statistical evidence of non-inferiority within a one-point tolerance. Paired uncertainty is needed. The global-threshold and Target100 rows cannot demonstrate a calibration improvement because they change both threshold and evaluation population; the paper correctly states this. The label-free, video-disjoint calibration is not itself evidence of label leakage.
The main NExT-QA result comes from a 1,000-question subset explicitly acknowledged as repeatedly used during development. This does not establish contamination, but an untouched evaluation population is needed to assess generalization after freezing all choices. Report video-level split identifiers or manifests, an uncompressed accuracy reference, and confidence intervals that account for multiple questions per video. The current study also covers only Qwen2.5-VL-7B, one frame protocol, and multiple-choice answer scoring, limiting broader claims about video MLLMs.
W4. The stopping signal is not clearly connected to the answer-producing execution.
Eq. (9) stops when the predicted answer distribution changes little for the next scheduled increment. Even with a perfect one-step predictor, a small immediate change need not imply that later increments are unimportant. This is particularly relevant because Section 4.2 documents non-monotonic prediction/correctness behavior. That exploratory sweep does not show that STOP-D fails, but it motivates measuring false early stops and comparing cheap multi-step lookahead with the proposed first-small-change rule.
There is a second mismatch: STOP-D predicts states along budgets K_t, whereas continued samples execute M(K)=min(2K,136) and then apply late pruning. The PDF does not establish that the supervised transition states represent this final execution or that their local JS changes predict its accuracy/cost trade-off. Eq. (2) is a useful objective formulation, not a demonstrated guarantee. Please specify the actual reference, tolerance, and threshold-selection procedure, and show the resulting quality-cost curve and held-out constraint satisfaction for the deployed execution. A formal guarantee is not necessary if these empirical checks are convincing.
W5. The practical efficiency advantage remains unestablished.
Appendix A.2 does measure runtime, and it would be incorrect to say that the paper only reports token counts. Nevertheless, Route2K's model-side TTFT is 446.126 ms versus 389.929 ms for FastVID, approximately 14.4% slower; its post-ViT time is approximately 1.96 times FastVID's. Route2K is faster than the uncompressed model and the measured FlashVID point, but Table 4 does not report the corresponding accuracies on that timing subset or establish a quality-matched frontier. The 3.8% TTFT difference from FlashVID also needs uncertainty given the small subset and non-interleaved measurements.
Executed visual token-layers are a useful partial accounting measure, but they exclude the full effects of repeated text processing, routing, vision encoding, and hardware behavior. Routing itself takes 25.0 ms of the reported 81.3-ms post-ViT time, and all visual encoding is still performed up front. The next decisive experiment is a joint accuracy-latency comparison on the same examples, with seed-free fixed policies and baseline budget sweeps, reporting both calls and runtime variation. A token-layer improvement can still be a worthwhile research contribution; it simply does not currently establish a deployment advantage.
Questions for authors
What are STOP-D's exact architecture, losses, state definitions, basis-fitting procedure, training data, and optimization settings? What numerical reference loss, tolerance, and thresholds produced the reported policies?
Does STOP-D outperform direct budget prediction, seed-confidence stopping, and route permutations at comparable execution cost? How accurate are its recursively predicted transitions and stop decisions?
What is the accuracy-cost frontier of fixed-width policies after removing their unnecessary seed pass? Does adaptive routing outperform a seed-free fixed width around 94-96 under the Table 3 schedule?
Do the cross-benchmark conclusions hold under identical answer readouts, video-disjoint evaluation populations, and paired uncertainty estimates?
How well does stopping on local predicted JS control the quality of the actual expanded-and-pruned final pass, and is there an accuracy-matched end-to-end latency improvement?