acceptodds
Under review as a conference paper at ICLR 2027

Look Back to Think Less: Rethinking Visual Token Pruning Efficiency under Multimodal Reasoning Cost

Abstract

Visual token pruning aims to reduce the inference cost of vision-language models (VLMs). Existing methods typically retain a fixed subset of visual tokens during prefill, reducing computation through input compression. However, as VLMs evolve toward stronger reasoners, we observe an overlooked shift in computational burden from prefill to decoding. Discarded visual evidence and fragmented context often trigger repeated verification and multiple answer attempts that substantially inflate generation length. Motivated by this observation, we propose VISTA, a training-free framework that jointly controls visual evidence retention, dynamic access, and reasoning termination. VISTA constructs a compact and diverse visual reserve during prefill, dynamically revisits relevant evidence under a fixed visual budget based on attention trajectories, and uses answer consistency with lightweight periodic probes to terminate reasoning once a stable candidate emerges. We extensively evaluate VISTA on three VLMs across seven multimodal benchmarks covering document understanding, visual reasoning, and medical VQA. Under strict visual budgets, VISTA reduces average generation length by 36.3% while improving average performance by 25.4% over the strongest baseline, highlighting the potential benefits of jointly optimizing visual compression and subsequent decoding for end-to-end inference efficiency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Token Pruning, Vision-Language Models, Multimodal Reasoning, Inference Efficiency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1zWAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes a training-free visual token pruning method for efficient VLM reasoning. It first constructs a cached pool of diverse visual tokens, then dynamically selects visible tokens from the pool based on current and historical attention during decoding. It also uses answer-consistency probes to terminate reasoning early when a stable answer is detected. Experiments show that it outperforms other pruning methods with shorter generation length on seven multimodal benchmarks.

Critical strengths
  1. The paper addresses an important practical problem in improving the inference efficiency of reasoning-capable VLMs. It highlights that computation saved by visual token pruning during prefill may be offset by increased decoding costs.
  2. The empirical evaluation is extensive, covering multiple VLM backbones, relevant pruning baselines, and different visual token budgets. The paper also reports generation length, inference latency, and TFLOPs, with component-level ablations.
Critical weaknesses
  1. The main comparisons don’t fully match the visual-token budgets. VISTA exposes c tokens per decoding step but caches a larger 2c reserve during prefill. Although the reported end-to-end speedups are encouraging, accuracy-latency comparisons across budgets could better establish the efficiency trade-off.
Questions for authors
  1. How frequently are 2c reserve tokens actually activated during decoding? Can these events be related to error recovery?
  2. Could the authors explain the substantial performance degradation compared to Vanilla (e.g., InfoVQA on Qwen3-VL-4B-Thinking and ChartQA on InternVL3.5-4B-Thinking) with representative failures?
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We sincerely appreciate your acknowledgment and constructive comments to improve our work. Below, we will address your concerns point by point. All additional results and analyses have been incorporated into the revised manuscript and highlighted in orange.

Q1: How frequently are 2c reserve tokens actually activated during decoding? Can these events be related to error recovery?

We appreciate your comment on reserve-token activation and error recovery. Here, Table R1 reports the rate of decoding steps with at least one activation and initially invisible reserve tokens activated at least once during decoding on Qwen3-VL-4B-Thinking under a 10% visual-token budget. We define an activation as a reserve token transitioning from the invisible state to the visible set. On average, activations occur in 85.7% of decoding steps, and 92.4% of initially hidden reserve tokens are activated at least once, indicating a continuous revisit of reserve tokens throughout decoding rather than remaining as an unused cache.

We further investigate the relationship between dynamic activation and error recovery using the Fixed variant, which does not activate invisible reserve tokens during decoding, under otherwise identical settings (consistent with Table 7 in revised manuscript). Among questions answered incorrectly by Fixed, an average of 28.1% are corrected by VISTA, which improves average accuracy from 55.0% to 61.9%. These results suggest that dynamic activation effectively recovers critical visual evidence and thereby recover errors. Following your suggestion, we have incorporated these results and analyses into the revised manuscript (Lines 501 and Table 8).

Table R1. Reserve-token activation and error recovery on Qwen3-VL-4B-Thinking with a 10% decoding visual-token budget. Fixed disables dynamic activation. Wrong → Right measures corrected answers across all questions, while Fixed errors recovered measures the recovery rate among Fixed's errors.

DatasetDecoding steps with activation (%)Hidden reserve tokens activated at least once (%)Fixed Acc. (%)VISTA Acc. (%)Wrong→RightFixed errors recovered (%)
ChartQA87.388.761.271.313.835.5
InfoVQA79.297.548.856.714.027.3
MathVista86.195.554.058.610.322.4
MMMU90.287.756.160.911.927.2
Avg85.792.455.061.912.528.1

Q2: Could the authors explain the substantial performance degradation compared to Vanilla (e.g., InfoVQA on Qwen3-VL-4B-Thinking and ChartQA on InternVL3.5-4B-Thinking) with representative failures?

We appreciate your suggestion on representative failure cases. For InfoVQA and ChartQA, questions typically require identifying specific labels in small regions of an image, making them highly dependent on critical visual evidence (highlighted in red in Figure S4). Existing pruning methods select visual tokens only once before generation. If critical regions are omitted, they cannot be recovered during subsequent decoding. Moreover, unlike mathematical or other reasoning tasks, these questions are difficult to to be inferred directly from text alone. Consequently, models could repeatedly reassess candidate answers, ultimately leading to reasoning inflation and substantial accuracy degradation. Thus, the additional decoding overhead could even far exceed the computational savings achieved by pruning during prefill, resulting in significantly increased overall computational costs. Following your suggestion, we have added representative failure cases in Line 1040 of the revised manuscript to further illustrate this phenomenon.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W1: Although the reported end-to-end speedups are encouraging, accuracy-latency comparisons across budgets could better establish the efficiency trade-off.

We appreciate your suggestion to provide accuracy-latency comparisons across budgets. We summarize the results across all seven benchmarks in Table R2. We observe that for VLMs with stronger reasoning capabilities, under aggressive visual-token pruning, the overall computational bottleneck shifts from prefill to decoding. The additional decoding overhead can outweigh the computational savings from prefill, leading to significantly higher total computational costs and latency, which remains consistent across different budgets. Furthermore, under same prefill computational cost (2.7 TFLOPs), VISTA at a 10% decoding visual-token budget still achieves higher accuracy than the best baseline at 20% (65.6% vs. 62.6%). Meanwhile, VISTA reduces the average generation length by 39.1% and end-to-end latency by 38.2%. Thus, suggesting that VISTA achieves a more favorable accuracy-latency trade-off by effectively mitigating reasoning inflation while maintaining high accuracy.

Table R2. Accuracy-latency comparisons across budgets. Average over all seven benchmarks

MethodBudgetAcc (%)LengthPrefill TFLOPsDecoding TFLOPsTotal TFLOPsTime (s/q)
DivPrune10%52.32,2301.624.025.630.4
ZOO-Prune10%50.02,5431.627.629.237.2
MMTok10%48.52,2611.624.225.832.5
VISTA10%65.61,2052.712.615.315.2
DivPrune20%62.61,9792.721.624.324.6
ZOO-Prune20%62.32,0772.722.525.226.6
MMTok20%59.62,0892.722.825.526.9
VISTA20%71.31,0735.111.316.413.4

Thanks for the detailed response. My major concerns are solved and I am happy to keep my acceptance recommendation.

Many thanks for your positive feedback and for the time and effort you have devoted to improving our work. We sincerely appreciate your constructive comments and valuable suggestions.

Reviewer_esb1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VISTA addresses the observation that visual token pruning can increase generation length and offset prefill savings in reasoning VLMs. The framework combines a scale-aware visual reserve, attention-trajectory-guided evidence revisiting, and answer-consistency-based early termination. Experiments on three models and seven datasets report improved accuracy and shorter generations compared with static pruning methods, with additional runtime measurements, component ablations, and multi-seed evaluations.

Critical strengths
  1. A valuable end-to-end view of pruning efficiency. The paper demonstrates that reducing visual input does not necessarily reduce total inference cost: longer decoding can offset prefill savings. Generation-length distributions, temperature experiments, and results under InternVL’s explicit thinking prompt strengthen this observation beyond isolated examples.

  2. Broad and substantial empirical improvements. At 10% visual retention on Qwen3-VL-4B, VISTA achieves 65.6 average accuracy versus 52.3 for the strongest baseline in Table 1. The paper also reports wall-clock improvements, integration with multiple pruning methods, and multi-seed results. Table 5 includes useful reserve-matched controls rather than relying exclusively on comparisons with static pruning.

Critical weaknesses
  1. The efficiency advantage of dynamic access is insufficiently isolated. VISTA prefills and caches approximately 2c visual tokens and scores attention over the reserve, while the nominal budget counts only the c visible tokens. Equal visible-token counts therefore do not imply equal computation or memory. Table 5’s All-visible control achieves nearly identical accuracy and generation length to VISTA (65.6 vs. 66.1; 2294 vs. 2271), but their runtime and memory are not compared. A termination-only control is also missing. These comparisons are needed to establish the benefit of dynamic revisiting beyond retaining more evidence and stopping earlier.

  2. The termination criterion is underspecified for open-ended answers. Equation 6 and Appendix A.2 use only the next-token distribution, although several benchmarks require multi-token numerical or textual answers. First-token agreement does not establish complete-answer consistency. Candidate construction, matching against multi-token answers, and final-answer production after periodic termination remain unclear. Because termination contributes substantially to the length reduction, these details and its premature-stopping error rate are important.

Questions for authors
  1. Matched-cost comparison: Please compare VISTA with All-visible using identical reserves and termination rules, reporting accuracy, latency, and peak memory. Are attention readout, selection, and probes included in the reported TFLOPs? Are invisible tokens physically excluded from attention computation or merely masked?

  2. Probe implementation: How are the candidate set A, answer matching, and final output defined for multi-token answers? What fraction of terminated examples would become correct if generation continued?

  3. Termination-only control: How much improvement comes from applying RT alone to static pruning and the full-token model? This would distinguish general early-stopping benefits from those enabled by evidence revisiting.

  4. Qwen prompting: Do the main conclusions hold with Qwen’s unmodified thinking template? The InternVL experiments are useful, but a within-Qwen comparison would clarify the effect of manually closing the thinking block.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

We sincerely appreciate your acknowledgment and constructive comments to improve our work. Below, we will address your concerns point by point. All additional results and analyses have been incorporated into the revised manuscript and highlighted in orange.

Q1: Please compare VISTA with All-visible using identical reserves and termination rules, reporting accuracy, latency, and peak memory. Are attention readout, selection, and probes included in the reported TFLOPs? Are invisible tokens physically excluded from attention computation or merely masked?

We appreciate your suggestion regarding the All-visible control. We first clarify that all variants in Table 7(in revised manuscript) adopt the same early termination rule as VISTA. Both methods incur comparable costs for caching the visual reserve. All related costs have already been included in all reported TFLOPs in Figure 3 and Figure S1. Here, Tables R1 and Table R2 further summarize the comparisons with and without early-stop probing under a 10% budget on Qwen3-VL-4B-Thinking, respectively. Under both matched settings, VISTA still achieves shorter average generation lengths with comparable accuracy. Specifically, with early-stop probing, VISTA reduces the length from 1,829 to 1,734 tokens and the end-to-end runtime by 7.7%. Moreover, probe frequency and computational cost remain nearly identical between the two variants. We also observe similar improvements under no early-stop settings.

A further breakdown of computational costs shows that VISTA further reduces visual-token attention computation by physically excluding invisible visual tokens from the main attention calculations during decoding. With and without early-stop probing, the average visual-token attention cost decreases from 174.2 to 77.2 GFLOPs per question and from 255.9 to 115.7 GFLOPs per question, respectively, particularly on visually intensive tasks like InfoVQA and long-reasoning tasks such as MMMU. Thus, suggesting that dynamic access enables the model to focus on key visual information more quickly while further reducing per-step computational costs without compromising accuracy.

Following your suggestion, we have incorporated these results into Line 1017 and Table S8 of the revised manuscript.

Table R1. Runtime and computational cost comparison between VISTA and All-visible with early-stop probing, averaged over two runs. All paired values are reported as VISTA / All-visible. Wall-clock time measures end-to-end execution time per question, while visual-token attention and reserve-attention hook denote the decoding attention cost and dynamic token reselection overhead, respectively.

DatasetWall-clock (s/q)Visual-token attention (GFLOPs/q)Peak KV (MB/seq)Reserve-attention hook (GFLOPs/q)Probes (calls/q)Probe cost (TFLOPs/q)Length (tokens/q)Accuracy (%)
MMMU42.3/43.9108.4/225.8602/6175.0/-8.1/8.30.60/0.613874/396160.7/60.1
ChartQA12.9/13.318.6/38.8143/1461.3/-2.2/2.20.16/0.16718/73271.1/71.2
MathVista21.1/24.243.8/98.8282/3072.5/-4.3/4.30.30/0.301687/184659.9/60.2
InfoVQA9.6/11.8137.9/333.3218/2358.4/-2.2/2.30.16/0.16657/77556.9/57.1
Avg21.5/23.377.2/174.2311/3264.3/-4.2/4.30.30/0.311734/182962.1/62.1

Table R2. Computational cost between VISTA and All-visible without early-stop probing, averaged over two runs. All paired values are reported as VISTA / All-visible.

DatasetVisual-token attention (GFLOPs/q)Peak KV (MB/seq)Reserve-attention hook (GFLOPs/q)Length (tokens/q)Accuracy (%)
MMMU144.0/300.9785/8096.69/-5145/527961.3/60.6
ChartQA29.2/65.0201/2232.14/-1124/125171.2/71.2
MathVista77.1/165.5466/4974.45/-2966/318359.8/61.1
InfoVQA212.7/492.2270/29512.97/-1013/117256.5/57.0
Avg115.7/255.9430/4566.56/-2562/272162.2/62.4
AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

Q2: How are the candidate set A, answer matching, and final output defined for multi-token answers? What fraction of terminated examples would become correct if generation continued?

We appreciate your comments on early termination. Here, we will first clarify the probe-based reasoning termination and analyze multi-token answers.

Reasoning termination: A\mathcal{A} contains the option letters for multiple-choice questions and the full tokenizer vocabulary for numeric and free-form questions. During reasoning, the answer prompt zz is periodically applied, returning an option letter for multiple-choice questions or the first token v^τ\hat v_\tau of a potential numeric or textual answer with its probability κτ\kappa_\tau. For matching rule, reasoning termination is triggered if the token returned by the probe matches an answer candidate explicitly stated, or if two consecutive periodic probes return the same token with probability of at least 0.99 each. The probe prompt is then removed from the cache, and the model finishes the current sentence. If it has already written a complete answer, we use it. Otherwise, we append the same final-answer prompt zz to the actual output and let the model generate the answer.

Multi-token answers: We compare early termination with continued reasoning for both single-token and multi-token answers on Qwen3-VL-4B-Thinking using two identical runs. Tables R3 and Table R4 summarize the resulting answer flips and accuracy differences. Overall, early termination has a negligible impact on accuracy across both answer types. Among early-terminated multi-token questions, continued reasoning produces both incorrect-to-correct and correct-to-incorrect flips at similar rates. Moreover, these flip rates are comparable to the variations observed among questions that never trigger early termination, suggesting that early termination does not introduce systematic errors to final answers. Table R4 further confirms it across individual datasets, without a consistent disadvantage for multi-token answers. Thus, the probe only determines when reasoning stops, while the final answer is still generated by the model itself.

Table R3. Single-token vs. multi-token answers on Qwen3-VL-4B-Thinking at a 10% budget on 6 public datasets. Each question is paired with an identical run without reasoning termination. "To correct / incorrect" counts questions whose correctness differs between the two runs.

Gold lennTerminated (%)Terminated: To correct/incorrect if continued (%)Never terminated: to correct/to incorrect (%) in 2 runsAccuracy change (pts)
14,74444.06.2/5.94.0/4.5-0.13
22,06930.33.8/4.83.6/2.9+0.30
3+7,65925.93.1/3.92.4/2.3+0.21
≥2 (all multi-token)9,72826.83.3/4.12.6/2.4+0.23
All14,47232.44.6/4.93.0/3.0+0.11

Table R4. Single-token vs. multi-token gold answers across datasets on Qwen3-VL-4B-Thinking at a 10% visual-token budget.

DatasetGold lennTerminated (%)Would become correct/incorrect (%)Never terminated change (%)
ChartQA137533.36.4/5.64.4/4.8
ChartQA≥22,12534.93.4/4.52.7/2.3
InfoVQA150338.65.7/4.13.9/3.6
InfoVQA≥22,29835.33.3/5.13.4/3.9
MathVista154265.55.1/4.54.8/3.7
MathVista≥245870.36.5/6.85.9/4.4
MMMU180558.57.0/8.98.1/6.0

Following your suggestions, we have further incorporated these results and discussions in Line 215, Line 398, Line 759 and Table 3 in revised manuscript.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

Q3: How much improvement comes from applying RT alone to static pruning and the full-token model? This would distinguish general early-stopping benefits from those enabled by evidence revisiting.

We appreciate your suggestion to include a termination-only control. Here, we further apply RT independently to both static pruning (DS) and the full-token Vanilla setting on Qwen3-VL-4B-Thinking, as summarized in Table R5. The results demonstrate that even under the full-token Vanilla setting, our proposed probe-based termination strategy effectively reduces the output length by approximately 29% while maintaining comparable accuracy. By comparing its performance with static pruning, we further observe that the benefits of RT could be influenced by the quality of visual information. Following your suggestion, we have incorporated these results into Table 6 and Table S9, with discussions in Line 485 of the revised manuscript.

Table R5. Effect of reasoning termination (RT) on Vanilla and static pruning (DS only). Each entry reports accuracy (%) / average generated length (tokens).

MethodChartQAInfoVQADocVQAScienceQARenalQAMathVistaMMMUAvg
Vanilla87.4/49977.7/51690.4/18393.2/85841.8/61375.6/2,50869.6/4,48876.5/1,381
Vanilla+RT87.6/40378.3/45790.4/13491.8/63240.3/52177.9/1,44366.3/3,27276.1/980
Static pruning (DS)51.9/1,90445.0/1,43880.6/41682.9/1,12537.2/59251.1/3,03956.5/5,29257.9/1,972
DS+RT51.5/1,19245.2/1,12380.0/31982.1/91837.0/55749.1/2,06356.9/4,05557.4/1,461

Q4: Do the main conclusions hold with Qwen’s unmodified thinking template? The InternVL experiments are useful, but a within-Qwen comparison would clarify the effect of manually closing the thinking block.

We appreciate your suggestion to evaluate Qwen using its unmodified thinking template. Here, we conducted additional comparative experiments using the official reasoning template of Qwen3-VL-4B-Thinking, with the results summarized in Tables R-6a and Table R-6b. We observe findings consistent with our previous experiments. With thinking mode enabled, reasoning inflation remains significant. The static pruning baselines still generate substantially longer outputs than Vanilla, while VISTA continues to achieve higher accuracy and shorter generation lengths than all baselines. Together with our previous findings, in models with stronger reasoning capabilities, static pruning causes the model to repeatedly inspect the remaining clues and reassess candidate answers (Figures 3B–3D), ultimately leading to unnecessary increases in generation length, overall computational cost, and runtime. Following your suggestion, we have further incorporated additional results into Table 2 and Table S1 in revised manuscript.

Table R-6a. Qwen3-VL-4B-Thinking with native thinking template at 10% budget. Each paired value reports accuracy (%) / average generated length.

MethodChartQAInfoVQADocVQAMathVistaMMMURenalQAAvg
Vanilla (100%)86.4/1,66478.9/1,31889.1/63676.8/3,51268.2/6,00745.6/3,58274.2/2,787
FastV33.8/3,48327.0/3,79343.1/2,87740.2/4,21654.0/6,23436.3/3,94939.1/4,092
SparseVLM29.2/3,70227.5/3,74441.4/2,80244.1/4,24153.9/6,27339.2/3,87639.2/4,106
DivPrune48.1/4,36536.9/4,59562.9/2,63148.9/4,61955.4/6,79440.8/3,67248.8/4,446
MMTok38.7/4,46435.1/4,29758.3/2,84843.1/4,69356.5/6,58338.9/3,86145.1/4,458
PruneSID34.8/4,69629.8/4,88850.3/3,22046.5/4,70554.8/6,80541.0/3,69342.9/4,668
ZOO-Prune44.1/4,90933.2/5,08754.1/3,34450.9/4,76056.8/6,99739.3/3,79446.4/4,815
VISTA (ours)70.1/1,65257.9/1,46685.8/58360.1/2,51461.9/4,63442.6/2,12663.1/2,163

Table R-6b. Qwen3-VL-4B-Thinking with native thinking template at 20% budget. Each paired value reports accuracy (%) / average generated length

MethodChartQAInfoVQADocVQAMathVistaMMMURenalQAAvg
Vanilla (100%)86.4/1,66478.9/1,31889.1/63676.8/3,51268.2/6,00745.6/3,58274.2/2,787
FastV56.4/3,82643.6/3,33869.5/1,77349.2/4,48058.8/6,56440.6/3,76153.0/3,957
SparseVLM57.6/3,81843.1/3,12568.4/1,75752.7/4,38958.4/6,62242.0/3,92753.7/3,940
DivPrune67.0/3,35251.5/3,39177.3/1,74358.9/4,48559.0/6,77741.7/3,61559.2/3,894
MMTok61.2/3,62050.7/3,42176.1/1,76250.8/4,62558.3/6,90242.4/3,99356.6/4,054
PruneSID58.9/4,25447.9/3,93473.5/2,08457.5/4,68760.9/6,91441.2/3,80756.6/4,280
ZOO-Prune67.3/3,86250.6/3,91274.3/2,06459.9/4,60462.7/6,73142.1/3,75159.5/4,154
VISTA (ours)78.8/1,18569.0/1,10088.1/47667.2/2,30164.7/4,30044.4/2,16068.7/1,920
Reviewer_Dhm9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that visual token pruning can save prefill computation while making later reasoning longer. VISTA addresses this with a diverse visual reserve, attention-guided access to that reserve during decoding, and answer-consistency probes for early stopping. Experiments cover three VLMs and seven datasets. On Qwen3-VL-4B at 10% visible tokens, the reported average score rises from 52.3 for the strongest baseline to 65.6, while generation length falls by 36.3%.

Critical strengths
  1. The paper makes a useful practical point: fewer visual tokens do not necessarily mean cheaper inference. Table S1 supports the argument with measured runtime, rather than generation length alone; VISTA averages 18.3 seconds per question versus 29.8 for DivPrune on the reported five-dataset evaluation.

  2. The evaluation does a good job separating the components. Tables 4 and 5 test evidence revisiting and stopping, including a same-reserve All-visible control, while Table S8 reports three-seed results. The improvements are also tested on InternVL with its official thinking prompt, which helps support the broader finding.

Critical weaknesses
  1. The stopping rule is underspecified for multi-token answers. Equation (6) and Appendix A.2 use only the next-token distribution, but DocVQA, InfoVQA, and numeric questions can require longer answers. Agreement on the first token need not imply agreement on the complete answer. This matters because early stopping accounts for a substantial part of the efficiency gain.

  2. The visible-token budget is not the full resource budget. VISTA prefills and caches roughly 2c visual tokens and reads attention over the reserve. Table S1 is useful, but Table 5 gives no corresponding runtime or memory comparison with All-visible, which reaches nearly the same accuracy. The benefit of dynamic access over simply exposing that reserve is therefore less clear.

Questions for authors
  1. Clarification: How are complete numeric and text answers compared with the one-token probe? Please specify the candidate set A, the matching rule, and how the final answer is completed after stopping. Can different answers sharing their first token cause premature termination?

  2. Assessment-relevant comparison: For the Table 5 All-visible control, what are the wall-clock time and peak KV-cache memory under the same stopping rule? Please include reserve-attention and probe overhead. This would clarify the practical gain from dynamic access itself.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We sincerely appreciate your acknowledgment and constructive comments to improve our work. Below, we will address your concerns point by point. All additional results and analyses have been incorporated into the revised manuscript and highlighted in orange.

Q1: How are complete numeric and text answers compared with the one-token probe? Please specify the candidate set A, the matching rule, and how the final answer is completed after stopping. Can different answers sharing their first token cause premature termination?

We appreciate your comments on early termination. Here, we will first clarify the probe-based reasoning termination and analyze multi-token answers.

Reasoning termination: We first want to clarify that the probe only checks the model's current answer tendency and serves only as a stopping signal. The final answer is always produced by the model itself. In Eq. (6), A\mathcal{A} contains the option letters for multiple-choice questions and the full tokenizer vocabulary for numeric and free-form questions. During reasoning, the answer prompt zz is periodically applied, returning an option letter for multiple-choice questions or the first token v^τ\hat v_\tau of a potential numeric or textual answer with its probability κτ\kappa_\tau. For matching rule, reasoning termination is triggered if the token returned by the probe matches an answer candidate explicitly stated, or if two consecutive periodic probes return the same token with probability of at least 0.99 each. The probe prompt is then removed from the cache, and the model finishes the current sentence. If it has already written a complete answer, we use it. Otherwise, we append the same final-answer prompt zz to the actual output and let the model generate the answer.

Multi-token answers: We compare early termination with continued reasoning for both single-token and multi-token answers on Qwen3-VL-4B-Thinking using two identical runs. Tables R1 and Table R2 summarize the resulting answer flips and accuracy differences. Overall, early termination has a negligible impact on accuracy across both answer types. Among early-terminated multi-token questions, continued reasoning produces both incorrect-to-correct and correct-to-incorrect flips at similar rates. Moreover, these flip rates are comparable to the variations observed among questions that never trigger early termination, suggesting that early termination does not introduce systematic errors to final answers. Table R2 further confirms it across individual datasets, without a consistent disadvantage for multi-token answers. Thus, the probe only determines when reasoning stops, while the final answer is still generated by the model itself.

Table R1. Single-token vs. multi-token answers on Qwen3-VL-4B-Thinking at a 10% budget on 6 public datasets. Each question is paired with an identical run without reasoning termination. "To correct / incorrect" counts questions whose correctness differs between the two runs.

Gold lennTerminated (%)Terminated: To correct/incorrect if continued (%)Never terminated: to correct/to incorrect (%) in 2 runsAccuracy change (pts)
14,74444.06.2/5.94.0/4.5-0.13
22,06930.33.8/4.83.6/2.9+0.30
3+7,65925.93.1/3.92.4/2.3+0.21
≥2 (all multi-token)9,72826.83.3/4.12.6/2.4+0.23
All14,47232.44.6/4.93.0/3.0+0.11

Table R2. Single-token vs. multi-token gold answers across datasets on Qwen3-VL-4B-Thinking at a 10% visual-token budget.

DatasetGold lennTerminated (%)Would become correct/incorrect (%)Never terminated change (%)
ChartQA137533.36.4/5.64.4/4.8
ChartQA≥22,12534.93.4/4.52.7/2.3
InfoVQA150338.65.7/4.13.9/3.6
InfoVQA≥22,29835.33.3/5.13.4/3.9
MathVista154265.55.1/4.54.8/3.7
MathVista≥245870.36.5/6.85.9/4.4
MMMU180558.57.0/8.98.1/6.0

Regarding shared first-token answers, it’s an interesting and important case. We examined such cases and found only 20 questions among roughly 15.4k in total. Of these, 15 triggered early termination, with only 2 answers changing from correct to incorrect, which are broadly consistent with the overall answer-flip patterns, suggesting limited additional bias beyond the model's inherent answer variability.

Following your suggestions, we have further incorporated these results and discussions in Line 215, Line 398, Line 759 and Table 3 in revised manuscript.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Q2: Assessment-relevant comparison: For the Table 5 All-visible control, what are the wall-clock time and peak KV-cache memory under the same stopping rule? Please include reserve-attention and probe overhead.

We appreciate your suggestion regarding the All-visible control. We first clarify that all variants in Table 7 (in revised manuscript) adopt the same early termination rule as VISTA. Both methods incur comparable costs for prefilling and caching the visual reserve. Tables R3 and Table R4 further summarize the comparisons with and without early-stop probing under a 10% budget on Qwen3-VL-4B-Thinking, respectively. Under both matched settings, VISTA still achieves shorter average generation lengths with comparable accuracy. Specifically, with early-stop probing, VISTA reduces the length from 1,829 to 1,734 tokens and the end-to-end runtime by 7.7%. Moreover, probe frequency and computational cost remain nearly identical between the two variants. Similar improvements are also observed under no early-stop settings.

A further breakdown of computational costs shows that VISTA further reduces visual-token attention computation by physically excluding invisible visual tokens from the main attention calculations during decoding. With and without early-stop probing, the average visual-token attention cost decreases from 174.2 to 77.2 GFLOPs per question and from 255.9 to 115.7 GFLOPs per question, respectively, particularly on visually intensive tasks like InfoVQA and long-reasoning tasks such as MMMU. Thus, suggesting that dynamic visual-token access enables the model to focus on key visual information more quickly while further reducing per-step computational costs without compromising accuracy.

Following your suggestion, we have incorporated these results and discussion into Line 1017 and Table S8 of the revised manuscript.

Table R3. Runtime and computational cost comparison between VISTA and All-visible on Qwen3-VL-4B-Thinking with early-stop probing, averaged over two runs. All paired values are reported as VISTA / All-visible. Wall-clock time measures end-to-end execution time per question, while visual-token attention and reserve-attention hook denote the decoding attention cost and dynamic token reselection overhead, respectively.

DatasetWall-clock (s/q)Visual-token attention (GFLOPs/q)Peak KV (MB/seq)Reserve-attention hook (GFLOPs/q)Probes (calls/q)Probe cost (TFLOPs/q)Length (tokens/q)Accuracy (%)
MMMU42.3/43.9108.4/225.8602/6175.0/-8.1/8.30.60/0.613874/396160.7/60.1
ChartQA12.9/13.318.6/38.8143/1461.3/-2.2/2.20.16/0.16718/73271.1/71.2
MathVista21.1/24.243.8/98.8282/3072.5/-4.3/4.30.30/0.301687/184659.9/60.2
InfoVQA9.6/11.8137.9/333.3218/2358.4/-2.2/2.30.16/0.16657/77556.9/57.1
Avg21.5/23.377.2/174.2311/3264.3/-4.2/4.30.30/0.311734/182962.1/62.1

Table R4. Computational cost between VISTA and All-visible without early-stop probing, averaged over two runs. All paired values are reported as VISTA / All-visible.

DatasetVisual-token attention (GFLOPs/q)Peak KV (MB/seq)Reserve-attention hook (GFLOPs/q)Length (tokens/q)Accuracy (%)
MMMU144.0/300.9785/8096.69/-5145/527961.3/60.6
ChartQA29.2/65.0201/2232.14/-1124/125171.2/71.2
MathVista77.1/165.5466/4974.45/-2966/318359.8/61.1
InfoVQA212.7/492.2270/29512.97/-1013/117256.5/57.0
Avg115.7/255.9430/4566.56/-2562/272162.2/62.4

Thank you for clarifying the stopping rule. The paired runs address my main concern about multi-token answers. The reported accuracy difference is only 0.23 percentage points, although the small number of shared-prefix cases limits what we can conclude about that specific failure mode.

The matched All-visible comparison also addresses my efficiency concern. With the same stopping rule, VISTA reduces average runtime from 23.3 to 21.5 seconds per question and peak KV cache from 326 to 311 MB per sequence, with the same average accuracy. I maintain my weak acceptance recommendation.

Many thanks for your positive feedback and for the time and effort you have devoted to improving our work. We sincerely appreciate your constructive comments and valuable suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.