acceptodds
Under review as a conference paper at ICLR 2027

Visual Tokens Should Not Be Pruned Forever: Budgeted Visual Token Refresh for Multi-Turn LVLMs

Abstract

Large vision-language models (LVLMs) often retain only a subset of visual tokens to reduce inference costs. However, existing static pruning methods typically select tokens according to the initial question and reuse the same token set throughout subsequent dialogue turns. Consequently, when the focus of a conversation changes, visual information required by a new question may remain inaccessible even though it is present in the image. We formulate this limitation as the problem of adapting the active visual context to follow-up questions under a fixed token budget and propose Budgeted Visual Token Refresh (BVTR). BVTR initially activates 288 of 576 visual tokens and, upon receiving a follow-up question, replaces up to 48 tokens according to their question-conditioned importance. Because directly inserting KV states can mix representations computed under inconsistent visual contexts, BVTR-Dense reconstructs the preceding context using the refreshed token set and replays the answer tokens generated in the previous turn. BVTR-Sparse further reduces reconstruction costs by preserving the original token order and selectively updating only a subset of deep-layer KV states. On the frozen COCO-TS1000 evaluation with LLaVA-1.5-7B, BVTR-Dense improves accuracy over the fixed 288-token baseline by 1.4 percentage points, with a significant paired difference under McNemar’s test (). In a four-turn topic-switching evaluation, it increases the proportion of conversations in which all three follow-up questions are answered correctly from 68.0% to 75.5% (), while also yielding a significant improvement on an MME multi-turn adaptation. On a separate confirmatory evaluation of 1,000 images, BVTR-Sparse improves accuracy over the fixed baseline by 2.1 percentage points (). It reduces incremental peak GPU memory by 35.4% while achieving an observed accuracy only 0.1 percentage points below that of the full 576-token context, and reduces paired end-to-end latency by 15.9% relative to BVTR-Dense. These results demonstrate that adapting the visual context to shifts in conversational focus under a fixed token budget provides an effective trade-off between accuracy and efficiency for multi-turn LVLMs, distinguishing BVTR from static pruning approaches that treat visual-token removal as a permanent decision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal large language models, visual token pruning, multi-turn dialogue, visual context refresh, KV cache, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CdquOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BVTR replaces visual tokens according to each follow-up question while keeping the active token budget fixed. It updates the cache through dense reconstruction or sparse repair. SR1000 accuracy increases from 88.9% to 91.0%, and MT4 sequence success rises from 68.0% to 75.5%. Gains on VQAv2 and with strong initial selectors are limited.

Critical strengths
  1. Recovering visual evidence needed by later questions under a fixed token budget addresses a practical problem.
  2. Random and previous-question refresh controls, together with multi-turn and cross-model results, support the effectiveness of question-conditioned refresh.
Critical weaknesses
  1. Unclear benefit over simpler or stronger baselines. Global reselection achieves 90.9% versus 91.0% for BVTR. Refresh provides no gain with a strong initial selector. Ablations also show no advantage from selecting repair tokens by changes in cached keys and values.
  2. Incomplete efficiency evaluation. Sparse96 takes 2.06 times as long as Fixed. Full replays historical answers, although its unchanged visual context normally permits cache reuse. This choice needs justification. Incremental peak memory alone does not establish total memory requirements.
Questions for authors
  1. How are the datasets constructed and filtered? Does filtering depend on omissions or errors from a particular selector or model?
  2. With standard cache reuse for Full, what are each method’s latency and total peak GPU memory?
  3. On the full test set, how does BVTR compare with global reselection and updating only newly activated tokens in accuracy, latency, and memory?
  4. Please specify token scoring, replacement counts, and how historical question–answer caches are updated in turns three and four.
Reviewer_gwKeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Budgeted Visual Token Refresh (BVTR) for multi-turn vision–language conversations. BVTR maintains a fixed-size active visual-token set and replaces a limited number of tokens according to the current question. It evaluates dense cache reconstruction and a sparse variant that selectively updates deeper-layer states.

Critical strengths
  1. Well-motivated problem. The paper addresses a relevant limitation of static visual-token pruning: evidence discarded for the initial question may become necessary later. Treating visual selection as a revisable, fixed-budget process provides a useful perspective on multi-turn context management.
  2. Useful controlled experiments. Comparisons against random refresh and previous-question refresh support the importance of current-question relevance.
Critical weaknesses
  1. Incomplete latency comparison. Appendix C indicates that Full-576 uses context reconstruction and answer replay. An additional full-token baseline that reuses its existing KV cache would clarify BVTR’s practical accuracy–latency trade-off under conventional multi-turn inference.
  2. Limited justification for specific design choices. Global reselection achieves similar accuracy, while simpler repair variants match Sparse96 in the ablation. The benefits of bounded replacement and drift-based repair are therefore insufficiently demonstrated.
  3. Narrow empirical evaluation. Gains are not statistically significant on VQAv2 and are absent with the TransPrune selector. Longer conversations and cross-model generalization of Sparse96 remain insufficiently evaluated.
  4. Incomplete discussion of related work. The paper lacks a sufficiently comprehensive review and analysis of closely related studies, such as [1, 2, 3].

[1] Yang S, Chen Y, Tian Z, et al. Visionzip: Longer is better but not necessary in vision language models[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 19792-19802.

[2] Wang H, Yu Z, Spadaro G, et al. Folder: Accelerating multi-modal large language models with enhanced performance[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 23614-23625.

[3] Ju C, Wang H, Cheng H, et al. Turbo: Informativity-driven acceleration plug-in for vision-language large models[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 436-455.

Questions for authors
  1. How does BVTR compare with Full-576 when the latter retains and incrementally extends its existing KV cache? Please report end-to-end latency and total peak memory under this setting.
  2. How are COCO-SR1000 and COCO-MT4-200 constructed, especially the conditions used to identify omitted visual evidence?
  3. What accuracy or efficiency benefits do bounded replacement and drift-based repair provide over simpler alternatives?
  4. How are historical QA states updated, and how does refresh cost scale with dialogue length?
Reviewer_uj1wOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates visual-token pruning for multi-turn large vision–language models. The authors argue that reusing the visual-token subset selected for the first question may discard evidence needed by later questions when the user’s visual focus shifts. They formulate the problem as updating the active visual-token set under a fixed token budget and introduce Budgeted Visual Token Refresh (BVTR). At each follow-up turn, BVTR uses question-conditioned attention scores to replace a bounded number of previously active tokens with inactive but newly relevant ones. To address the resulting inconsistency between new token states and the existing KV cache, the paper considers dense reconstruction and a selective sparse contextual-repair implementation. Experiments on LLaVA-1.5-7B and Qwen3-VL-8B-Instruct report improvements in controlled omission-recovery settings, while the gains are limited on more general VQA questions and with a stronger initial token selector.

Critical strengths
  1. A relevant multi-turn problem formulation and an intuitive refresh framework. The paper identifies a meaningful limitation of static visual-token pruning: evidence excluded for the first question may be needed when the user’s visual focus changes in later turns. Framing token management as a revisable, fixed-budget problem is therefore relevant to multi-turn visual understanding. The proposed question-conditioned replacement rule is straightforward, keeps the active-token budget constant, and does not require retraining the underlying vision–language model.
  2. Attention to cache consistency and informative experimental comparisons. The authors recognize that replacing visual tokens also affects context-dependent KV states, and compare dense reconstruction with a selective sparse-update implementation rather than assuming cache entries can be swapped independently. The evaluation includes static pruning, random refresh, global reselection, multiple question shifts, and latency and memory measurements. It also reports limited or nonsignificant gains under some evaluation conditions. These comparisons provide useful evidence about both the potential benefit and the boundaries of token refresh.
Critical weaknesses
  1. The methodological novelty is limited, and the bounded-replacement principle lacks a demonstrated advantage over simpler alternatives. The central procedure uses question-conditioned token scores to deactivate low-ranked active tokens and reactivate high-ranked inactive tokens. This is largely an adaptation of existing relevance scoring and token-selection operations to a multi-turn setting, rather than a substantially new selection or optimization mechanism. More importantly, global reselection achieves 90.9% accuracy versus 91.0% for BVTR-Sparse96 in the main controlled evaluation. The post-hoc study also reports the same observed accuracy for replacement budgets from M = 16 onward. Consequently, the results do not convincingly show why bounded replacement is preferable to directly reselecting a token subset, or what distinctive algorithmic benefit the proposed constraint provides.
  2. The necessity and validity of sparse contextual repair are not sufficiently established. The proposed sparse implementation ranks retained tokens by their KV-state change at an intermediate layer and selectively recomputes deeper-layer states. However, the remaining cached states were still computed under an earlier visual context, and the paper provides little quantitative evidence about the residual inconsistency or its effect on model representations. More critically, the ablation shows identical observed accuracy when repairing only newly activated tokens, adding high-change retained tokens, adding random or low-change retained tokens, or using dense reconstruction. These results do not substantiate the importance of the KV-change criterion, the choice of K = 96, or the proposed sparse-repair mechanism beyond a much simpler update rule.
  3. The empirical gains depend strongly on controlled omission-recovery conditions, with limited evidence of broad practical utility. The strongest results are obtained on a diagnostic benchmark explicitly designed to expose visual evidence discarded by first-turn pruning. Although this establishes that refresh can recover omitted information, it does not establish how frequently or severely this failure occurs in natural multi-turn conversations. On VQAv2-Shift500, BVTR-Sparse96 improves over static pruning by only 0.34 percentage points without statistical significance (p = 0.570). The TransPrune-based evaluation likewise shows no improvement. Moreover, the cross-model result validates dense refresh rather than the proposed Sparse96 configuration. Stronger evidence is needed that the approach yields consistent benefits outside the specific conditions constructed to favor refresh.
  4. The additional inference cost substantially weakens the proposed accuracy–efficiency trade-off. Visual-token pruning is primarily motivated by inference efficiency, yet BVTR must rescore candidate visual tokens and update cached states at follow-up turns. BVTR-Sparse96 has a median follow-up latency of 296.2 ms, compared with 143.7 ms for static pruning, a 2.06× increase. Its latency advantage over full-token inference (335.9 ms) is relatively small, while the dense Qwen3-VL implementation is slower than full-token inference. Although incremental-memory reductions are reported, the paper does not establish that the often modest accuracy improvements justify the extra computational cost in realistic workloads. A more detailed cost breakdown and evaluation over longer, natural dialogues would be needed to demonstrate a compelling deployment benefit.
Questions for authors
  1. Novelty and bounded replacement. What substantive algorithmic advantage does limiting the number of swapped tokens offer over simply selecting a fresh set of B tokens at every turn? Given the similar accuracy of global reselection and BVTR, and the flat ablation results for several replacement budgets, can the authors demonstrate an independent benefit of the bounded-refresh constraint?
  2. Necessity of the sparse-repair rule. Why should intermediate-layer KV-state changes identify the retained tokens that most need recomputation? How do the authors explain the identical observed accuracy for newly activated tokens only, high-change selection, random selection, low-change selection, and dense reconstruction? Could they compare state discrepancies and downstream behavior against the dense reference?
  3. Conditions under which refresh is useful. Can the authors characterize which question types, spatial shifts, and levels of first-turn information loss actually make static pruning fail? Is there a lightweight mechanism for deciding whether to refresh at a given turn, rather than incurring the selection and repair costs even when the existing tokens are sufficient?
  4. Efficiency and scalability. Can the authors separate the end-to-end cost of question-conditioned token scoring, state recomputation, answer replay, and cache management? How does the accuracy–latency–memory trade-off change with longer conversations, larger token budgets, and different numbers of follow-up turns, particularly compared with static pruning and full-token inference?
  5. Generalization beyond the diagnostic setting. Can the authors evaluate on more natural multi-turn VQA dialogues where changes of visual focus are not explicitly constructed to recover first-turn omissions? In addition, can the sparse contextual-repair implementation itself be tested across model families, rather than transferring only the dense-refresh accuracy effect?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.