acceptodds
Under review as a conference paper at ICLR 2027

Prune Once, Route per Turn: Constructing Persistent Visual Memory for Multi-Turn VLM Inference

Abstract

Large vision-language models (VLMs) have achieved impressive performance across diverse multimodal tasks, but their substantial inference cost becomes particularly challenging in multi-turn interaction, where the same visual input is repeatedly queried. Although visual token pruning provides an effective way to reduce inference cost, our empirical analysis reveals that naively extending single-turn pruning strategies to multi-turn scenarios yields two key limitations: cross-turn pruning mismatch, in which visual evidence discarded for one query becomes essential for subsequent turns; and costly query adaptation, where static retained subsets fail to accommodate evolving queries, whereas recomputing query-aware representations every turn introduces substantial latency overhead. Motivated by these findings, we introduce PersistRoute, a simple training-free framework that decouples session-level visual memory construction from turn-wise visual Key-Value (KV) routing. Specifically, PersistRoute comprises two complementary components: (1) Session Memory Prune (SMPrune), which conducts one-shot, query-agnostic visual-token pruning to build a diverse visual memory reused throughout the whole session, mitigating cross-turn pruning mismatch; and (2) Dynamic Query-Aware Visual KV Routing (DQVR), which selects query-relevant visual KVs from this persistent memory for each incoming query, enabling adaptive query processing without repeated pruning or memory reconstruction. Evaluated across multiple multi-turn benchmarks and diverse VLM backbones, PersistRoute establishes a more favorable performance-efficiency Pareto frontier compared to prior baselines. Using 50% persistent visual memory with merely 5% routed visual KVs, it preserves 96.2% of the full-model performance on MT-VQA-v2 and 95.6% on MT-GQA, delivering up to 2.54× end-to-end session-level inference speedup.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Visual Token Pruning, Efficient Inference, Multi-Turn Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wxK9OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses efficient multi-turn VLM inference. Existing visual token pruning methods are designed for single-turn settings, causing cross-turn evidence mismatch and costly per-turn query adaptation. The authors propose PersistRoute, a training-free framework that decouples session-level visual memory construction from turn-wise visual routing. SMPrune performs one-shot, query-agnostic pruning to build a persistent visual memory shared across the session, using three visual criteria: Session Sufficiency, Evidence Removability, and Future-Query Resilience. DQVR then dynamically routes query-relevant visual KVs from this memory at each turn, while unselected KVs remain available for later queries. Experiments on three multi-turn benchmarks (MT-VQA-v2, MT-GQA, ConvBench) and five VLM backbones show that PersistRoute retains 96.2% and 95.6% of original performance with 50% persistent memory and 5% routed visual KVs, achieving up to 2.54× session end-to-end speedup. Ablations validate the complementary roles of SMPrune and DQVR.

Critical strengths

The paper targets a genuinely valuable problem: efficient multi-turn VLM inference, where most prior token pruning methods are single-turn only. As multi-turn deployment grows, addressing cross-turn evidence mismatch and per-turn adaptation cost is practically important, making the problem framing relevant to real-world VLM serving.

The decoupled design—persistent visual memory plus turn-wise KV routing—is simple, training-free, and easy to integrate. It achieves strong accuracy retention (e.g., ~96% at 5% routed KVs) on several backbones, showing practical potential even if components are not fundamentally new.

Critical weaknesses
  1. The two-stage design is essentially a concatenation of visual token pruning and query-aware KV routing. SMPrune performs query-agnostic visual selection; DQVR then routes among the surviving M tokens. If SMPrune discards an object relevant to a later turn, DQVR cannot recover it—a limitation the authors acknowledge. In practice, the query-agnostic SMPrune relies on CLIP-space cosine similarity to define token uniqueness, but CLIP features are not object-instance discriminative. Tokens from the same object may have low similarity, while tokens from different objects may have high similarity. As a result, the decoupling is just a reasonable engineering choice, but the methodological novelty is limited: neither component is new, and the combination does not fundamentally solve the cross-turn evidence loss problem it identifies.
  2. Experimental evaluation is incomplete in several respects. (1) Efficiency reporting lacks KV cache footprint analysis across compression ratios and batch-size scalability experiments; only batch=1 is tested. (2) The baseline set omits SparseVILA (ICCV 2025), which addresses the same multi-turn setting with a similar decoupled prefill-decode paradigm, and DART (EMNLP 2025). (3) Ablations do not systematically compare SMPrune against alternative query-agnostic constructors, nor provide guidance on optimal M/K allocation under a fixed total budget.
Questions for authors

See the weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer wxK9 (Part 1/3)

We sincerely appreciate the reviewer's careful evaluation and recognition of the importance of efficient multi-turn VLM inference. We have carefully considered the valuable feedback and provide point-by-point responses supported by additional experiments and analyses. We hope these responses address the reviewer's concerns.

[W1(1)] The two-stage design is essentially a concatenation of visual token pruning and query-aware KV routing.

We acknowledge that visual pruning and query-aware KV selection have been individually explored. However, our proposed PersistRoute is not intended as a simple concatenation of these techniques; it addresses two distinct, interconnected requirements of multi-turn inference: preserving visual evidence before future queries are known and efficiently accessing relevant evidence as the conversation evolves.

Specifically, our SMPrune constructs evidence-diverse persistent memory through Session Sufficiency, Evidence Removability, and Future-Query Resilience, while our DQVR dynamically selects question-relevant visual KVs at decoder layer-group entrances and reuses selections across consecutive layers.

To clarify the methodological differences from existing approaches, Table R1 compares representative methods in their technical routes, persistent-memory designs, turn-wise adaptation mechanisms, and cross-layer routing reuse.

Table R1. Comparison of Representative Methods in Multi-Turn Visual Evidence Allocation.

MethodTechnical RoutePersistent Session MemoryTurn-Wise AdaptationCross-Layer Routing ReuseTraining-FreeMT-GQA Acc. (%) ↑
SEATSStage-adaptive pruningNo dedicated memoryNo dedicated turn-wise routingNoYes59.51
DARTPivot-based token pruningNo dedicated memoryNo dedicated turn-wise settingNoYes58.12
MetaCompressLearned token compressionLearned compressed representationNo dynamic KV routingNoNo58.83
SparseVILACache + per-layer retrievalSalience-based memoryQuery-aware retrievalNo (per-layer retrieval)Yes60.29
PersistRoute (Ours)Persistent memory + group-wise routingEvidence-diverse memory (SMPrune)Question-adaptive KV routing (DQVR)Yes (layer-group reuse)Yes62.07

As summarized in Table R1, our proposed PersistRoute distinguishes itself by jointly incorporating evidence-diverse persistent memory, turn-wise adaptive KV access, and layer-group routing reuse in a training-free framework. This coordinated design specifically targets cross-turn evidence mismatch and repeated routing overhead.

Importantly, we further validate that the improvements do not arise merely from combining two existing stages. Table R2 presents a controlled constructor–router crossover under identical memory and active-KV budgets.

Table R2. Constructor–Router Crossover.

ConstructorRouterAcc. (%) ↑E2E (ms) ↓
SparseVILASparseVILA60.29576.74
SparseVILADQVR60.79481.65
SMPruneSparseVILA61.35589.47
SMPruneDQVR62.07491.53

As shown in Table R2, under matched M=288 and K=58, our SMPrune independently contributes +1.06–1.28 pp, while our DQVR contributes +0.50–0.72 pp. Furthermore, four-group DQVR reduces routing overhead by 77.3% relative to per-layer routing with only 0.252 pp accuracy loss.

These results support the distinct algorithmic contributions of persistent evidence construction and efficient question-adaptive routing, rather than attributing the gains solely to their two-stage combination.

AuthorsOpenReview ↗

Response to Reviewer wxK9 (Part 2/3)

[W1(2)] If SMPrune discards an object relevant to a later turn, DQVR cannot recover it—a limitation the authors acknowledge.

As the reviewer points out, our DQVR cannot recover visual tokens permanently discarded by SMPrune. We therefore aim to minimize irreversible loss during memory construction while enabling different future queries to access different subsets of the retained evidence, rather than claiming to eliminate such loss.

To assess this limitation, we examine persistent-memory quality and later-turn evidence access. At M/N=50%, our SMPrune achieves the highest Utility Capture (57.75) and Critical Survival Retention (65.04), and the lowest Restoration AUC (0.24) among the evaluated constructors (Appendix Table 13). In three-turn sessions, SMPrune retains 73.90% of annotated future-target region tokens. Conditional on retained evidence, DQVR achieves an 81.35% future-routing hit rate, and 25.87% of previously inactive target observations become active in later turns. These results demonstrate that persistent evidence preservation and subsequent query-adaptive access address complementary aspects of cross-turn information loss.

[W1(3)] The query-agnostic SMPrune relies on CLIP-space cosine similarity to define token uniqueness, but CLIP features are not object-instance discriminative.

We agree with the reviewer's concern that CLIP cosine similarity alone does not guarantee object-instance discrimination. However, our SMPrune does not rely solely on global similarity: Session Sufficiency incorporates regional salience, complemented by Evidence Removability and Future-Query Resilience. To examine this concern, we ablate regional coverage with DQVR disabled: Global + Region improves Full-Session accuracy over Global-only on GQA (60.16 vs. 59.97%) and VQA-v2 (58.82 vs. 58.70%). Across all measurable instance pairs, same-instance similarity exceeds different-instance similarity by 0.086 on average (95% CI [0.070, 0.103]).

These findings support the value of regional coverage and provide evidence of average instance-level separation in CLIP features.

[W2(1)] Efficiency reporting lacks KV cache footprint analysis across compression ratios and batch-size scalability experiments; only batch=1 is tested.

As requested, we extend standard three-turn MT-GQA efficiency evaluation with unique live KV footprint measurements across compression ratios (Table R3) and batch-size scalability experiments (Table R4), directly addressing the gaps in the original BS=1 profiling.

Table R3. Unique Live KV Footprint (MiB).

MethodK/N=5%K/N=10%K/N=30%
SparseVILA912.1941.11056.1
PersistRoute364.3393.3508.2
Reduction60.1%58.2%51.9%

As shown in Table R3, our proposed PersistRoute reduces the measured unique live KV footprint by 51.9–60.1% relative to SparseVILA.

Table R4. Batch-Size Scalability.

MethodBS=1 ↑BS=2 ↑BS=4 ↑Peak Mem. BS=4 ↓
Full1.6082.4563.31321.292
SparseVILA1.7342.7844.16417.803
PersistRoute2.0343.4885.47216.932

Throughput: sessions/s; peak memory: GiB.

As shown in Table R4, at BS=4, PersistRoute achieves 31.4% higher throughput than SparseVILA with 0.871 GiB lower peak memory, demonstrating that its benefits extend beyond BS=1.

[W2(2)] The baseline set omits SparseVILA (ICCV 2025), which addresses the same multi-turn setting with a similar decoupled prefill-decode paradigm, and DART (EMNLP 2025).

Following the reviewer's suggestion, we add the missing SparseVILA (ICCV 2025) and DART (EMNLP 2025) comparisons under matched final visual-token budgets, as reported in Table R5.

Table R5. Additional Baseline Comparison.

MethodK/N=5% ↑K/N=10% ↑K/N=30% ↑
Full64.1664.1664.16
SparseVILA59.8960.2960.20
DART (Turn-wise Refresh)54.4958.1261.64
PersistRoute61.3362.0762.65

As shown in Table R5, our PersistRoute achieves the highest accuracy at all three budgets while each method retains its distinct update mechanism. DART performs turn-wise token selection with its pivot-based duplication mechanism, whereas our framework constructs persistent memory once and performs subsequent query-adaptive routing.

AuthorsOpenReview ↗

Response to Reviewer wxK9 (Part 3/3)

[W2(3)] Ablations do not systematically compare SMPrune against alternative query-agnostic constructors, nor provide guidance on optimal M/K allocation under a fixed total budget.

To address the requested systematic constructor comparison, we note that Appendix Tables 8(a) and 13 already evaluate our SMPrune against several query-agnostic constructors under matched settings, including memory quality without DQVR. We further compare four query-agnostic constructors (Random, VisionZip, PruMerge, and ZOO-Prune) and two first-turn-dependent alternatives (CDPruner and DART) under identical memory budgets and DQVR routing (Table R6).

Table R6. Persistent-Memory Constructor Comparison.

ConstructorK/N=5% ↑K/N=12.5% ↑K/N=25% ↑
VisionZip56.1356.3156.70
PruMerge56.0356.3056.21
ZOO-Prune56.5756.5956.68
Random55.9056.4756.54
CDPruner (Q1)55.7456.8456.93
DART (Q1 Constructor)57.1657.9558.41
SMPrune57.2658.4358.52

As shown in Table R6, on LLaVA-1.5-7B/VQA-v2 Full-Session, our SMPrune achieves the highest accuracy at all three budgets, outperforming both query-agnostic and first-turn-dependent alternatives. These results support its effectiveness in constructing persistent visual memory without relying on conversation queries.

M/K allocation guidance. To address the request for budget-dependent allocation guidance, the original Figure 4 and Appendix Figure 6 systematically explore M/N=10–70% and K/M=5–75%. Table R7 reports the highest-accuracy measured M/K configurations satisfying each latency or memory limit.

Table R7. Resource-Aware M/K Allocation.

Resource TargetMKAcc. (%) ↑E2E (ms) ↓Mem. (GiB) ↓
Latency ≈400 ms1162956.25396.1915.15
Latency ≈500 ms28814458.52497.6315.56
Latency ≈550 ms40414259.18544.0516.46
Latency ≈575 ms34626059.33573.6516.89
Memory ≈15.0 GiB58653.33357.5015.00
Memory ≈15.5 GiB1168756.76413.6515.30
Memory ≈16.75 GiB40420259.26564.0716.64

Allocation insights. As illustrated in Table R7, the best measured allocation varies with resource constraints. Under a 400 ms latency budget, M=116/K=29 achieves 56.25% accuracy; increasing the budget to 500 ms favors M=288/K=144 (58.52%). These results provide concrete, resource-aware M/K allocation guidance: sufficient M preserves future-relevant evidence, while K controls active evidence access. The preferred balance depends on the available latency and memory budget rather than a universal fixed ratio.

We again thank the reviewer for the constructive suggestions. We hope the additional evidence clarifies our design choices and addresses the concerns raised, and we will incorporate the new results into the revised manuscript.

Reviewer_UdFLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets visual token pruning in multi-turn VLM inference. Pilot studies show that tokens pruned for one question are often needed later, while re-selecting tokens at every turn is costly. The proposed PersistRoute is training-free and has two stages. SMPrune prunes once per session to build a query-agnostic persistent visual memory, combining CLS-attention salience with local and global diversity scores. DQVR then, at each turn, uses the current question's query–key attention to pick the Top-K visual KVs from that memory for each group of decoder layers. With 50% memory and 5% routed KVs on LLaVA-1.5-7B, it keeps 96.2% / 95.6% of full accuracy on MT-VQA-v2 / MT-GQA (3.80 / 3.55 points above the best baseline at the same K/N), and it reports up to 2.54× session speedup on LLaVA-NeXT-7B across three benchmarks and five backbones.

Critical strengths
  1. The pilot studies are interesting and set up the problem well. Fig. 1 separates two failure modes that are easy to blur together: evidence that a single-turn pruner throws away and a later question then needs, and the latency cost of fixing this by re-selecting at every turn (430.60 → 829.68 ms for FastV, 398.61 → 608.30 ms for CDPruner). This makes the motivation concrete rather than hypothetical.

  2. The paper points at a real pain point in multi-turn inference, namely how visual information is stored versus how it is used, and gives a clean answer. Separating "what to keep for the whole session" from "what to look at for this question" ("prune once, route per turn") is a simple and sensible principle. The method is training-free and works inside the frozen backbone's own attention path, so it is easy to adopt.

  3. The experiments are detailed and the argument is careful. The paper covers three benchmarks and five backbones (LLaVA-1.5-7B/13B, LLaVA-NeXT-7B, Qwen2.5-VL-7B, Qwen3-VL-8B), a full efficiency profile (TTFT, prefill, decode, session E2E, peak memory), and many ablations: memory × routing factorization, routing rules, memory constructors, layer-group granularity, criteria, and hyperparameters. Hyperparameters are fixed once on a development set built from the VQA-v2 training split (App. B.3), the Pre-LLM and In-LLM ratios are reported side by side in Table 1, and pseudocode and code are provided.

Critical weaknesses
  1. The SMPrune criteria are named more ambitiously than what they compute, and the module reads like a combination of existing cues. Session Sufficiency, Evidence Removability and Future-Query Resilience are in practice CLS attention (as in VisionZip and PruMerge), dissimilarity to spatial neighbors, and global uniqueness, i.e., salience plus diversity, which existing pruners such as DivPrune and CDPruner already use. CLIP's CLS attention also favors the dominant subject that matters for image–text contrast, while the paper's own motivation (Fig. 1(a)) is to keep the background and small objects that later questions may ask about. Because this signal is global, the method depends on the regional grid, which is the most sensitive hyperparameter (4×4 / 6×6 / 8×8 give 56.43 / 56.92 / 55.95 in Table 12, while k and τ move accuracy by at most 0.39), and that is hard to connect to "sufficiency". Future-Query Resilience is global uniqueness computed without any query, so resilience to future questions is not something the design ensures (the Limitations section itself calls it "a query-independent proxy").

  2. DQVR is very close to query-aware KV selection in text LLMs, and this connection is not discussed. SnapKV (Li et al., 2024) keeps the KV entries that receive the most attention from an observation window of prompt tokens, and Quest (Tang et al., 2024) loads the Top-K KV pages scored by (an upper bound of) the current query's q·k. DQVR scores the cached visual keys with the current question's q·k attention and gathers the Top-K. The main differences are that the score is averaged over heads, shared within each of four layer groups, and computed once per turn instead of per head or per decoding step. Neither line of work is cited, and the multimodal KV-cache methods that the paper does cite in §2 (LOOK-M, AirCache, VL-Cache, MadaKV, HybridKV) are not compared against, even though they also reduce the visual KVs used during decoding. The paper's own control points the same way: its Global Query-Aware Top-K baseline is only 0.41–0.54 points behind DQVR (Table 4). I think DQVR's novelty lies mainly in applying this known idea on top of a persistent visual memory, and the paper should position and evaluate it that way.

  3. The two components are tightly coupled, it is hard to tell what each contributes, and DQVR's own gain on a fixed memory is small. On its own, SMPrune is not a better constructor than existing pruners. At the same 50% memory with DQVR off, its macro accuracy (59.49) is below CDPruner (59.80), ZOO-Prune (59.77) and PruMerge (59.57), and only 0.82 points above random selection (58.67), even though it leads on the paper's own diagnostics such as WTR and Utility Capture (Table 13). Used directly as a pruner, it is the weakest at 5% (49.36 vs. 53.51–54.22, Table 8(b)). In the other direction, routing adds little once the memory is fixed. Over Static-Q1, DQVR gains 0.23–0.91 points with SMPrune memory and 0.30–0.75 with full memory on LLaVA-NeXT-7B (Table 3). On MT-GQA it is 1.53–2.10 points above random routing and 0.41–0.54 above a global query-aware Top-K (Table 4). Fig. 4 tells the same story: memory size M/N moves accuracy by 4.8–5.7 points, while K/M moves it by only 0.57–2.05. The combination does work better than the parts, and Table 8(a) shows that other constructors barely improve as K grows (e.g., ZOO-Prune 56.57 → 56.68 from 5% to 25%) while SMPrune does, but the paper does not explain why SMPrune memory benefits from routing so much more. As written, I cannot tell how much of the headline improvement belongs to SMPrune, to DQVR, or simply to the larger memory (see Questions 2 and 3).

Questions for authors
  1. Could the routing score be dominated by sink-like visual tokens? DQVR shares one selection across all heads and averages over every question token, including function words like "what / is / the" (Eq. 6), so tokens that score high for any query may fill the Top-K. Does this explain why routing from the full memory is worse than routing from SMPrune memory at every ratio (e.g., 57.65 vs. 58.34 at K/N = 5%, Table 3)? Per-head selection, content-word-only routing, or excluding sink tokens would help, as would the overlap of routed sets across questions in a session.

  2. How does SMPrune do without routing, i.e., with all 288 tokens attended at every turn? This control decides whether DQVR is needed. Table 11 gives its accuracy (58.82 on VQA-v2 with DQVR disabled), which is on par with most routed results (e.g., 58.22 at K/N = 10% in Table 2), but only on the diagnostic subset and without latency. Since latency depends mostly on M rather than K (raising K from 144 to 720 only moves E2E from 652 to 738 ms, Table 9), please add this variant to Tables 1 and 2. A per-stage latency breakdown would also help: on LLaVA-1.5-7B PersistRoute is about 95 ms slower than VisionZip Static Reuse, but Prefill and Decode explain only about 29 ms of the gap.

  3. Can you compare at equal memory and cost? Table 1 matches only K/N, but at 5% PersistRoute keeps ten times more visual tokens before the LLM than the baselines (0.50 vs. 0.05). On the same benchmark, SMPrune + DQVR leads the best direct pruner by 3.04 points at 5%, but the best constructor given the same 50% memory and DQVR by only 0.69 (Table 8). Please add (a) two-stage baselines with the same 50% pre-LLM ratio, such as IVTP, VScan, or a pre-LLM pruner followed by FastV; (b) CDPruner and random constructors in Table 8(a), since CDPruner is the most accurate at 50% memory (Table 13); (c) SnapKV- or Quest-style routers on the same memory; and (d) accuracy at matched latency and peak memory.

if all of the questions could be resolved, I would consider to raise my score, thank you!

6 comments
AuthorsOpenReview ↗

Response to Reviewer UdFL (Part 1/5)

We sincerely thank the reviewer for the careful assessment, encouraging comments, and specific suggestions. Below, we address each concern separately and explain how the new experiments test the questions raised.

[W1] The SMPrune criteria are named more ambitiously than what they compute, and the module reads like a combination of existing cues.

W1.1 — "a combination of existing cues." We agree that CLS salience and feature diversity are established primitives, and do not claim otherwise. SMPrune's contribution is their joint use to construct a query-agnostic, persistent evidence pool for unknown future questions, rather than a new salience or diversity primitive. Its three criteria operationalize retaining salient evidence (Session Sufficiency), limiting local redundancy (Evidence Removability), and favoring globally distinctive evidence (Future-Query Resilience). Unlike single-turn pruning, the resulting M tokens remain accessible for later query-adaptive routing.

W1.2 — "CLIP's CLS attention also favors the dominant subject." We clarify that Fig. 1(a) illustrates cross-turn evidence mismatch, rather than implying that SMPrune specifically targets background or small objects. Our objective is to construct persistent, query-agnostic visual memory that preserves complementary evidence for uncertain future queries. Since global CLS attention may overemphasize dominant subjects, we incorporate regional salience to complement global evidence selection. To isolate its contribution, we compare Global-only with Global + Region on LLaVA-1.5-7B, holding the other criteria fixed and disabling DQVR (Table R1).

Table R1. Regional Salience Ablation.

VariantGQA ↑VQA ↑Macro ↑WTR ↑Utility ↑CSR ↑
Global-only59.9758.7059.3491.4656.3763.10
Global + Region60.1658.8259.4992.7457.7565.04

Regional salience improves GQA/VQA accuracy by 0.19/0.12 pp, WTR by 1.28 pp, and CSR by 1.94 pp. Thus, the regional component improves evidence retention diagnostics, although its isolated accuracy gain is modest.

W1.3 — "the regional grid ... is the most sensitive hyperparameter. Resilience to future questions is not something the design ensures." We acknowledge the reviewer's concern about grid sensitivity: the original Table 12 reports 56.43/56.92/55.95 for 4×4/6×6/8×8 grids. We selected 6×6 on a separate development split. We also clarify that FQR is a query-independent proxy for future-query robustness, not a guarantee of preserving all future evidence.

W1.4 — Does the constructor work better in its intended persistent-memory role? To address the concern that SMPrune's standalone accuracy is not consistently best, we compare six constructors with identical M/N=50% and the same DQVR router, using VQA-v2 Full-Session (Table R2).

Table R2. Persistent-Memory Constructor Comparison (VQA-v2 Full-Session, Acc. %).

ConstructorK/N=5% ↑12.5% ↑25% ↑
Random55.9056.4756.54
CDPruner55.7456.8456.93
VisionZip56.1356.3156.70
PruMerge56.0356.3056.21
ZOO-Prune56.5756.5956.68
SMPrune57.2658.4358.52

SMPrune exceeds the strongest alternative by 0.69/1.59/1.59 pp at K/N=5%/12.5%/25%.

[W2] DQVR is very close to query-aware KV selection in text LLMs, and this connection is not discussed.

W2.1 — Relation to "SnapKV" and "Quest." We appreciate these references and agree that query-aware KV selection itself is not new. SnapKV uses observation-window attention for prompt-KV compression; Quest uses query-dependent bounds to select KV pages. DQVR instead routes visual KVs from a session-persistent memory once per dialogue turn and shares each selection within decoder layer groups. Its distinction is the multimodal evidence pool, turn-wise update schedule, and group-shared access—not the use of q·k scores alone. We will cite and position both works explicitly in §2.

W2.2 — "Global Query-Aware Top-K ... only 0.41–0.54 points behind DQVR." To test whether DQVR's advantage survives stronger query-aware controls, as suggested, Table R3 compares Static-Q1, Global Query Top-K, and SnapKV-/Quest-style variants under the same SMPrune memory (M/N=50%, K/N=10%). We also include Qwen2.5-VL-7B, alongside LLaVA-1.5-7B, on both MT-GQA and MT-VQA-v2 to check whether the result is backbone-specific.

AuthorsOpenReview ↗

Response to Reviewer UdFL (Part 2/5)

Table R3. Query-Aware Router Comparison.

RouterLLaVA GQA ↑LLaVA VQA ↑Qwen GQA ↑Qwen VQA ↑LLaVA Routing (ms/turn) ↓Qwen Routing (ms/turn) ↓
Static-Q160.3373.3257.1184.742.753.89
Global Query Top-K61.6075.2457.9386.696.278.84
Quest-style61.9575.0158.4786.56115.6290.60
SnapKV-style61.7274.9558.5586.4087.2984.86
DQVR62.0775.7358.6186.888.1510.62

DQVR achieves the highest accuracy in all four model–dataset settings. For example, on LLaVA MT-GQA it reaches 62.07%, versus 61.60% for Global Query Top-K, while on Qwen MT-VQA-v2 it reaches 86.88%, versus 86.69%. Its measured routing overhead is also lower than the evaluated SnapKV-/Quest-style implementations.

W2.3 — "shared within each of four layer groups." The reviewer correctly identifies layer-group sharing as a key design choice. To quantify its benefit instead of merely describing it, Table R4 varies group count on LLaVA-NeXT-7B/MT-GQA with M/N=50% and K/N=12.5%. We also include SparseVILA, a method that performs query-aware visual KV retrieval at each decoder layer, for comparison.

Table R4. Layer-Group Routing Ablation.

StrategyGroupsAcc. (%) ↑Routing (ms/turn) ↓
DQVR160.2386.921
DQVR260.9067.112
DQVR (default)462.1368.849
DQVR862.19213.284
DQVR (per-layer)3262.38838.901
SparseVILA (per-layer)3261.59089.211

Relative to per-layer DQVR, four-group routing reduces measured selection overhead from 38.901 to 8.849 ms/turn (−77.3%), with only 0.252 pp lower accuracy. This directly addresses why grouping is used: it trades a small accuracy difference for substantially less repeated selection work.

[W3] The two components are tightly coupled, it is hard to tell what each contributes, and DQVR's own gain on a fixed memory is small.

W3.1 — "hard to tell what each contributes." To explicitly disentangle the contributions of SMPrune and DQVR, we conduct a controlled 2×2 constructor–router crossover on LLaVA-1.5-7B/MT-GQA, fixing M=288 and K=58 (Table R5).

Table R5. Constructor × Router Crossover.

ConfigurationConstructorRouterAcc. (%) ↑E2E (ms) ↓
SparseVILASparseVILARetrieval60.29576.74
SparseVILA + DQVRSparseVILADQVR60.79481.65
SMPrune + SparseVILASMPruneRetrieval61.35589.47
PersistRouteSMPruneDQVR62.07491.53

With the router fixed, replacing SparseVILA's constructor with SMPrune adds 1.06–1.28 pp; with the constructor fixed, replacing SparseVILA retrieval with DQVR adds 0.50–0.72 pp. Together, PersistRoute improves accuracy by 1.78 pp and reduces E2E latency by 14.8% compared with SparseVILA. These results demonstrate the independent and complementary contributions of SMPrune and DQVR under matched budgets.

W3.2 — "DQVR's own gain on a fixed memory is small." We agree that its marginal accuracy gain can be modest; the design also targets less repeated routing work. Table R4 shows the 77.3% overhead reduction from layer grouping, while Table R5 shows that DQVR improves accuracy under either fixed constructor. The corrected LLaVA-NeXT comparison (Table R7) also makes the budget dependence clear: DQVR−Static-Q1 is +0.65/+0.91/+0.23 pp with SMPrune memory and +0.53/+0.53/+0.65 pp with Full memory at 5%/12.5%/25%. Thus, the argument for DQVR is an accuracy–efficiency trade-off, not a large universal accuracy improvement.

W3.3 — "why SMPrune memory benefits from routing so much more." Standalone pruning tests whether M tokens suffice when all M are used, whereas Table R2 tests the harder M→K access setting. SMPrune's scores rise from 57.26 to 58.52 as K/N increases from 5% to 25%, and it remains above the other tested constructors at each budget. This is consistent with its intended persistent-memory role; Q2 below separately measures whether accessing all M entries is preferable to routing K.

AuthorsOpenReview ↗

Response to Reviewer UdFL (Part 3/5)

[Q1] Could the routing score be dominated by sink-like visual tokens?

The reviewer raises three possible explanations for the unexpected results in Table 3, where Full+DQVR underperforms SMPrune+DQVR: function-word interference, head-averaged selection, and attention-sink dominance. We investigate these possibilities through controlled routing ablations.

Q1.1 — "averages over every question token, including function words." To assess whether function words dilute query-specific relevance, we evaluate Content-only routing, excluding function words from query aggregation.

Q1.2 — "shares one selection across all heads." To examine whether head averaging overlooks head-specific evidence, we evaluate Per-head routing, allowing each attention head to select its own visual KVs.

Q1.3 — "excluding sink tokens would help." To test whether consistently high-attention tokens dominate Top-K selection, we evaluate Sink-excluded routing under strict and relaxed criteria.

All variants use LLaVA-1.5-7B on the complete MT-GQA benchmark, with M=288 and K=58.

Table R6. Routing Mechanism Ablation.

VariantAcc. (%) ↑Routing (ms) ↓
DQVR (default)62.078.51
Content-only62.2313.77
Per-head62.3043.36
Sink-excluded (strict)62.0718.40
Sink-excluded (relaxed)62.0818.70

Findings. Content-only and Per-head routing yield modest accuracy improvements (+0.16/+0.23 pp) at increased routing costs (13.77/43.36 ms vs. 8.51 ms). Under strict (attention ≥ 2/M, CV ≤ 0.25) and relaxed (attention ≥ 2/M, CV ≤ 0.50) sink criteria, exclusion changes accuracy by at most 0.01 pp. These results support the accuracy–efficiency trade-off of our default aggregation, with no substantial benefit from the tested sink-exclusion strategies.

Q1.4 — "why routing from the full memory is worse than routing from SMPrune memory ... (Table 3)." To directly investigate this anomaly, we audited the original LLaVA-NeXT ablations and identified a visual preprocessing error confined to the Full+Static-Q1 and Full+DQVR configurations. This error retained visual KVs corresponding to padded-image regions in the routing candidate pool, allowing them to occupy part of the limited Top-K budget. We corrected both affected ablations and re-evaluated the complete benchmark. DQVR's scoring and KV-selection algorithm remain unchanged.

Table R7. Corrected Full-Memory Ablations and Cross-Backbone Verification (Acc. %, ↑).

BackboneMemory + RouterK/N=5%12.5%25%
LLaVA-NeXT-7BFull + Static-Q1 (corrected)57.8558.7959.86
Full + DQVR (corrected)58.3859.3260.51
SMPrune + Static-Q157.6958.2559.67
SMPrune + DQVR58.3459.1659.90
LLaVA-1.5-7BFull + DQVR57.9958.5758.96
SMPrune + DQVR57.2658.4358.52
Qwen2.5-VL-7BFull + DQVR60.7761.8762.16
SMPrune + DQVR59.7461.2161.37

Corrected results. The original Table 3 reported Full+DQVR below SMPrune+DQVR at K/N=5% (57.65% vs. 58.34%). After correction, Full+DQVR achieves 58.38%, slightly above SMPrune+DQVR (58.34%). Full+DQVR now achieves higher accuracy at all three budgets on LLaVA-NeXT (+0.04/+0.16/+0.61 pp). To further verify this trend, we evaluate LLaVA-1.5-7B and Qwen2.5-VL-7B, where Full+DQVR also achieves higher accuracy across all three budgets. The corrected results resolve the unexpected Table 3 ordering and identify the affected ablation preprocessing as its source. We will update the corresponding results in the revised manuscript.

Reviewer_8QMTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PersistRoute accelerates multi-turn vision-language inference over the same image. SMPrune constructs persistent visual memory once, while DQVR selects visual key–value entries for each turn. With 50% persistent memory and 5% active visual entries, it retains approximately 96% of baseline performance on two visual question answering benchmarks. A separate Full-Session setting achieves up to 2.54× speedup.

Critical strengths
  1. Practical efficiency gains. Table 2 reports a 2.54× session speedup on LLaVA-NeXT-7B with a 1.05 percentage-point accuracy drop.
  2. Controlled ablations. Tables 3–5 separately examine memory construction, dynamic routing, and the scoring criteria.
Critical weaknesses
  1. Limited framework novelty. Visual compression followed by query-aware selection already appears in SEATS [1]. Persistent visual caching with per-query retrieval already appears in SparseVILA [2]. The identifiable additions are mainly the scoring combination and layer-group routing. Without direct comparisons against the closest framework, their substantive contribution remains unclear.
  2. Limited efficiency validation. Accuracy experiments cover multiple benchmarks and backbones, while measured efficiency focuses on Full-Session evaluation and two LLaVA models. The performance efficiency advantage remains unverified under the standard three-turn protocol and on Qwen backbones.

Reference
[1] Stage-adaptive Token Selection for Efficient Omni-modal LLMs.
[2] SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference.

Questions for authors
  1. What specific algorithmic mechanisms are new relative to SEATS and SparseVILA?
  2. Under matched backbone, precision, persistent-memory size, and active visual-cache budget, how does PersistRoute compare with SparseVILA in accuracy, latency, and memory usage?
  3. What are the session-level confidence intervals for the key gains in Tables 3–5?
  4. Does the method still offer a better balance between accuracy and efficiency in conversations with three turns and on Qwen models?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 8QMT (Part 1/3)

We sincerely thank the reviewer for the careful evaluation and insightful feedback. Below, we address each comment point by point, supported by additional experiments and analyses.

[W1 & Q1] Limited framework novelty: What specific algorithmic mechanisms are new relative to SEATS and SparseVILA?

We appreciate the reviewer's concern regarding novelty. We acknowledge that visual pruning and query-aware retrieval have been explored in prior work. Our contribution is not the two-stage decomposition itself, but two complementary mechanisms specifically designed for multi-turn visual evidence allocation.

First, our proposed SMPrune constructs a query-agnostic, evidence-diverse persistent memory by jointly considering Session Sufficiency, Evidence Removability, and Future-Query Resilience. Unlike SEATS's progressive pruning and SparseVILA's salience-based memory construction, SMPrune explicitly considers regional coverage, evidence complementarity, and distinctiveness to preserve potentially useful information for uncertain future queries.

Second, our DQVR performs question-conditioned visual KV routing at decoder layer-group entrances, reusing selections across consecutive layers rather than repeating SparseVILA-style retrieval at every decoder layer.

To make these distinctions explicit, Table R1 compares the technical routes and key mechanisms of representative approaches, including SEATS, DART, MetaCompress, and SparseVILA.

Table R1. Comparison of Representative Methods in Multi-Turn Visual Evidence Allocation.

MethodTechnical RoutePersistent Session MemoryTurn-Wise AdaptationCross-Layer Routing ReuseTraining-FreeMT-GQA Acc. (%) ↑
SEATSStage-adaptive pruningNo dedicated memoryNo dedicated turn-wise routingNoYes59.51
DARTPivot-based token pruningNo dedicated memoryNo dedicated turn-wise settingNoYes58.12
MetaCompressLearned token compressionLearned compressed representationNo dynamic KV routingNoNo58.83
SparseVILACache + per-layer retrievalSalience-based memoryQuery-aware retrievalNo (per-layer retrieval)Yes60.29
PersistRoute (Ours)Persistent memory + group-wise routingEvidence-diverse memory (SMPrune)Question-adaptive KV routing (DQVR)Yes (layer-group reuse)Yes62.07

As shown in Table R1, our proposed PersistRoute uniquely combines evidence-diverse session-level memory construction, turn-wise question-adaptive KV routing, and cross-layer routing reuse among the compared methods, within a training-free framework.

To further distinguish these mechanisms from simply combining existing stages, we conduct a controlled constructor–router crossover under matched M=288 and K=58, as reported in Table R2.

Table R2. Framework Comparison and Component Crossover.

ConfigurationConstructorRouterAcc. (%) ↑E2E (ms) ↓
SEATSSEATS (50%)Native (20%)59.51613.27
SparseVILASparseVILARetrieval60.29576.74
SparseVILA + DQVRSparseVILADQVR60.79481.65
SMPrune + SparseVILASMPruneRetrieval61.35589.47
PersistRouteSMPruneDQVR62.07491.53

The four crossover configurations use LLaVA-1.5-7B/MT-GQA with M=288 and K=58; SEATS follows its native configuration.

As shown in Table R2, replacing the constructor with our SMPrune improves accuracy by 1.06–1.28 pp, while replacing the retrieval mechanism with our DQVR contributes 0.50–0.72 pp with the counterpart fixed. These controlled gains establish the independent contributions of the two mechanisms.

Table R3. Routing Granularity and Retrieval Overhead.

Routing StrategyGranularityAcc. (%) ↑Overhead (ms/turn) ↓
SparseVILA RetrievalPer-layer (32)61.59089.211
DQVRPer-layer (32)62.38838.901
DQVR (Ours)4 Groups62.1368.849

As shown in Table R3, on LLaVA-NeXT-7B/MT-GQA, our four-group DQVR reduces routing overhead by 77.3% relative to per-layer DQVR, with only 0.252 pp accuracy loss.

Together, Tables R1–R3 demonstrate that the proposed PersistRoute goes beyond a generic pruning–retrieval combination through distinct memory-construction and routing mechanisms with independently measurable accuracy and efficiency benefits.

AuthorsOpenReview ↗

Response to Reviewer 8QMT (Part 2/3)

[W2] Limited efficiency validation. The performance efficiency advantage remains unverified under the standard three-turn protocol and on Qwen backbones.

As the reviewer suggests, we extend efficiency evaluation to standard three-turn MT-GQA on both LLaVA-1.5-7B and Qwen2.5-VL-7B, addressing the previously missing Qwen profiling. Table R5 additionally compares Static Reuse and Turn-wise Refresh for VisionZip, CDPruner, and ZOO-Prune.

[Q2] Under matched backbone, precision, persistent-memory size, and active visual-cache budget, how does PersistRoute compare with SparseVILA in accuracy, latency, and memory usage?

Following the request for a matched comparison, Table R4 evaluates SEATS, SparseVILA, and our PersistRoute across three multi-turn benchmarks on the same LLaVA-1.5-7B backbone and final K/N budgets. To further control the persistent-memory and active-cache sizes, Table R2 fixes M=288 and K=58 in the SparseVILA–PersistRoute constructor–router crossover; Table R5 provides the corresponding three-turn latency and peak-memory comparison.

Table R4. Closest-Framework Accuracy Comparison (LLaVA-1.5-7B).

DatasetMethodK/N=5% ↑K/N=10% ↑K/N=30% ↑
MT-GQAFull64.1664.1664.16
SEATS57.3259.5161.91
SparseVILA59.8960.2960.20
PersistRoute61.3362.0762.65
MT-VQA-v2Full77.2777.2777.27
SEATS69.3571.9974.89
SparseVILA72.4572.9372.82
PersistRoute74.3275.7376.67
ConvBenchFull5.325.325.32
SEATS4.915.255.83
SparseVILA4.975.035.03
PersistRoute5.495.665.99

As shown in Table R4, our PersistRoute outperforms both SEATS and SparseVILA at all three budgets across all three benchmarks. At K/N=10% on MT-GQA, it improves accuracy over SparseVILA by 1.78 pp. The controlled crossover (Table R2) and three-turn profiling (Table R5) further address the reviewer's questions about the individual components, latency, and memory usage.

[Q4] Does the method still offer a better balance between accuracy and efficiency in conversations with three turns and on Qwen models?

As requested, we evaluate whether the accuracy–efficiency trade-off persists in three-turn conversations and on Qwen models. Table R5 compares accuracy, end-to-end latency, and peak GPU memory on both backbones using identical timing subsets.

Table R5. Standard Three-Turn Accuracy and Efficiency Comparison (MT-GQA, K/N=10%).

BackboneMethodUpdateAcc. (%) ↑E2E (ms) ↓Mem. (GiB) ↓
LLaVA-1.5-7BFullStatic64.16621.7117.07
VisionZipStatic57.33375.6415.41
VisionZipRefresh57.33517.4115.50
CDPrunerStatic57.93404.1715.66
CDPrunerRefresh59.46563.8315.66
ZOO-PruneStatic60.22476.8616.72
ZOO-PruneRefresh60.22602.4617.04
SparseVILARetrieval60.29576.7415.34
PersistRouteDQVR62.07491.5314.36
Qwen2.5-VL-7BFullStatic60.86960.1615.78
VisionZipStatic55.13471.7915.55
VisionZipRefresh55.13700.6515.59
CDPrunerStatic53.32479.0615.55
CDPrunerRefresh54.67698.9315.59
ZOO-PruneStatic57.70604.5715.95
ZOO-PruneRefresh57.50751.9115.98
SparseVILARetrieval58.33659.1715.94
PersistRouteDQVR58.61636.4715.73

As shown in Table R5, our PersistRoute improves accuracy over SparseVILA by 1.78/0.28 pp, reduces E2E latency by 14.8%/3.4%, and lowers peak memory on LLaVA/Qwen, respectively. Relative to Full, it achieves 1.26×/1.51× speedups.

The proposed PersistRoute is also faster than all evaluated Turn-wise Refresh variants. Although some Static Reuse methods are faster, they sacrifice accuracy and lack query-adaptive visual-KV access. The Qwen results demonstrate that the efficiency advantage extends beyond LLaVA, albeit with a smaller gain over SparseVILA.

AuthorsOpenReview ↗

Response to Reviewer 8QMT (Part 3/3)

[Q3] What are the session-level confidence intervals for the key gains in Tables 3–5?

As requested, we compute paired session-level bootstrap 95% confidence intervals for all 13 comparisons in the original Tables 3–5, resampling complete conversations to preserve within-session dependencies. The numerical estimates are reported in Table R6.

Table R6. Paired Bootstrap Confidence Intervals.

Original TableComparisonBudgetΔ Acc. (pp)95% CI
Table 3DQVR − Static-Q15%+0.65[0.02, 1.28]
DQVR − Static-Q112.5%+0.91[0.30, 1.52]
DQVR − Static-Q125%+0.23[−0.38, 0.84]
SMPrune − Full Memory (DQVR)5%-0.04[−0.11, 0.03]
SMPrune − Full Memory (DQVR)12.5%−0.16[−0.30, −0.03]
SMPrune − Full Memory (DQVR)25%−0.61[−0.93, −0.30]
Table 4DQVR − Global Top-K5%+0.54[0.11, 0.97]
DQVR − Global Top-K10%+0.47[0.04, 0.91]
DQVR − Global Top-K30%+0.41[−0.02, 0.85]
Table 5Full SMPrune − S-only—+0.64[0.16, 1.16]
Full SMPrune − w/o S—+0.64[0.15, 1.12]
Full SMPrune − w/o R—+0.47[0.02, 0.92]
Full SMPrune − w/o F—+0.55[0.07, 1.03]

As shown in Table R6, 10/13 point estimates are positive, and 10/13 nominal 95% intervals exclude zero. The remaining three cross zero, so those differences are less certain. We will include the interval estimates and bootstrap protocol in the revised Tables 3–5.

We thank the reviewer again for the insightful feedback. We hope these additional results and clarifications address the concerns raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.