- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper addresses efficient multi-turn VLM inference. Existing visual token pruning methods are designed for single-turn settings, causing cross-turn evidence mismatch and costly per-turn query adaptation. The authors propose PersistRoute, a training-free framework that decouples session-level visual memory construction from turn-wise visual routing. SMPrune performs one-shot, query-agnostic pruning to build a persistent visual memory shared across the session, using three visual criteria: Session Sufficiency, Evidence Removability, and Future-Query Resilience. DQVR then dynamically routes query-relevant visual KVs from this memory at each turn, while unselected KVs remain available for later queries. Experiments on three multi-turn benchmarks (MT-VQA-v2, MT-GQA, ConvBench) and five VLM backbones show that PersistRoute retains 96.2% and 95.6% of original performance with 50% persistent memory and 5% routed visual KVs, achieving up to 2.54× session end-to-end speedup. Ablations validate the complementary roles of SMPrune and DQVR.
Critical strengths
The paper targets a genuinely valuable problem: efficient multi-turn VLM inference, where most prior token pruning methods are single-turn only. As multi-turn deployment grows, addressing cross-turn evidence mismatch and per-turn adaptation cost is practically important, making the problem framing relevant to real-world VLM serving.
The decoupled design—persistent visual memory plus turn-wise KV routing—is simple, training-free, and easy to integrate. It achieves strong accuracy retention (e.g., ~96% at 5% routed KVs) on several backbones, showing practical potential even if components are not fundamentally new.
Critical weaknesses
- The two-stage design is essentially a concatenation of visual token pruning and query-aware KV routing. SMPrune performs query-agnostic visual selection; DQVR then routes among the surviving M tokens. If SMPrune discards an object relevant to a later turn, DQVR cannot recover it—a limitation the authors acknowledge. In practice, the query-agnostic SMPrune relies on CLIP-space cosine similarity to define token uniqueness, but CLIP features are not object-instance discriminative. Tokens from the same object may have low similarity, while tokens from different objects may have high similarity. As a result, the decoupling is just a reasonable engineering choice, but the methodological novelty is limited: neither component is new, and the combination does not fundamentally solve the cross-turn evidence loss problem it identifies.
- Experimental evaluation is incomplete in several respects. (1) Efficiency reporting lacks KV cache footprint analysis across compression ratios and batch-size scalability experiments; only batch=1 is tested. (2) The baseline set omits SparseVILA (ICCV 2025), which addresses the same multi-turn setting with a similar decoupled prefill-decode paradigm, and DART (EMNLP 2025). (3) Ablations do not systematically compare SMPrune against alternative query-agnostic constructors, nor provide guidance on optimal M/K allocation under a fixed total budget.
Questions for authors
See the weaknesses
3 comments
Response to Reviewer wxK9 (Part 2/3)
[W1(2)] If SMPrune discards an object relevant to a later turn, DQVR cannot recover it—a limitation the authors acknowledge.
As the reviewer points out, our DQVR cannot recover visual tokens permanently discarded by SMPrune. We therefore aim to minimize irreversible loss during memory construction while enabling different future queries to access different subsets of the retained evidence, rather than claiming to eliminate such loss.
To assess this limitation, we examine persistent-memory quality and later-turn evidence access. At M/N=50%, our SMPrune achieves the highest Utility Capture (57.75) and Critical Survival Retention (65.04), and the lowest Restoration AUC (0.24) among the evaluated constructors (Appendix Table 13). In three-turn sessions, SMPrune retains 73.90% of annotated future-target region tokens. Conditional on retained evidence, DQVR achieves an 81.35% future-routing hit rate, and 25.87% of previously inactive target observations become active in later turns. These results demonstrate that persistent evidence preservation and subsequent query-adaptive access address complementary aspects of cross-turn information loss.
[W1(3)] The query-agnostic SMPrune relies on CLIP-space cosine similarity to define token uniqueness, but CLIP features are not object-instance discriminative.
We agree with the reviewer's concern that CLIP cosine similarity alone does not guarantee object-instance discrimination. However, our SMPrune does not rely solely on global similarity: Session Sufficiency incorporates regional salience, complemented by Evidence Removability and Future-Query Resilience. To examine this concern, we ablate regional coverage with DQVR disabled: Global + Region improves Full-Session accuracy over Global-only on GQA (60.16 vs. 59.97%) and VQA-v2 (58.82 vs. 58.70%). Across all measurable instance pairs, same-instance similarity exceeds different-instance similarity by 0.086 on average (95% CI [0.070, 0.103]).
These findings support the value of regional coverage and provide evidence of average instance-level separation in CLIP features.
[W2(1)] Efficiency reporting lacks KV cache footprint analysis across compression ratios and batch-size scalability experiments; only batch=1 is tested.
As requested, we extend standard three-turn MT-GQA efficiency evaluation with unique live KV footprint measurements across compression ratios (Table R3) and batch-size scalability experiments (Table R4), directly addressing the gaps in the original BS=1 profiling.
Table R3. Unique Live KV Footprint (MiB).
| Method | K/N=5% | K/N=10% | K/N=30% |
|---|---|---|---|
| SparseVILA | 912.1 | 941.1 | 1056.1 |
| PersistRoute | 364.3 | 393.3 | 508.2 |
| Reduction | 60.1% | 58.2% | 51.9% |
As shown in Table R3, our proposed PersistRoute reduces the measured unique live KV footprint by 51.9–60.1% relative to SparseVILA.
Table R4. Batch-Size Scalability.
| Method | BS=1 ↑ | BS=2 ↑ | BS=4 ↑ | Peak Mem. BS=4 ↓ |
|---|---|---|---|---|
| Full | 1.608 | 2.456 | 3.313 | 21.292 |
| SparseVILA | 1.734 | 2.784 | 4.164 | 17.803 |
| PersistRoute | 2.034 | 3.488 | 5.472 | 16.932 |
Throughput: sessions/s; peak memory: GiB.
As shown in Table R4, at BS=4, PersistRoute achieves 31.4% higher throughput than SparseVILA with 0.871 GiB lower peak memory, demonstrating that its benefits extend beyond BS=1.
[W2(2)] The baseline set omits SparseVILA (ICCV 2025), which addresses the same multi-turn setting with a similar decoupled prefill-decode paradigm, and DART (EMNLP 2025).
Following the reviewer's suggestion, we add the missing SparseVILA (ICCV 2025) and DART (EMNLP 2025) comparisons under matched final visual-token budgets, as reported in Table R5.
Table R5. Additional Baseline Comparison.
| Method | K/N=5% ↑ | K/N=10% ↑ | K/N=30% ↑ |
|---|---|---|---|
| Full | 64.16 | 64.16 | 64.16 |
| SparseVILA | 59.89 | 60.29 | 60.20 |
| DART (Turn-wise Refresh) | 54.49 | 58.12 | 61.64 |
| PersistRoute | 61.33 | 62.07 | 62.65 |
As shown in Table R5, our PersistRoute achieves the highest accuracy at all three budgets while each method retains its distinct update mechanism. DART performs turn-wise token selection with its pivot-based duplication mechanism, whereas our framework constructs persistent memory once and performs subsequent query-adaptive routing.
Response to Reviewer wxK9 (Part 3/3)
[W2(3)] Ablations do not systematically compare SMPrune against alternative query-agnostic constructors, nor provide guidance on optimal M/K allocation under a fixed total budget.
To address the requested systematic constructor comparison, we note that Appendix Tables 8(a) and 13 already evaluate our SMPrune against several query-agnostic constructors under matched settings, including memory quality without DQVR. We further compare four query-agnostic constructors (Random, VisionZip, PruMerge, and ZOO-Prune) and two first-turn-dependent alternatives (CDPruner and DART) under identical memory budgets and DQVR routing (Table R6).
Table R6. Persistent-Memory Constructor Comparison.
| Constructor | K/N=5% ↑ | K/N=12.5% ↑ | K/N=25% ↑ |
|---|---|---|---|
| VisionZip | 56.13 | 56.31 | 56.70 |
| PruMerge | 56.03 | 56.30 | 56.21 |
| ZOO-Prune | 56.57 | 56.59 | 56.68 |
| Random | 55.90 | 56.47 | 56.54 |
| CDPruner (Q1) | 55.74 | 56.84 | 56.93 |
| DART (Q1 Constructor) | 57.16 | 57.95 | 58.41 |
| SMPrune | 57.26 | 58.43 | 58.52 |
As shown in Table R6, on LLaVA-1.5-7B/VQA-v2 Full-Session, our SMPrune achieves the highest accuracy at all three budgets, outperforming both query-agnostic and first-turn-dependent alternatives. These results support its effectiveness in constructing persistent visual memory without relying on conversation queries.
M/K allocation guidance. To address the request for budget-dependent allocation guidance, the original Figure 4 and Appendix Figure 6 systematically explore M/N=10–70% and K/M=5–75%. Table R7 reports the highest-accuracy measured M/K configurations satisfying each latency or memory limit.
Table R7. Resource-Aware M/K Allocation.
| Resource Target | M | K | Acc. (%) ↑ | E2E (ms) ↓ | Mem. (GiB) ↓ |
|---|---|---|---|---|---|
| Latency ≈400 ms | 116 | 29 | 56.25 | 396.19 | 15.15 |
| Latency ≈500 ms | 288 | 144 | 58.52 | 497.63 | 15.56 |
| Latency ≈550 ms | 404 | 142 | 59.18 | 544.05 | 16.46 |
| Latency ≈575 ms | 346 | 260 | 59.33 | 573.65 | 16.89 |
| Memory ≈15.0 GiB | 58 | 6 | 53.33 | 357.50 | 15.00 |
| Memory ≈15.5 GiB | 116 | 87 | 56.76 | 413.65 | 15.30 |
| Memory ≈16.75 GiB | 404 | 202 | 59.26 | 564.07 | 16.64 |
Allocation insights. As illustrated in Table R7, the best measured allocation varies with resource constraints. Under a 400 ms latency budget, M=116/K=29 achieves 56.25% accuracy; increasing the budget to 500 ms favors M=288/K=144 (58.52%). These results provide concrete, resource-aware M/K allocation guidance: sufficient M preserves future-relevant evidence, while K controls active evidence access. The preferred balance depends on the available latency and memory budget rather than a universal fixed ratio.
We again thank the reviewer for the constructive suggestions. We hope the additional evidence clarifies our design choices and addresses the concerns raised, and we will incorporate the new results into the revised manuscript.
Response to Reviewer wxK9 (Part 1/3)
We sincerely appreciate the reviewer's careful evaluation and recognition of the importance of efficient multi-turn VLM inference. We have carefully considered the valuable feedback and provide point-by-point responses supported by additional experiments and analyses. We hope these responses address the reviewer's concerns.
[W1(1)] The two-stage design is essentially a concatenation of visual token pruning and query-aware KV routing.We acknowledge that visual pruning and query-aware KV selection have been individually explored. However, our proposed PersistRoute is not intended as a simple concatenation of these techniques; it addresses two distinct, interconnected requirements of multi-turn inference: preserving visual evidence before future queries are known and efficiently accessing relevant evidence as the conversation evolves.
Specifically, our SMPrune constructs evidence-diverse persistent memory through Session Sufficiency, Evidence Removability, and Future-Query Resilience, while our DQVR dynamically selects question-relevant visual KVs at decoder layer-group entrances and reuses selections across consecutive layers.
To clarify the methodological differences from existing approaches, Table R1 compares representative methods in their technical routes, persistent-memory designs, turn-wise adaptation mechanisms, and cross-layer routing reuse.
Table R1. Comparison of Representative Methods in Multi-Turn Visual Evidence Allocation.
As summarized in Table R1, our proposed PersistRoute distinguishes itself by jointly incorporating evidence-diverse persistent memory, turn-wise adaptive KV access, and layer-group routing reuse in a training-free framework. This coordinated design specifically targets cross-turn evidence mismatch and repeated routing overhead.
Importantly, we further validate that the improvements do not arise merely from combining two existing stages. Table R2 presents a controlled constructor–router crossover under identical memory and active-KV budgets.
Table R2. Constructor–Router Crossover.
As shown in Table R2, under matched M=288 and K=58, our SMPrune independently contributes +1.06–1.28 pp, while our DQVR contributes +0.50–0.72 pp. Furthermore, four-group DQVR reduces routing overhead by 77.3% relative to per-layer routing with only 0.252 pp accuracy loss.
These results support the distinct algorithmic contributions of persistent evidence construction and efficient question-adaptive routing, rather than attributing the gains solely to their two-stage combination.