acceptodds
Withdrawn from ICLR 2027

ViFlow-KV: Visual-Flow Guided KV Cache Eviction For Multimodal Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon multimodal reasoning combines dense visual inputs with growing textual trajectories, making KV cache memory a major inference bottleneck for Large Multimodal Models (LMMs). KV cache eviction mitigates this burden by selectively retaining high-utility states under a limited cache budget. However, existing methods typically apply a modality-agnostic eviction policy, overlooking both the distinct retention requirements of visual and textual states and their evolving interaction during reasoning. Our attention analysis characterizes a heterogeneous visual flow: LMMs gradually shift the access to visual information from the original visual inputs toward later generated hidden states. Furthermore, later queries revisit both the original visual evidence and earlier generated states that carry visual information, revealing a long-range reuse of visual information. Guided by these observations, we propose ViFlow-KV, a visual-flow-guided KV eviction framework for multimodal reasoning. At each eviction step, ViFlow-KV assigns modality-specific budgets to visual and textual states, while incorporating visual-anchor signals to preserve reasoning-relevant visual evidence across evolving carriers. By preserving visual evidence across evolving visual and textual carriers, ViFlow-KV maintains critical context throughout long reasoning while keeping cache highly compressed. Extensive experiments show that ViFlow-KV consistently outperforms existing eviction methods, matching full attention performance at 2k token budget while achieving up to 3.05 throughput speedup.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Reasoning, Large Multimodal Models, Attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QGfjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ViFlow-KV, a training-free KV cache eviction method for multimodal reasoning. It combines evolving modality-demand allocation, which dynamically divides each head’s cache budget between visual and textual states, with visual-anchor importance estimation. Experiments cover six multimodal benchmarks, with additional evaluations across model scales and families. At a 2k token budget, the method retains 98% of the FullKV average score. Separate system experiments report up to 3.05x throughput improvement over FullKV.

Critical strengths

The paper addresses a relevant problem with a training-free design. Maintaining access to visual evidence throughout a long reasoning trajectory is important under constrained KV memory. The combination of evolving modality allocation and visually informed importance estimation provides a practical way to address this problem without retraining the model. ViFlow-KV outperforms the selected baselines across all four cache budgets, and the component ablations show that both EMDA and VAIE contribute.

Critical weaknesses
  • The main insight that visual and textual states require different retention policies is already present in prior work. MadaKV (ACL 2025)[1] explicitly allocates modality-specific budgets within each attention head in proportion to attention-derived modality importance, which is closely related to EMDA’s allocation rule. CSP[2] studies separate treatment of intra- and inter-modality attention, and SparseMM[3] allocates budgets according to visual head preferences.

  • The observation that previously unimportant visual evidence may become relevant again also motivates query-aware sparse retrieval that preserves historical KV entries, as in Quest[4], and learned dynamic selection, as in DSA[5]. It would be great to evaluate a non-evicting dynamic-selection setting or establish whether its modality-aware design remains beneficial there.

[1] https://arxiv.org/abs/2506.15724

[2] https://arxiv.org/abs/2412.04652

[3] https://arxiv.org/abs/2506.05344

[4] https://arxiv.org/abs/2406.10774

[5] https://arxiv.org/abs/2512.02556

Questions for authors
  1. Beyond using a recent pre-RoPE query centroid and updating the allocation during generation, what distinguishes EMDA from the attention-proportional modality budgeting in MadaKV? Which aspects of the proposed allocation are intended as new contributions?

  2. Can the authors compare against a strong modality-aware eviction baseline (e.g., MadaKV, SparseMM) under matched cache budgets and eviction schedules, and evaluate the additional benefit of VAIE on top of that baseline?

  3. At matched accuracy, how does ViFlow-KV’s decoding efficiency compare with the eviction baselines, allowing each method to use the KV budget required to reach that accuracy? End-to-end latency or throughput comparisons that include eviction overhead would help establish whether the improved accuracy-budget trade-off translates into practical efficiency gains.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Thanks for your insightful comments and questions.

W1. [The main insight that visual and textual states require different retention policies is already present in prior work.]

A: Thank you. Our work provides the first in-depth characterization of KV cache dynamics in long-horizon multimodal reasoning, revealing how visual evidence propagates from original visual states into generated textual representations. This insight guides evolving modality-aware budget allocation and visual-flow-guided importance estimation, enabling ViFlow-KV to identify and preserve important visual and visually grounded textual states as reasoning progresses. Thus, our main contribution goes beyond simply assigning different retention policies/budgets to visual and textual states.

In contrast, MadaKV estimates static head-level modality preferences from the input context, SparseMM offline identifies visually relevant attention heads and allocates larger budgets, while CSP refines importance estimation by distinguishing intra- and inter-modality attention. However, they do not explicitly account for the growing importance of generated textual states that carry visual evidence during subsequent reasoning and reflection. We further provide additional comparisons with both methods in our response to Q2.

W2. [It would be great to evaluate a non-evicting dynamic-selection setting or establish whether its modality-aware design remains beneficial there.]

A: Thank you for the insightful question. Our work focuses on KV cache compression through permanent eviction, whereas Quest and DSA dynamically select KV entries for attention without permanently removing them. Due to their different optimization objectives, ViFlow-KV cannot be directly transferred to non-evicting retrieval settings. Given the orthogonal nature of the two approaches, we explore their combination by integrating ViFlow-KV into Quest, examining the compatibility of this modality-aware design with query-aware retrieval. The results are shown below:

MethodKV BudgetRetrieval BudgetMMStarCharXiv RQMathVistaAvg.
QuestFull51270.0047.2072.8063.33
Quest + ViFlow-KV2,04851268.9347.9071.7062.84

The results demonstrate that ViFlow-KV can be effectively integrated into Quest, substantially reducing the KV retention budget while maintaining comparable performance. Furthermore, these results suggest that the insights from visual flow could extend beyond eviction-based compression, offering a general perspective for multimodal reasoning. We leave further exploration to future work.

Q1. [Beyond using a recent pre-RoPE query centroid and updating the allocation during generation, what distinguishes EMDA from the attention-proportional modality budgeting in MadaKV? Which aspects of the proposed allocation are intended as new contributions?]

A: Thanks. The pre-RoPE query centroid and continuous budget updates are precisely what make EMDA effective and distinctive. Our visual-flow analysis firstly reveals a gradual shift in reliance from original visual states to generated textual states during reasoning. Accordingly, the pre-RoPE query centroid accurately estimates current modality demands, while continuous budget updates dynamically track their evolution during reasoning. We provide fine-grained ablation studies of these design choices of EMDA in our response to Reviewer gEux's W1. In contrast, MadaKV allocates modality budgets based on attention mass from the initial prompt. However, it could not explicitly account for the decreasing reliance on original visual states and increasing reliance on generated textual states as reasoning progresses. We provide additional comparisons with MadaKV in our response to Q2 to further demonstrate the effectiveness of EMDA.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Q2. [Can the authors compare against a strong modality-aware eviction baseline (e.g., MadaKV, SparseMM) under matched cache budgets and eviction schedules, and evaluate the additional benefit of VAIE on top of that baseline?]

A: Thank you. We conduct additional comparisons with MadaKV and SparseMM under matched KV cache budgets and eviction schedules. Furthermore, We integrate VAIE into both baselines to evaluate its additional benefits. We also report the average generated length to examine the impact of different KV retention strategies. The results are presented below:

KV Budget = 512

MethodMMStarMathVistaCharXiv RQAvg. Accuracy (%)Avg. Generated Length
SparseMM61.2762.6026.3050.067916
SparseMM + VAIE62.1364.9028.8051.947397
MadaKV53.7354.3024.6044.215164
MadaKV + VAIE59.5355.9029.6048.344768
ViFlow-KV63.9366.6040.2056.912623

KV Budget = 2048

MethodMMStarMathVistaCharXiv RQAvg. Accuracy (%)Avg. Generated Length
SparseMM66.9371.7041.3059.985421
SparseMM + VAIE68.1373.4043.1061.545211
MadaKV66.4770.1045.0060.523440
MadaKV + VAIE67.7373.2047.7062.883085
ViFlow-KV71.4778.0053.3067.592522

The results show that ViFlow-KV consistently outperforms MadaKV and SparseMM while generating substantially fewer reasoning tokens. By examining the reasoning trajectories, we find that these baselines allocate larger budgets to original visual tokens, leaving insufficient capacity for visually grounded textual states and leading to redundant visual verification. In contrast, ViFlow-KV better preserves both original visual evidence and its historical textual carriers, facilitating evidence reuse and reducing redundant reasoning. Furthermore, VAIE consistently improves both baselines, confirming the benefits of retaining visually grounded historical states. We will incorporate the above analysis in the revision.

Q3. [At matched accuracy, how does ViFlow-KV’s decoding efficiency compare with the eviction baselines, allowing each method to use the KV budget required to reach that accuracy?]

A: Increasing the KV budget would increase GPU memory consumption and limit batch size. In contrast, ViFlow-KV achieves better accuracy with only marginal additional eviction overhead, offering a more practical trade-off than retaining more KV states at a higher memory cost. To evaluate decoding efficiency at comparable accuracy, we compare ViFlow-KV with R-KV while allowing each method to use its own KV budget. The results are shown below:

MethodBudgetMMStarMathVistaCharXiv RQAvg.Throughput (tok/s)
R-KV204870.6776.0050.7065.791033.5
R-KV256071.4777.2052.3067.00955.1
ViFlow-KV204871.4778.0053.3067.591007.8

The results show that ViFlow-KV achieves higher average accuracy with a smaller KV budget than R-KV. Notably, even when increasing its budget to 2560, R-KV still falls short of ViFlow-KV in average accuracy while incurring lower throughput. Further increasing the KV budget would incur more memory costs and further reduce efficiency. Overall, this better trade-off highlights the advantage of ViFlow-KV's visual-flow-guided retention strategy.

AuthorsOpenReview ↗

Looking Forward to Your Reply

Thank you once again for taking the time to review our paper and for providing such insightful and constructive feedback.

We have carefully considered each of your comments and have provided detailed responses. We sincerely hope that our efforts adequately address your concerns and contribute positively to your evaluation.

As the author-reviewer discussion period concludes on Nov 18 (AoE), we would greatly appreciate any further feedback you may have. If you have any additional questions or require any clarifications, please do not hesitate to reach out to us.

Reviewer_iKUbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies KV cache compression for long multimodal reasoning. It dynamically allocates cache capacity according to visual and textual attention demands and uses queries with strong visual attention to select states for retention. The method requires no training and outperforms the compared cache eviction methods on six benchmarks. With a budget of 2,048 tokens, it recovers 98.36% of the average performance of the full cache.

Critical strengths

1.The method considers both the original visual states and subsequent textual states that carry visual information, closely connecting its design to the practical needs of long multimodal reasoning. 2.Experiments cover multiple cache budgets, six benchmarks, and different models. Adding either component individually also improves performance, supporting the method's effectiveness and some degree of transferability (Tables 1, 2, and 4).

Critical weaknesses

1.The direct baselines are mainly SnapKV, R-KV, and Ada-RKV. Although the related work discusses multimodal cache allocation methods such as SparseMM, direct comparisons are missing. The advantage over methods more closely aligned with this task setting therefore remains unclear.

2.Validation of the proposed explanation based on “visual flow” remains limited. The similarity and attention co-occurrence in Figure 3 show an association, and Table 2 demonstrates the components' effectiveness, but these results do not establish whether the gains depend on the identified carriers of visual information. Controls using a fixed visual/text cache ratio and randomly or uniformly selected anchors would test the design rationale more directly.

3.The throughput comparisons in Table 3 use the maximum batch size supported by each method. For example, the 3.05× improvement corresponds to increasing the batch size from 21 to 141. This demonstrates throughput gains from memory savings, but results at the same batch size are also needed to assess the latency impact of the eviction computation itself.

Questions for authors

1.At the same batch size, how do task latency and peak memory usage compare with FullKV and R-KV? Do these measurements include the overhead of query caching, importance computation, and cache reorganization?

2.With the same cache budget and number of anchors, how do a fixed modality ratio, random anchors, and uniformly selected anchors compare with the current design?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Thanks for your constructive comments and suggestions.

W1. [The advantage over methods (i.e. SparseMM) more closely aligned with this task setting therefore remains unclear.]

A: Thank you for this constructive suggestion. SparseMM primarily targets multimodal long-context understanding with direct answer generation, while our work focuses on long-horizon multimodal reasoning with continuously growing textual trajectories. To further assess our competitiveness, we compare ViFlow-KV with SparseMM under matched KV cache budgets and eviction schedules. The results are shown follow:

KV BudgetMethodMMStarMathVistaCharXiv RQAvg.
512SparseMM61.2762.6026.3050.06
ViFlow-KV63.9366.6040.2056.91
2048SparseMM66.9371.7041.3059.98
ViFlow-KV71.4778.0053.3067.59

The results show that ViFlow-KV consistently outperforms SparseMM across all settings, demonstrating its effectiveness in long-horizon multimodal reasoning.

W2. [Validation of the proposed explanation based on “visual flow” remains limited. Controls using a fixed visual/text cache ratio and randomly or uniformly selected anchors would test the design rationale more directly.]

A: Thank you. We conduct additional ablations on CharXiv RQ with a KV budget of 512, comparing ViFlow-KV against fixed visual cache ratios (5%, 10%, 25%, and 50%) and alternative anchor selection strategies using random or uniform sampling. For the latter, we maintain the same number of anchors as ViFlow-KV and sample them from the same window as our method. The results are presented below.

VariantAccuracy (%)Δ\Delta
ViFlow-KV40.20—
Fixed Modality Allocation
Fixed visual ratio (5%)36.40−3.80
Fixed visual ratio (10%)36.40−3.80
Fixed visual ratio (25%)34.90−5.30
Fixed visual ratio (50%)30.20−10.00
Alternative Anchor Selection
w/o VAIE (EMDA only)39.40−0.80
Random anchor selection38.30−1.90
Uniform anchor selection38.60−1.60

The results show that both fixed modality allocation and alternative anchor selection consistently underperform ViFlow-KV. The degradation under static allocation demonstrates the need to adapt KV budgets to evolving visual–textual demands, while the advantage of visual-anchor selection confirms the importance of visually grounded signals for historical KV retention.

Furthermore, we provide a case study (Appendix C) comparing how ViFlow-KV and R-KV retain original visual tokens and their visually matched textual states throughout reasoning. The results show that ViFlow-KV preserves substantially more visually matched textual states, ultimately producing the correct answer. In contrast, R-KV gradually loses critical visually grounded textual states, leading to incorrect reasoning. This case study provides further qualitative support for visual flow and the importance of preserving visually grounded textual states.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W3. [The throughput comparisons in Table 3 use the maximum batch size supported by each method. This demonstrates throughput gains from memory savings, but results at the same batch size are also needed to assess the latency impact of the eviction computation itself.]

A: Thank you. To fully evaluate the efficiency of ViFlow-KV, we conduct comparisons with FullKV and R-KV under two configurations from our original evaluation: (1) 2K visual input, 512 text input, and 8K generation (batch size 42); and (2) 4K visual input, 512 text input, and 16K generation (batch size 21). Both batch sizes correspond to the maximum supported by FullKV. We report end-to-end latency, throughput, and peak memory usage, including fine-grained KV eviction overhead. The results are as shown below:

2K Visual Input, 8K Generation

MethodKV BudgetLatency (s)Throughput (tok/s)Memory (GiB)
FullKVFull316.501087.177.68
R-KV512155.102218.419.71
ViFlow-KV512158.242174.321.16
R-KV2048222.311547.731.90
ViFlow-KV2048230.361493.633.42

4K Visual Input, 16K Generation

MethodKV BudgetLatency (s)Throughput (tok/s)Memory (GiB)
FullKVFull597.85575.576.39
R-KV512259.541325.717.92
ViFlow-KV512262.891308.818.43
R-KV2048332.911033.523.66
ViFlow-KV2048341.411007.824.42

The results show that ViFlow-KV substantially reduces end-to-end latency and peak memory usage compared with FullKV under matched batch sizes. Compared with R-KV, ViFlow-KV achieves a more favorable accuracy–efficiency trade-off with substantially better reasoning performance and comparable inference speed. Furthermore, we break down the latency of its individual components, including query caching, EMDA, anchor selection, importance computation, and cache reorganization, under the 4K visual input, 16K generation, and 512-token KV budget setting. The detailed measurements are reported below:

Query Caching (ms)EMDA (ms)Anchor Selection (ms)Importance Computation (ms)Cache Reorganization (ms)
33.6417.5010.9011.769.24

Overall, these results demonstrate that ViFlow-KV retains practical efficiency advantages under matched batch size.

Q1. [At the same batch size, how do task latency and peak memory usage compare with FullKV and R-KV? Do these measurements include the overhead of query caching, importance computation, and cache reorganization?]

A: Thanks. We have explained it in our response to Weakness 3.

Q2. [With the same cache budget and number of anchors, how do a fixed modality ratio, random anchors, and uniformly selected anchors compare with the current design?]

A: Thanks. We have explained it in our response to Weakness 2.

AuthorsOpenReview ↗

Looking Forward to Your Reply

Thank you once again for taking the time to review our paper and for providing such insightful and constructive feedback.

We have carefully considered each of your comments and have provided detailed responses. We sincerely hope that our efforts adequately address your concerns and contribute positively to your evaluation.

As the author-reviewer discussion period concludes on Nov 18 (AoE), we would greatly appreciate any further feedback you may have. If you have any additional questions or require any clarifications, please do not hesitate to reach out to us.

Reviewer_gEuxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies KV cache compression for long-horizon multimodal reasoning. Through attention analyses, the authors characterize “visual flow,” whereby visual evidence is accessed through both original visual states and historically generated text that serves as additional carriers. Building on R-KV, they propose ViFlow-KV, which combines Evolving Modality-Demand Allocation (EMDA) to dynamically allocate visual and textual cache budgets with Visual-Anchor Importance Estimation (VAIE) to improve historical state scoring. Evaluation across six multimodal benchmarks, multiple models, and cache budgets, together with component ablations and throughput measurements, shows improvements over the selected baselines. At a 2K-token budget, the method recovers 98.36% of FullKV performance on average and achieves up to a 3.05× throughput improvement in the corresponding efficiency setting.

Critical strengths
  1. The paper addresses a practical bottleneck with a simple, training-free approach. It jointly handles KV cache pressure from dense visual inputs and continuously growing reasoning traces by combining modality-specific budget allocation with state importance estimation. This setting better reflects the deployment needs of long-horizon multimodal reasoning than input-only visual token compression, while avoiding additional training lowers the adoption cost.

  2. The accuracy improvements are consistent across the evaluated settings. The method outperforms the selected baselines across six benchmarks and four cache budgets, with gains extending to additional

Critical weaknesses
  1. The ablations do not isolate the contributions of the core design choices. EMDA simultaneously introduces modality-specific budget partitioning, dynamic allocation, and pre-RoPE query aggregation, but the paper only ablates the entire module. VAIE likewise changes the query selection criterion while expanding historical query coverage, without controls using randomly or uniformly selected queries or a matched query count. Thus, the results establish that the modules are effective as a whole, but do not demonstrate that the gains specifically arise from evolving modality demand and visual-anchor selection. This limits the assessment of the technical contributions.

  2. The baseline coverage is insufficient to establish competitiveness against recent strong methods. The main experiments compare only against SnapKV, R-KV, and Ada-RKV, omitting recent long-reasoning compression methods with potentially compatible settings. TriAttention [1] is particularly relevant: it also performs window-based eviction under a fixed cache budget, and its pre-RoPE analysis motivates part of the proposed design. Given differences in model architectures and positional encoding, the authors should provide an appropriately adapted comparison or explain concrete compatibility barriers. The current results establish improvements over the selected baselines but do not support broader claims of superiority.

  3. The novelty relative to closely related work is insufficiently delineated. VisionPulse [2] already studies how visual demand changes across reasoning steps and uses visual attention mass to dynamically adjust visual-token retention budgets. These observations alone therefore do not establish novelty. Although its step-wise visual selection differs from permanent KV eviction, making a direct comparison under identical cache budgets potentially inappropriate, the paper should clarify the conceptual and methodological distinctions and identify the independent contribution of jointly managing visual and historical textual states. The current presentation of “visual flow” does not sufficiently distinguish prior observations from new findings, making the degree of novelty difficult to assess.

References:

[1] Weian Mao, Xi Lin, Wei Huang, Yuxin Xie, Tianfu Fu, Bohan Zhuang, Song Han, and Yukang Chen. TriAttention: Efficient Long Reasoning with Trigonometric KV Compression. arXiv:2604.04921, 2026.

[2] Hengbo Xu, Shengjie Jin, Yanbiao Ma, and Zhiwu Lu. VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning. ICML, 2026.

Questions for authors

Please refer to weaknesses.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Thank you for your insightful comments and questions.

W1. [The ablations do not isolate the contributions of the core design choices.]

A: Thank you for pointing this out. We would further separately elaborate these design choices.

Evolving Modality Demand Allocation (EMDA)

To further isolate the contributions of EMDA's key design choices, we conduct fine-grained ablations on CharXiv RQ with a KV budget of 512. For dynamic allocation, we consider two static alternatives: (1) freezing the head-specific modality budget ratios after their initial estimation, and (2) assigning fixed visual ratios. For pre-RoPE query aggregation, we compare the centroid-based estimation with direct averaging of attention distributions from the same queries. Removing modality-specific budget partitioning means assigning a unified KV budget to visual and textual states, which yields the VAIE-only variant evaluated in our original ablation study. The results are shown below:

VariantAccuracy (%)Δ\Delta
ViFlow-KV40.20—
w/o pre-RoPE query aggregation38.00−2.20
Frozen after initial allocation33.50−6.70
Fixed visual ratio (5%)36.40−3.80
Fixed visual ratio (10%)36.40−3.80
Fixed visual ratio (25%)34.90−5.30
Fixed visual ratio (50%)30.20−10.00

The results validate both design choices. The degradation under static allocation demonstrates the importance of adapting KV budgets to evolving modality demands, while the advantage of pre-RoPE query aggregation supports its effectiveness in estimating current modality demands. This ablation also provides support for the visual-flow identified in our attention analysis.

Visual-Anchor Importance Estimation (VAIE)

To further examine the effectiveness of visual-anchor selection, we replace the selected visual anchors with an equal number of alternative anchors using two strategies: (1) random sampling and (2) uniform sampling from the newly generated reasoning window. The results are shown below:

VariantAccuracy (%)Δ\Delta
ViFlow-KV40.20—
w/o VAIE (EMDA only)39.40−0.80
Random anchor selection38.30−1.90
Uniform anchor selection38.60−1.60

The results show that visual-anchor selection consistently outperforms both random and uniform sampling, indicating that its effectiveness stems from identifying visually relevant reasoning states rather than merely expanding query coverage.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W2. [The baseline coverage is insufficient to establish competitiveness against recent strong methods. ]

A: Thank you for the question. TriAttention aims to achieve Textual KV compression by exploiting the high concentration of pre-RoPE Q/K representations, which provides valuable insights that motivate our method design. However, multimodal reasoning introduces additional challenges to this assumption, as visual and textual keys exhibit markedly different distributions, as illustrated in Appendix A.1. To further assess its effectiveness in multimodal reasoning, we additionally evaluate TriAttention under the same KV cache budgets on three benchmarks. The results are shown below:

BudgetMethodMMStarMathVistaCharXiv RQAvg.
512TriAttention59.6758.4032.3050.12
ViFlow-KV63.9366.6040.2056.91
2048TriAttention70.4775.8049.5065.26
ViFlow-KV71.4778.0053.3067.59

The results show that ViFlow-KV consistently outperforms TriAttention across all evaluated benchmarks and cache budgets, demonstrating its effectiveness in multimodal reasoning. We will include these comparisons in the revision.

W3. [The novelty relative to closely related work is insufficiently delineated.]

A: Thank you. VisionPulse provides valuable insights into multimodal reasoning by revealing how generated tokens attend to original visual tokens varies throughout decoding (visual → text). Specifically, it introduces a visual-specific sparse attention that dynamically adjusts visual-token retention. However, our work reveals how visual evidence from original visual states is carried by earlier generated textual states and subsequently reactivated during later reasoning (visual → earlier text → later text), guiding joint visual–textual KV management under constrained memory. We clarify the distinctions below in terms of analytical findings and methodological design.

Analytical Findings

Our fine-grained analysis reveals two important findings: (1) textual states expressing the same visual evidence exhibit stronger key representation similarity to the corresponding visual states (Figure 3a); and (2) when visual evidence is reactivated, its matched historical textual states receive substantially higher attention than original visual states and unrelated states (Figure 3b). Together, these findings distinguish our analysis from VisionPulse by revealing how visual evidence is carried forward and revisited through historical textual states.

Method Design

Our analysis reveals that both modality demands and the carriers of visual evidence evolve throughout reasoning. Accordingly, ViFlow-KV combines evolving modality-demand allocation with visual-flow-guided importance estimation to identify and retain critical visual and visually grounded textual states. Unlike VisionPulse's step-wise visual-token selection for reducing attention computation, ViFlow-KV performs persistent KV eviction, jointly managing original visual states and historical textual carriers to reduce KV memory consumption while accounting for accurate multimodal reasoning.

AuthorsOpenReview ↗

Looking Forward to Your Reply

Thank you once again for taking the time to review our paper and for providing such insightful and constructive feedback.

We have carefully considered each of your comments and have provided detailed responses. We sincerely hope that our efforts adequately address your concerns and contribute positively to your evaluation.

As the author-reviewer discussion period concludes on Nov 18 (AoE), we would greatly appreciate any further feedback you may have. If you have any additional questions or require any clarifications, please do not hesitate to reach out to us.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.