acceptodds
Under review as a conference paper at ICLR 2027

DiaStream: Distribution-Aware KV Cache for Streaming 3D Reconstruction

Abstract

In streaming visual geometry transformers that maintain a key-value (KV) cache, effectively preserving scene context under a fixed cache budget remains challenging. Existing methods consider token importance and key-space diversity independently, overlooking the underlying distribution of cached keys and limiting their ability to retain tokens that are both informative and non-redundant. We observe that cached keys exhibit multi-directional structured redundancy and that their statistical characteristics vary substantially across transformer layers, suggesting that token retention and layer-wise cache allocation should be jointly considered. Based on this insight, we propose , a training-free distribution-aware KV cache management framework. combines Important-Distinctive Caching (IDC), which preserves tokens that are both distinctive and reconstruction-relevant by using statistical leverage and attention-based importance, with Key Statistics-Aware Budget Allocation (KABA), which dynamically allocates the global cache budget according to layer-wise key statistics. Experiments on multiple streaming 3D reconstruction benchmarks show that outperforms existing memory-bounded methods in 3D reconstruction and camera pose estimation while requiring at most 30% of the KV cache budget used by previous methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Vision, Streaming Visual Geometry, KV Cache Management

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GL6pOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

DiaStream is a training-free, distribution-aware KV-cache management method that improves streaming 3D reconstruction under limited memory by jointly considering token distinctiveness, attention-based importance, and layer-wise differences.

Critical strengths

The research problem addressed in this paper, improving memory efficiency and scalability in streaming 3D reconstruction, is important and highly relevant to the field.

Critical weaknesses
  1. This paper appears to combine existing cache-management techniques in a more elaborate manner with only limited incremental improvements. It is therefore difficult for readers to identify clear methodological novelty over other related works.
  2. Both the main experiments and the ablations evaluate long sequences using relatively small sampling intervals. However, the paper does not systematically examine whether the method remains effective and generalizable under larger intervals. Without such evaluation, the practical value of this type of efficiency-oriented method remains insufficiently demonstrated.
Questions for authors
  1. Please check the weaknesses.
  2. Please provide reconstruction comparison at larger frame intervals (20, 50, 100).
Reviewer_nYZyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

DiaStream proposes a training-free KV-cache management framework for streaming 3D reconstruction. Important-Distinctive Caching (IDC) selects tokens using leverage-based distinctiveness and temporally aggregated attention scores, while Key Statistics-Aware Budget Allocation (KABA) distributes cache capacity across transformer layers. Implemented on StreamVGGT, the method improves reconstruction and full-sequence camera-pose results over the evaluated bounded-cache baselines at substantially smaller configured token budgets.

Critical strengths

The paper addresses a practical obstacle to streaming visual geometry: preserving useful historical information while limiting cache growth. This approach requires no additional policy or model training. Reconstruction results are strong across the evaluated bounded-cache baselines. The improvements extend to full-sequence camera-pose estimation, and Table 3 includes details of comparisons at matched configured budgets. The component ablations support combining distinctiveness and importance scores with adaptive allocation. Additional experiments are extensive, and examine random-projection seeds, temporal sampling strides, allocation sharpness, and outdoor depth estimation.

Critical weaknesses
  1. Configured budgets do not necessarily equal actual cache sizes. Equation 2 imposes a global budget, but Section 3.4 and Appendix A.1 explicitly allow that budget to be exceeded when preserving anchor tokens. The magnitude and frequency of this excess are not reported. Since reduced cache requirements are central to the contribution, actual retained token counts and peak memory usage are needed to interpret both the headline savings and the matched-budget comparisons. This does not invalidate bounded-memory operation, but it leaves the precise resource advantage uncertain.
  2. The methodological novelty could use sharper positioning. Compactor already combines approximate leverage scores with attention-based scores for KV retention, as the submission acknowledges. IDC's contribution therefore rests on its adaptation to streaming visual geometry and its temporal scoring choices. The existing ablations demonstrate component utility, but do not fully isolate the gains attributable to these changes over the earlier formulation.
  3. The connection between KABA's statistics and required cache capacity is insufficiently established. Leverage participation measures how evenly leverage is distributed across tokens, which is not necessarily equivalent to information diversity. For example, identical keys can have uniform leverage and therefore maximal participation despite being redundant. Raw key magnitude is also a heuristic for representation demand. The ablations support the combined allocation empirically, but stronger justification or more qualified interpretation would help explain why these statistics should determine capacity.
  4. Broader positioning against online tracking systems should be more comprehensive. The paper includes relevant cache-management baselines, but omits KV-Tracker and AMB3R-VO. These are relevant to its broader online geometry and camera-tracking claims, although their architectures and memory assumptions differ. KV-Tracker's memory grows with keyframes, so it does not directly solve the same fixed-budget problem; nevertheless, its relationship to the proposed approach merits discussion. Comparisons with chunk-based systems should likewise distinguish output latency and processing assumptions.
  5. Detail on a claim: At one point in the paper, authors claim "...chunk-wise inference (Deng et al., 2026; Xie et al., 2026), or SLAM-style sequential processing (Maggio et al., 2025). While these approaches substantially improve scalability, their reliance on global attention over all input views still limits efficient processing" - worth noting that chunk-wise inference requires global attention over a chunk; not over an entire sequence. It is a case of bounded attention.
Questions for authors
  1. How often and by how much does anchor preservation exceed the configured budget? Do the improvements persist under strictly enforced, matched total cache sizes?
  2. Which changes beyond Compactor's leverage-plus-attention formulation account for IDC's improvements?
  3. What evidence connects leverage participation and raw key magnitude to a layer's required cache capacity? (Presentation: The qualitative figures and their enlarged regions should be more legible. Algorithm 2's importance-update condition also appears inconsistent with the explanation that current-frame tokens initially lack attention-based importance; please clarify the update order.)
Reviewer_FwtPOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  1. The paper addresses a practically important problem and reports substantial full-sequence accuracy improvements. Limiting the KV cache while preserving long-term scene information is an important challenge for deploying streaming geometry models. In Table 1, compared with OVGGT, DiaStream reduces full-sequence CD on 7-Scenes from 0.028 to 0.020 and increases F1 from 0.867 to 0.923. On full-sequence NRGBD, CD decreases from 0.042 to 0.026. In Table 2, it achieves the lowest full-sequence ATE among the compared bounded-cache methods on 7-Scenes, Replica, and TUM. These results suggest meaningful practical potential.

  2. The evaluation has good breadth and includes useful evidence on comparison fairness and robustness. The experiments cover geometric reconstruction, camera trajectories, and outdoor depth estimation. Table 3a provides comparisons at a matched nominal budget of 60K, as well as results for DiaStream at 200K and 1.2M, supporting the view that its gains cannot be explained solely by differences in default budgets. The five random seeds in Table 4b and the temporal-stride experiments in Table 6 also strengthen confidence in robustness. These existing experiments deserve credit; the remaining concerns primarily involve actual resource accounting and attribution of the gains to specific mechanisms.

  3. The method has a clear structure and can be integrated without retraining. Using second-order key-distribution structure to complement attention-based usage statistics is a reasonable design. IDC and KABA address token selection and layer-wise allocation, respectively, making the overall framework easy to understand. Under uniform allocation in Table 3b, combining the two IDC scores substantially improves performance over either score alone, while the complete KABA further improves the results. These experiments provide initial evidence that the components are complementary.

Critical strengths
  1. The paper addresses a practically important problem and reports substantial full-sequence accuracy improvements. Limiting the KV cache while preserving long-term scene information is an important challenge for deploying streaming geometry models. In Table 1, compared with OVGGT, DiaStream reduces full-sequence CD on 7-Scenes from 0.028 to 0.020 and increases F1 from 0.867 to 0.923. On full-sequence NRGBD, CD decreases from 0.042 to 0.026. In Table 2, it achieves the lowest full-sequence ATE among the compared bounded-cache methods on 7-Scenes, Replica, and TUM. These results suggest meaningful practical potential.

  2. The evaluation has good breadth and includes useful evidence on comparison fairness and robustness. The experiments cover geometric reconstruction, camera trajectories, and outdoor depth estimation. Table 3a provides comparisons at a matched nominal budget of 60K, as well as results for DiaStream at 200K and 1.2M, supporting the view that its gains cannot be explained solely by differences in default budgets. The five random seeds in Table 4b and the temporal-stride experiments in Table 6 also strengthen confidence in robustness. These existing experiments deserve credit; the remaining concerns primarily involve actual resource accounting and attribution of the gains to specific mechanisms.

  3. The method has a clear structure and can be integrated without retraining. Using second-order key-distribution structure to complement attention-based usage statistics is a reasonable design. IDC and KABA address token selection and layer-wise allocation, respectively, making the overall framework easy to understand. Under uniform allocation in Table 3b, combining the two IDC scores substantially improves performance over either score alone, while the complete KABA further improves the results. These experiments provide initial evidence that the components are complementary.

Critical weaknesses
  1. The implementation can exceed the global budget, but actual occupancy is not quantified, weakening verification of the central cache-saving claim. Equation (2) defines the problem using a strict global budget B. However, Equation (10) and Algorithm 3 first distribute B according to allocation weights and then raise any layer budget below the anchor count to |A^l|, without deducting the corresponding capacity from other layers. Lines 314-315 in the main text and lines 708-710 in the appendix explicitly acknowledge that this can cause the actual number of retained KV tokens to exceed the configured B. Consequently, the reported 30K/60K budgets cannot directly be interpreted as strict upper bounds on actual cache occupancy at every step, and the ratio of 60K to 200K alone does not establish a 70% reduction in actual KV storage. This does not imply that the cache grows without bound as sequence length increases, but it affects the interpretation of strictly matched-budget comparisons and reported saving ratios. The authors should report the actual cache usage for each dataset.

  2. IDC's incremental contribution over existing leverage-attention fusion methods is not sufficiently isolated. Compactor (Chari & Van Durme, 2025) already uses Gaussian right-sketching to approximate leverage scores, combines them with attention scores, and retains KV tokens according to the blended scores. The submission cites this work in both the related work and method sections. DiaStream's specific differences include streaming causal attention histories, head-wise normalization followed by concatenation, freezing scores after a limited observation window, mean normalization, and special scoring for current-frame tokens. These adaptations may be valuable, but the single-score ablations in Table 3b do not establish their benefit over a direct adaptation of the existing blended scoring approach. A Compactor-style online adaptation should be evaluated using the same StreamVGGT backbone, actual cache budget, and causal information constraints, while separately controlling IDC and KABA.

  3. The connection between KABA's statistics and layer-wise representation demand lacks direct evidence. The participation ratio in Equation (8) measures how evenly leverage scores are distributed across tokens; it does not directly measure the number of key-space directions or the marginal benefit of additional cache capacity. For example, for n identical nonzero keys, each leverage score is 1/n, giving psi = n. A token set that uniformly covers multiple independent directions and also has uniform leverage scores can yield the same psi = n. Thus, this statistic alone cannot distinguish these two distributions despite their substantially different directional redundancy.

  4. The system-efficiency evaluation does not fully substantiate the claimed resource advantages for deployment. The paper reports short-sequence FPS but does not provide peak GPU memory, actual KV storage, full-sequence per-frame latency curves, or component runtimes. In Table 1, DiaStream achieves 10.83 FPS with a nominal 60K budget, slightly below OVGGT at 200K (11.65 FPS) and below STAC (13.78 FPS). Thus, a smaller token budget does not directly translate into higher throughput.

  5. The attention-score update pseudocode is inconsistent with the textual description. Algorithm 1 computes current-frame attention only after eviction. The appendix also states that new tokens have no attention history and therefore receive a score of 2*rho_hat. However, lines 6-8 of Algorithm 2 compute u_i when "i belongs to frame t," without clearly specifying how historical tokens are updated during their first T observations.

Questions for authors
  1. Under the 30K and 60K configurations, what are the actual retained KV token counts and KV storage in bytes for each dataset? How frequently, and by how much, does the anchor-floor adjustment exceed the configured budget? Do the reported accuracy advantages and cache-saving ratios hold when the global budget B is strictly enforced?

  2. How does IDC compare with a Compactor-style online adaptation using the same StreamVGGT backbone, actual cache budget, causal information constraints, and layer-allocation policy? Can this comparison be conducted with and without KABA to isolate IDC's incremental contribution?

  3. How do the authors justify using the participation ratio in Equation (8) to estimate layer-wise representation demand, given that both identical keys and keys uniformly covering multiple independent directions can yield psi = n? What evidence supports its ability to identify differences in layer-wise cache requirements despite this ambiguity?

  4. Can the authors report peak GPU memory, actual KV storage, full-sequence per-frame latency, and component runtimes alongside FPS? Does the reported throughput include all cache-management operations, and are the baselines evaluated using the same timing scope? These measurements would help quantify the resource trade-off given that DiaStream uses a smaller nominal cache but has lower reported FPS than OVGGT and STAC.

  5. Is the condition "i belongs to frame t" in lines 6-8 of Algorithm 2 correct, given that new tokens have no attention history before eviction? Please clarify when u_i is computed and how historical tokens are updated during their first T observations, and provide pseudocode consistent with the stated execution order.

1 comment

Thank you for the comprehensive and detailed responses. The additional experiments and clarifications have addressed my major concerns, particularly regarding actual cache occupancy, the contribution of IDC, and system efficiency. The authors have also provided useful clarifications on KABA and the implementation details. Overall, I am satisfied with the rebuttal and consider my concerns adequately addressed. I will maintain my original rating of 3 (Weak Accept).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.