DiaStream: Distribution-Aware KV Cache for Streaming 3D Reconstruction

In streaming visual geometry transformers that maintain a key-value (KV) cache, effectively preserving scene context under a fixed cache budget remains challenging. Existing methods consider token importance and key-space diversity independently, overlooking the underlying distribution of cached keys and limiting their ability to retain tokens that are both informative and non-redundant. We observe that cached keys exhibit multi-directional structured redundancy and that their statistical characteristics vary substantially across transformer layers, suggesting that token retention and layer-wise cache allocation should be jointly considered. Based on this insight, we propose $\text{DiaStream}$, a training-free distribution-aware KV cache management framework. $\text{DiaStream}$ combines Important-Distinctive Caching (IDC), which preserves tokens that are both distinctive and reconstruction-relevant by using statistical leverage and attention-based importance, with Key Statistics-Aware Budget Allocation (KABA), which dynamically allocates the global cache budget according to layer-wise key statistics. Experiments on multiple streaming 3D reconstruction benchmarks show that $\text{DiaStream}$ outperforms existing memory-bounded methods in 3D reconstruction and camera pose estimation while requiring at most 30% of the KV cache budget used by previous methods.