acceptodds
Withdrawn from ICLR 2027

StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstructing dense 3D geometry from continuous video is severely bottlenecked by the unbounded growth of streaming KV caches. We present StreamCacheVGGT, a constant-budget visual geometry transformer equipped with two training-free modules. Cross-Layer Consistency-Enhanced Scoring (CLCES) tracks rank stability across Transformer layers to yield robust token importance scores and uncertainty estimates. Uncertainty-Gated Deferred Compaction (UGDC) then routes candidates through either immediate retain, merge, and evict compaction or a small deferred pool for short-term verification. This deferred pool operates strictly within the fixed per-layer budget (), ensuring the persistent memory footprint remains completely unchanged. Across diverse indoor, outdoor, long-sequence, and depth benchmarks, our method significantly improves geometric completeness and long-horizon stability while transparently reporting the expected throughput trade-off of deferred verification.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
streaming 3d reconstuction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6784OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes StreamCacheVGGT, a training-free cache-management framework for streaming visual geometry transformers. Cross-Layer Consistency-Enhanced Scoring refines FFN-residual importance scores using rank stability across layers. Uncertainty-Gated Deferred Compaction then combines immediate retention, merging, and eviction with a short verification pool inside a fixed cache budget. The paper reports reconstruction, pose, depth, and long-sequence results, together with throughput and memory measurements.

Critical strengths
  • The motivation is practical. Maintaining useful geometric context under a bounded streaming KV cache is important for long-sequence reconstruction.
  • The method combines inexpensive importance proxies with a concrete attempt to avoid irreversible decisions on uncertain tokens, while remaining compatible with FlashAttention.
  • The evaluation covers several indoor, outdoor, depth, and long-sequence benchmarks. Providing controlled same-backbone results and acknowledging the accuracy-throughput trade-off and dynamic-scene limitations are useful.
Critical weaknesses
  • My biggest concern is that the contribution of deferred verification is not isolated. Table 7 compares eviction, CLCES, and CLCES+UGDC, but UGDC introduces both immediate token merging and the deferred branch. Therefore, its improvements cannot be attributed specifically to uncertainty-gated verification. A CLCES+immediate-merge variant without deferral, followed by a matched-budget deferred variant, is necessary. Random or recency-based deferral with the same reserved capacity would also test whether cross-layer uncertainty is informative.
  • The figures in the paper are not informative. The figures are mostly just boxes with numbers, making them hard to read and get meaningful information. They should be at least polished for publication.
  • The deferred branch is underspecified despite being central to the method. Appendix C gives H=2 and a recurrence threshold, but does not define the descriptor, deferred capacity, pool-overflow policy, motion-residual threshold, or dynamic-token TTL. It is also unclear whether deferred tokens participate in attention and how their layer-specific keys and values are preserved or reconstructed when a descriptor is promoted. These details determine both the online algorithm and whether its claimed memory accounting is valid.
  • Several result tables and their explanations need to be reconciled. Table 7 shows that CLCES alone worsens 7-Scenes Accuracy from 0.0259 to 0.0306, whereas Table 11 reports an improvement from approximately 0.026 to 0.021 at the stated default lambda and window. Appendix E says the complete method is best on continuous KITTI, but Table 10 gives lower errors for CLCES alone at both 300 and 500 frames. Table 8 also shows substantial long-sequence completeness regressions, for example 0.479 versus 0.057 for InfiniteVGGT on Classroom. These results may use different protocols, but the distinctions and the resulting limits on the claims need to be explicit.
  • The assumptions behind scoring and merging need stronger empirical support. Rank dispersion is a heuristic uncertainty measure, rather than a demonstrated predictor of harmful eviction. Similarly, Appendix B's minimum-variance interpretation assumes that merged keys observe one underlying feature and that FFN scores are inverse noise variances, neither of which is established. Cosine-nearest key fusion also does not generally preserve softmax attention or token multiplicity. A comparison with simple merging, measurements of attention or geometry error, and stress tests under motion and viewpoint changes would clarify when these approximations are safe.
Questions for authors

Please see above weaknesses.

Reviewer_VNTPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces StreamCacheVGGT, a constant-budget visual geometry transformer equipped with two training-free modules. StreamCacheVGGT achieves memory-efficient streaming 3D reconstruction through two training-free modules: CLCES and UGDC. CLCES module tracks the consistency of token rankings within a sliding window. It calculates the standard deviation and assign higher score bonus to stable tokens. The novelty lies in tracking the stability of token importance rankings across layers, rather than relying on single-layer FFN residuals. UGDC module divides cache compression into two branches: the immediate branch and the deferred branch. The immediate branch makes a three-level decision including retain, merge, or evict for confident candidates. The deferred branch delays verification by two frames for uncertain boundary tokens.

Critical strengths

This paper solves a practical problem. Bounded inference state matters for continuous geometry estimation. The paper addresses both which information to preserve and when to make an irreversible compression decision.

Single-layer FFN residuals are noisy. CLCES tracks rank stability across 5 layers to filter transient spikes, improving Long3D accuracy from 0.108 to 0.090 without breaking FlashAttention compatibility.

StreamCacheVGGT achieves byte-exactness with constant memory usage while avoiding information loss caused by hard eviction through a merging mechanism. This saves memory without sacrificing information.

Critical weaknesses

The novelty of CLCES appears incremental. It reweights existing FFN-based importance scores using cross-layer rank variance, which I think is quite standard and lacking in originality. The assumption that stable rankings indicate geometric usefulness is not sufficiently validated, and comparisons against simple score averaging or rank aggregation are missing.

The merging strategy largely combines off-the-shelf modules from token-merging methods such as ToMe. Thus the novelty appears limited. Also, the manuscript does not sufficiently justify why key similarity and score-weighted KV fusion should preserve geometric information.

I found Table7 combines immediate merging and deferred verification inside UGDC. Consequently, its improvement could arise entirely from merging. A configuration with CLCES and immediate retain/merge/evict but no deferred pool is essential. The paper also lacks UGDC without CLCES, so the interaction between the two components is unresolved.

StreamCacheVGGT performs very poorly in dynamic scenes. The Merge mechanism averages out rapidly moving objects, resulting in excessive smoothing. The paper claims to be a general-purpose streaming solution, but it seems only suitable for static scenarios and barely works for dynamic scenes.

Questions for authors

Table 8 reports noticable worse mean completeness than InfiniteVGGT. Classroom completeness error is approximately 8.4 times larger. How is the claimed improvement in long-horizon completeness supported?

Because I didn't find ablation separates its effect from conventional token merging under the same memory budget, how can the gains attributed to UGDC be credited to deferred verification?

Table 7 shows UGDC provides negligible improvement on KITTI (0.1745 -->0.1748), while achieving significant gains on static indoor scenes. What movement speed do the Token Merge and Deferred Verification mechanisms fail and the merge mechanism transitions from beneficial to harmful?

Reviewer_63AYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StreamCacheVGGT proposes two training-free modules for bounded-memory streaming 3D reconstruction: CLCES refines token importance using cross-layer rank stability, while UGDC combines immediate retention/merging/eviction with two-frame verification of uncertain candidates within the existing cache budget.

Critical strengths
  1. Addresses the practical challenge of KV-cache growth in long-video reconstruction.
  2. Requires no additional training and remains compatible with FlashAttention.
  3. Reports improvements on selected reconstruction benchmarks and approximately stable throughput over the tested sequence lengths.
Critical weaknesses
  1. The central contribution is not clear. UGDC introduces both token merging and deferred verification, but the ablations do not separate their effects. A merging-only baseline and a simple ungated deferral baseline are necessary to establish the value of uncertainty-gated verification.
  2. Experimental reporting contains unresolved inconsistencies. CLCES worsens 7-Scenes Accuracy from 0.0259 to 0.0306 in Table 7, whereas Table 11 reports improvement from 0.026 to 0.021 at the default settings, without clearly explaining the configuration differences. Appendix E also claims that the full method is best on continuous KITTI, although Table 10 favors CLCES alone.
  3. Essential implementation details are missing. Descriptor construction, promotion back into the KV cache, deferred-pool allocation, and overflow handling are insufficiently specified. The budget equation alone does not substantiate byte-exact memory equivalence when auxiliary storage is included.
  4. Benefits are task-dependent. The full method underperforms eviction-only on Bonn and Sintel depth in Table 10, limiting support for broad robustness claims.
Questions for authors

please refer to the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.