acceptodds
Under review as a conference paper at ICLR 2027

S2Tok: Streaming 3D Gaussian Reconstruction with Persistent Spatial Tokens

Abstract

Streaming 3D reconstruction requires more than a sequence of geometric predictions: it requires a persistent scene state that can incorporate new evidence and remain renderable as observations arrive. Latent spatial tokens offer a promising representation for this purpose, but constructing them from an image collection leaves open how to maintain them online, where each observation may both revisit known regions and reveal new content. We introduce S2Tok, a feed-forward framework that maintains a size-adaptive, persistent scene state from uncalibrated image streams. Its central idea is to distinguish updates to the existing representation from selective expansion. A spatially informed transformer integrates each incoming observation with the persistent scene tokens, while a learned admission module selectively expands the representation to limit redundant storage. A hierarchical decoder and Gaussian head convert the evolving state into non-pixel-aligned 3D Gaussians, enabling novel-view rendering without caching previous frames. Experiments across four benchmarks demonstrate competitive streaming rendering quality with compact Gaussian representations. These results support latent spatial tokens as a persistent computational state for online 3D reconstruction, combining learned scene updates with explicit Gaussian rendering.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D reconstruction, 3D gaussian splatting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WTiMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes S2Tok for online Gaussian reconstruction from uncalibrated image streams. It maintains persistent scene tokens and learns when to add new tokens using geometric novelty supervision and self-critical sequence training (SCST). The current tokens are jointly decoded into a Gaussian scene. Experiments on four datasets report competitive rendering quality with substantially fewer Gaussians.

Critical strengths
  1. The persistent token representation supports online reconstruction with compact Gaussian outputs. On 50-view DL3DV, the method uses 66K Gaussians at 81 ms per frame.

  2. The experiments cover four datasets, including cross-dataset evaluation on Mip-NeRF 360 and ScanNet++. The method achieves higher PSNR than the evaluated feed-forward streaming baselines on DL3DV, RealEstate10K, and Mip-NeRF 360.

Critical weaknesses
  1. The admission objective does not directly optimize the quality–memory tradeoff. The SCST reward measures reconstruction error, while the KL term keeps the policy close to a geometric novelty oracle (Eqs. 29–32). Neither term explicitly accounts for token count or update cost. Compactness is therefore controlled indirectly by the oracle and admission threshold. The benefit of this policy optimization over oracle distillation remains limited: at the same 66K Gaussian budget, SCST improves Final/Intermediate PSNR by only 0.10/0.08 dB (Table 3).

  2. The state update remains expensive as the scene grows. Eq. 4 only appends tokens; admitted tokens are never merged or removed. Meanwhile, global attention, scene attention, and the hierarchical decoder operate on the accumulated state (Eqs. 1, 2, and 6), with quadratic attention computation in the number of scene tokens. Selective admission slows growth but cannot reduce an already expanded state, so per-frame computation increases with the accumulated token count.

  3. The frozen appearance memory is insufficiently justified. Appendix D.2 fixes each token's color feature at admission, while its scene feature and decoded Gaussians continue to change. Later observations can refine appearance through the main feature path, but cannot update the high-frequency color bank. The paper does not explain how this asymmetry affects reconstruction across updates. In the 50-view setting, Final PSNR drops from 19.11 to 18.00 relative to Intermediate on DL3DV, and from 22.63 to 20.54 on ScanNet++ (Tables 1 and 7); the contribution of the color-memory design to this degradation is unresolved.

  4. The comparison with streaming Gaussian reconstruction is incomplete. StreamGS (Li et al., 2025) is discussed but not evaluated. It directly addresses online Gaussian reconstruction and redundancy reduction, making it an important baseline for the claimed quality–efficiency tradeoff. Comparisons with pointmap predictors through point splatting do not establish this advantage.

Questions for authors
  1. Could the authors clarify how the reconstruction reward and oracle regularization balance quality against state size?
  2. Could the authors characterize how per-frame runtime and memory scale with the number of retained tokens?
  3. Could the authors explain the rationale for freezing the color bank while updating scene tokens?
  4. Could the authors provide a comparison with StreamGS (Li et al., 2025) under the same protoco, reporting rendering quality, Gaussian count, and reconstruction time?
Reviewer_UwVxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a pose-free 3D Gaussian Splatting reconstruction method for streaming inputs, enabling incremental construction and optimization of 3D scene representations without requiring pre-computed camera poses. The paper aims to jointly address the challenges of streaming reconstruction, camera pose dependency, and redundancy in Gaussian representations.

Critical strengths

S1. The paper explores the integration of streaming and pose-free reconstruction, which has potential practical value.

Traditional 3D reconstruction pipelines often rely on pre-computed camera poses or access to complete image sequences. By enabling pose-free reconstruction from streaming inputs, this work explores a potentially useful setting with practical applications.

S2. The paper considers the compactness of Gaussian representations during incremental reconstruction, with an emphasis on efficiency.

As new frames are continuously introduced, controlling the number of Gaussians and preventing unbounded growth of the scene representation is an important problem. The proposed approach attempts to manage the size of the Gaussian representation through iterative updates, which is a practically relevant design consideration.

Critical weaknesses

W1. The application scope is relatively limited, and the practical motivation for streaming reconstruction is not sufficiently justified.

The paper primarily focuses on streaming reconstruction of static scenes. However, for entirely static environments, existing offline reconstruction methods already provide relatively mature solutions. The paper does not adequately explain why streaming reconstruction is particularly necessary in this setting, or what distinctive advantages the proposed method offers over existing reconstruction pipelines.

In particular, the current experiments do not sufficiently demonstrate the benefits of streaming reconstruction in terms of adaptability to dynamic environments, online responsiveness, or other practical deployment requirements. This raises concerns about the practical significance of the proposed approach.

W2. The visual quality of the reconstructed scenes is not entirely satisfactory.

Based on the qualitative results presented in the paper, some reconstructed scenes exhibit noticeable blurring, with insufficient recovery of fine-grained textures and details. These artifacts may limit the practical usability of the proposed method.

Given that high-quality scene representation is one of the primary objectives of 3D reconstruction, the paper would benefit from a more thorough analysis of the causes of these artifacts, as well as a clearer investigation of the trade-off between reconstruction quality and streaming efficiency.

W3. The technical novelty appears limited, and the distinctions from existing methods are not sufficiently clear.

Several key technical components explored in this paper have already been extensively investigated in prior work. For example:

  • Iterative Gaussian updates and density control: Related techniques have been explored in ReSplat, G3R, Gaussian Lens, Generative Densification, and other works.
  • Pose-free reconstruction: Representative approaches include NoPoSplat, VGGT, and D4RT.
  • Streaming 3D reconstruction: Related work such as StreamVGGT has already investigated this direction.

From the current methodological design, the proposed approach appears to be largely an integration of existing techniques, rather than introducing fundamentally new algorithmic insights or clearly differentiated technical mechanisms.

The paper should more explicitly clarify how each component differs from prior work and provide more targeted ablation studies to demonstrate whether their integration offers benefits beyond a straightforward combination of existing approaches. At present, the provided evidence is insufficient to support a strong claim of methodological novelty.

Questions for authors

Q1. For entirely static scenes, what are the key advantages of the proposed streaming reconstruction approach over existing incremental or offline reconstruction methods? Could the authors provide more concrete application scenarios or experimental evidence to justify the necessity of streaming reconstruction in this setting?

Q2. Compared with existing methods such as ReSplat, G3R, Gaussian Lens, Generative Dens., and StreamVGGT, what constitutes the core algorithmic novelty of this work? Could the authors provide more targeted ablation studies to demonstrate that the proposed method goes beyond a straightforward combination of existing techniques?

Q3. Some of the qualitative reconstruction results exhibit noticeable blurring artifacts. What do the authors consider to be the primary causes of these artifacts? Are they related to the streaming update mechanism or inaccuracies in camera pose estimation?

Q4. Could the authors provide additional comparisons between the proposed Gaussian management strategy and existing pruning or density control methods, particularly under matched reconstruction quality or comparable computational and memory budgets?

Q5. Could the authors provide visualizations of the incremental reconstruction process to better illustrate the behavior of the proposed method? For instance, it would be informative to distinguish between Gaussians that are updated and those that remain unchanged at each streaming step.

Reviewer_vKqwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes S2Tok, a feed-forward method for streaming 3D Gaussian reconstruction from uncalibrated images. It maintains a set of spatial tokens that are updated as new frames arrive, and adds new tokens when novel scene content is detected. The method uses a Stream-Spatial Transformer to update the scene representation and a learned admission module to control token growth. The tokens are decoded into 3D Gaussians for rendering. Experiments on four datasets show competitive reconstruction quality compared with existing streaming methods, while using fewer Gaussians.

Critical strengths
  1. Maintaining a persistent set of spatial tokens is an interesting approach for streaming reconstruction, and avoids storing previous frames or KV caches.

  2. The method uses much fewer Gaussians than existing streaming methods while achieving good rendering quality and reasonable inference speed.

  3. The paper includes useful ablations on the admission module, SCST, and the attention design.

Critical weaknesses
  1. The reconstruction quality is still considerably worse than non-streaming methods. S2Tok is initialized from ZipSplat, but performs 1–3 dB worse than ZipSplat with the same token budget. Most of the improvements are over geometry-based streaming methods such as CUT3R and Point3R, which are not primarily designed for novel-view synthesis. The results against StreamSplat are also mixed, especially on ScanNet++.

  2. The persistent state does not seem to preserve reconstruction quality well as more frames are processed. The Final PSNR is consistently lower than the Intermediate PSNR, suggesting that updating the scene tokens can degrade previously reconstructed regions. Also, removing the admission module actually improves Intermediate PSNR.

  3. Some important details about evaluation and scalability are missing. The method claims to work without camera poses, but it is not clear how the target cameras are obtained or aligned during evaluation. The experiments also use relatively few test scenes, and the model is only evaluated on streams up to 50 frames. Since the number of tokens keeps growing, the memory and runtime for longer sequences could be a problem. The training also requires substantial compute, but the model size and training cost are not clearly compared with the baselines.

Questions for authors
  1. How are the target camera poses and scales handled during evaluation? Are GT poses or any test-time alignment used, and is the same procedure applied to all baselines?

  2. Since S2Tok performs noticeably worse than ZipSplat at the same token budget, could the authors compare against a simple streaming version of ZipSplat, such as reconstructing sliding windows and merging the Gaussians?

  3. Why does the reconstruction quality decrease from Intermediate to Final evaluation? Have the authors tested longer sequences, such as 200 or 500 frames, to see how reconstruction quality and memory usage change as more frames are processed?

1 comment

Thanks for the response. I will keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.