acceptodds
Withdrawn from ICLR 2027

PersistSplat: Streaming Feed-Forward Reconstruction with Persistent Gaussian Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent feed-forward 3D Gaussian Splatting (3DGS) strive to overcome the limitation of per-pixel Gaussian allocation to build more compact scene representations. However, most follow an offline paradigm that requires all views to be acquired in advance and jointly processed, thus unsuitable for streaming inputs. Existing online compression methods predict dense Gaussians for each frame and subsequently compress redundancy, which severely limits reconstruction quality and efficiency. We propose PersistSplat, a streaming feed-forward 3DGS architecture for constructing compact scene representations. Its core idea is to leverage incoming views continuously refine existing Gaussians, while restricting Gaussian generation to previously unobserved scene content; this allows Gaussian growth to be adaptively determined by scene scale rather than increasing with the view count. Specifically, PersistSplat maintains a Persistent Gaussian Memory for each Gaussian, with a fixed anchor and an updating historical observation set: for regions in the incoming view already covered by existing Gaussians, the model writes the current observations into the corresponding historical observation sets and uses the associated fixed anchors as queries to adaptively retrieve and aggregate historical context to refine existing Gaussians, avoiding redundant generation; for previously unobserved regions, it generates new Gaussians and initializes their anchors and historical observation sets, enabling incremental scene modeling. To ensure stable memory evolution, we further introduce Global Geometry Alignment and Global Feature Alignment to correct geometric drift and feature deviations. Experiments show that PersistSplat effectively reduces redundant Gaussians, allowing it to gradually converge to a scene-scale-dependent upper bound; while matching or outperforming existing offline and online baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Feed-forward 3D Gaussian Splatting, Compact 3D scene representations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4vvxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses compact feed-forward 3D Gaussian Splatting from streaming, unposed image sequences. It argues that existing online methods predict dense Gaussians for every frame and then compress them, so the number of Gaussians keeps growing with the number of views. The proposed PersistSplat processes the stream in overlapping sliding windows with a frozen AnySplat backbone. The geometry of each window is aligned to a global coordinate system through Sim(3) alignment and a learned correction, and its features are corrected using the existing Gaussians as reference. Each Gaussian maintains a fixed anchor feature and a bounded set of historical observations. For regions of an incoming view already covered by existing Gaussians, the new observations are stored and used to refine those Gaussians, while new Gaussians are generated only for uncovered regions. The authors report that the method achieves rendering quality comparable to AnySplat with fewer Gaussians, and outperforms existing online methods on longer input sequences.

Critical strengths
  1. The design of adding new Gaussians only in regions not covered by previous observations is reasonable for streaming inputs. Under this design, regions that have already been reconstructed are refined rather than repeatedly regenerated, so the representation grows only with newly observed content instead of with every incoming frame.

  2. The authors conduct extensive experiments. They compare against a range of offline and online feed-forward methods on two datasets, under different numbers of input views and sequence lengths, and provide numerous qualitative comparisons.

Critical weaknesses
  1. Reconstruction from streaming inputs is a meaningful problem, but the reviewer finds that the setting adopted in this paper does not reflect the conditions under which streaming is actually needed, such as a number of views beyond the capacity of offline methods, real-time requirements or limited memory. The paper does not identify which of these it targets, leaving it unclear what problem the method actually aims to solve. The experimental setting further reflects this ambiguity: at most 150 views at 224×224 on small static scenes, which offline methods such as AnySplat can already handle, whereas the settings where streaming becomes necessary, such as long sequences of large-scale scenes like the kilometer-scale KITTI and Waymo sequences used in VGGT-Long [1], or dynamic scenes, are not covered.

  2. The reviewer considers the novelty of the method to be limited, as its main components closely follow existing designs. Adding new Gaussians only where the current map does not yet cover the scene, determined by rendered coverage and depth consistency, has been explored in online Gaussian reconstruction such as SplaTAM [2]; this paper mainly brings this strategy into a feed-forward setting. Maintaining a persistent memory associated with 3D locations and updating it with new observations has been explored in streaming reconstruction such as Point3R [3], and projecting existing Gaussians into the incoming view to decide how they are updated is similar to LongSplat [4]. The global geometry alignment follows the overlapping-chunk Sim(3) alignment in VGGT-Long [1], adding only a learned residual correction. The paper neither discusses these relationships nor compares with the corresponding designs, so the specific contribution beyond combining them remains unclear.

  3. The reviewer has concerns about the experimental design and the fairness of the comparisons. Except for AnySplat, the backbone of the proposed method, the baselines perform well below the results reported in their original papers, and some even rank differently relative to AnySplat. The reported Gaussian counts of some baselines are also inconsistent with the stated 224×224 resolution. In addition, offline methods are only compared with short inputs, where offline processing is already feasible, and are excluded from the long-sequence comparison, which also omits LongSplat [4]. Comparisons with AnySplat at a matched number of Gaussians and on runtime and memory are also missing. Several claims, such as the convergence of the Gaussian count and the stable performance over long sequences, are not supported by the reported results.

  4. The reviewer finds that the paper is not clearly written, and many essential details are missing. In the method section, the architectures of the learned modules, including the geometry correction, the feature correction and the Gaussian head, are not described, nor is it stated which components are trained and which are inherited from the frozen backbone. The training setup is also unclear, such as which views are rendered for supervision, how many windows are unrolled during training and how many training steps are used. The experimental section has similar problems. It does not describe how the input views are sampled, which frames are used as target views, how target poses are obtained for pose-free methods, or under what settings the baselines are run. As a result, the method and results cannot be reproduced or properly assessed from the paper. The paper also contains numerous grammatical errors and typos.

[1] K. Deng, Z. Ti, J. Xu, J. Yang, and J. Xie. VGGT-Long: Chunk it, Loop it, Align it — Pushing VGGT's Limits on Kilometer-scale Long RGB Sequences. ICRA, 2026.

[2] N. Keetha, J. Karhade, K. M. Jatavallabhula, G. Yang, S. Scherer, D. Ramanan, and J. Luiten. SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM. CVPR, 2024.

[3] Y. Wu, W. Zheng, J. Zhou, and J. Lu. Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory. arXiv:2507.02863, 2025.

[4] G. Huang, R. Wang, X. Gao, C. Sun, Y. Wu, S. Gao, and Y. Jia. LongSplat: Online Generalizable 3D Gaussian Splatting from Long Sequence Images. AAAI, 2026.

Questions for authors
  1. Were the baselines evaluated with their official checkpoints or retrained, and at which resolution was each of them run? Could the authors explain why their results are substantially lower than those reported in the original papers?

  2. Could the authors provide evidence that local geometric distortions remain after Sim(3) alignment, for example residual maps on overlapping views before and after the learned correction, or pose errors such as ATE?

  3. Could the authors evaluate the method in scenarios that genuinely require streaming input, rather than on RE10K and DL3DV, for example continuously captured driving sequences such as the KITTI and Waymo sequences used in VGGT-Long [1]? How do the Gaussian count, rendering quality, runtime and memory behave as frames keep arriving in such settings?

  4. The method relies heavily on the frozen AnySplat backbone for features, pointmaps and poses. Can the proposed framework be applied to newer backbones with different architectures and output formats, and do its benefits persist with a stronger backbone? If the gains are tied to AnySplat, the significance of the contribution would be limited.

[1] K. Deng, Z. Ti, J. Xu, J. Yang, and J. Xie. VGGT-Long: Chunk it, Loop it, Align it — Pushing VGGT's Limits on Kilometer-scale Long RGB Sequences. ICRA, 2026.

Reviewer_py3tOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PersistSplat, a feed-forward framework for streaming 3D Gaussian reconstruction from unposed image sequences. It processes incoming images in overlapping sliding windows using a frozen reconstruction backbone and introduces Global Geometry Alignment and Global Feature Alignment to align window-level predictions with the accumulated global representation.

Critical strengths
  1. The paper addresses a meaningful problem: reducing redundant Gaussian generation in streaming 3D reconstruction.
  2. The reported experiments show better novel-view synthesis quality with fewer Gaussians than the evaluated online baselines.
Critical weaknesses
  1. Efficiency evaluation. Gaussian count alone does not establish computational or memory efficiency, since PersistSplat stores per-Gaussian anchors and historical observations and performs additional alignment and refinement. The evaluation should report end-to-end runtime and peak GPU memory, including backbone inference and subsequent stages, across different sequence lengths, window sizes, and memory capacities. Runtime comparisons with SLAM-based reconstruction systems under comparable input and evaluation settings would further contextualize its practical efficiency.
  2. Scalability to long sequences. The reported experiments use at most 150 views. Considering the task of streaming 3DGS, long-term streaming scalability could be evaluated for better understanding. Specifically, evaluation on thousands of frames, covering both longer trajectories and repeated revisits to fixed scenes, would help assess accumulated drift, reconstruction stability, and whether Gaussian growth approaches saturation in fixed scenes. Qualitative results at different sequence lengths would also clarify how reconstruction quality evolves over time.
Questions for authors
  1. Please see the weakness section. I will raise scores if they are addressed.
  2. More ablation studies on hyperparameters will be useful for evaluating the proposed method (including the reconstruction quality, peak GPU memory, and end-to-end runtime under varied view counts). Moreover, additional qualitative results could help understand the advantages of the proposed method.
  3. Online evaluation protocol. For the online evaluation, which Gaussian state is used to evaluate each held-out view? For example, with inputs at poses 1, 11, 21, and back to 1, is held-out view 15 evaluated using the state after input 11, after input 21, or after the full sequence? Reporting performance both when a region is first reconstructed and after subsequent updates or revisits, using the same held-out views will help reveal whether later updates preserve or degrade previously reconstructed regions.
  4. Revisits and loop closure. How does the sliding-window framework handle revisits, particularly after accumulated pose drift? The current description does not appear to include explicit loop closure. Please clarify how revisited observations are associated with existing Gaussians, whether misalignment can lead to duplicate generation, and whether earlier geometry can be corrected.
Reviewer_tV3EOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses redundancy in streaming feed-forward 3D Gaussian Splatting. Pixel-aligned feed-forward methods emit Gaussians for every pixel of every frame, so the representation grows with view count even when the scene does not. Existing online methods mitigate this after the fact through merging or pruning.

PersistSplat instead decides, per incoming view, which regions are already represented. Built on a frozen AnySplat (VGGT-based) encoder with sliding windows, it:

  • projects existing Gaussians into the new view and uses depth consistency to identify a reliably covered region;
  • for covered regions, writes the new observation into each Gaussian's memory (a fixed anchor feature plus up to 8 stored observations) and re-decodes the Gaussian via anchor-queried attention over that memory, so existing Gaussians are refined rather than duplicated;
  • for uncovered regions, creates new Gaussians;
  • adds Global Geometry Alignment (Sim(3) plus a learned residual correction) and Global Feature Alignment (anchor-based feature correction) to keep windows consistent.

On RE10K and DL3DV, the method matches offline AnySplat with fewer Gaussians and strongly outperforms StreamGS and OF3GS while using roughly 10–30× fewer Gaussians on long sequences, with the Gaussian count saturating on fixed-scale scenes.

Critical strengths
  • Good motivation and a clear gap. Refining existing Gaussians when content is re-observed, and generating only for new content, is the right principle for streaming reconstruction. Among closely related streaming feed-forward methods, I am not aware of prior work combining unposed input, a persistent scene built from a moving camera, and refinement in place of pruning.
  • Strong compactness results. On fixed-scale RE10K, Gaussian count grows only from 451K to 489K between 50 and 150 views, versus millions for the online baselines.
  • Useful component ablations. Table 4 compares PGM against direct accumulation, opacity pruning, voxel merging, and uniform memory fusion.
Critical weaknesses

Fix Ablation Will Increase My Score to Weak Acceptance

1. The ablations lack the most informative baselines: training-free versions of the learned components. Several modules learn operations that have well-established, training-free counterparts, but the paper never compares against them. Without these comparisons, it is not possible to tell whether the learned components are necessary.

  • Persistent Gaussian Memory. Refinement amounts to a learned, reliability-weighted average of observations that nudges the fixed anchor (ā_g = a_g + 0.1·tanh(W_S s)). The only alternative tested is uniform fusion (Mem. Fus., Table 4). The natural baseline is a parameter-free weighted average: weight raw observations by their similarity to the anchor (e.g., cosine similarity) and by reliability ρ, with no learned projections. Since anchors and observations are sampled from the same foundation-model feature space, this baseline is straightforward and would show whether the learned attention adds anything beyond similarity-and-reliability weighting.
  • Global Geometry Alignment. The learned correction A_geo is compared only against Sim(3) alone, and adds just +0.24 dB by itself (Table 3). A standard training-free alternative is Sim(3) initialization followed by a lightweight bundle adjustment over each window, using the exact pixel correspondences available in the overlapping views and VGGT's predicted tracks or matches. The BA-refined points can then serve as control points to correct the dense pointmaps (e.g., per-view scale and shift, or a smooth spatially varying correction), yielding dense, aligned geometry without training. Recent VGGT-based streaming and SLAM pipelines use this kind of optimization-based chunk alignment. Please compare in pose accuracy, rendering quality, and runtime.
  • Same-backbone streaming baseline. PersistSplat uses a frozen AnySplat encoder, while StreamGS uses DUSt3R, so part of the 5–7 dB margin may come from the backbone. A simple streaming baseline on the same backbone (per-window AnySplat, Sim(3) chaining, naive accumulation plus merging or pruning) should appear in the main tables. Table 3 row 1 and Table 4 row 1 come close and could be promoted.
  • Hyperparameter sensitivity. None of the method's hyperparameters are ablated: λ₂ (update strength), λ₁ (reliability weight, including λ₁ = 0), J (memory capacity), the coverage depth tolerance δ_obs (whose value, and the form of ρ, are not given), the slot-merging thresholds, and window size and overlap. These directly control the compactness/quality trade-off.

2. The rationale for learning each component is not given. The method replaces several classical operations with learned modules (attention-based fusion, learned geometric correction, learned feature correction) but does not explain what each learned module achieves that its classical counterpart cannot. This adds training dependence (all modules are trained only on RE10K), complexity, and unspecified components: the architectures of A_geo, A_rel, and A_corr and the losses L_plane, L_point, and L_pose are not described, and the initialization of new Gaussians is only briefly specified. I would ask the authors to justify each learned module explicitly, ideally via the baselines in Concern 1. If a learned module offers a specific advantage (e.g., correcting high-frequency depth distortions that a smooth control-point correction cannot), please demonstrate it directly.

3. View-dependent appearance is not discussed. Several design choices systematically favor appearance consistent with a Gaussian's first observation:

  • the attention query is the fixed anchor, i.e., the first observation, so observations that look different score lower;
  • the update is strongly bounded (λ₂ = 0.1 with tanh);
  • viewing direction d_j is stored but not used in the attention (Eq. 5), so aggregation is not conditioned on view.

As a result, observations containing view-dependent effects (specular highlights, reflections, glass, water) are likely to be down-weighted as outliers, which may push Gaussians toward view-independent appearance. The paper does not mention this anywhere, including in the limitations. Please state the spherical-harmonic degree, discuss how view-dependent appearance is represented, and ideally show attention weights or results in reflective or transparent regions.

Secondary Concerns

  • Positioning relative to SLAM. The core mechanism (projective association by depth consistency, a per-Gaussian observation track, and creation only for unassociated regions) parallels data association and map maintenance in SLAM and incremental SfM (e.g., ORB-SLAM, KinectFusion/ElasticFusion, and coverage-based densification in SplaTAM). None of this literature is cited. Making the connection explicit would clarify what is new and would position the paper as a bridge between feed-forward 3D foundation models and SLAM.
  • Related work. Fuse-and-Refine (NeurIPS 2025) shares the philosophy of learned fusion over heuristic pruning and should be discussed, along with how its setting (dynamic scenes, fixed posed cameras, short history) differs. LongSplat (AAAI 2026) is cited but not compared; please explain why (e.g., its pose requirement).
  • Figure 1 illustrates "merge" versus "refine" with abstract cartoons but does not show the key decision: which Gaussians are associated and refined versus where new ones are created. Showing projected Gaussians, the depth check, and the resulting covered and uncovered masks on real frames would make the main idea intuitive.
  • Memory-state accounting. Each Gaussian carries an anchor and up to 8 slots, so the true streaming state may be much larger than the Gaussian count suggests. Please report total state size (and dimensions C and d), plus runtime and memory for the baselines.
  • Evaluation details. Please specify test scenes, context/target selection, and how target-view poses are obtained for this pose-free method; how the StreamGS results were obtained, as it does not appear to have a public implementation; and why Figure 1's numbers do not match Tables 1–2. Experiments run only at 224 × 224. Quality also declines with sequence length on fixed-scale RE10K (23.22 → 20.87 PSNR from 50 to 150 views) despite the saturating Gaussian count, which may reflect accumulated drift from pairwise window chaining.
Questions for authors
  1. How does PGM compare with a parameter-free weighting of raw observations by anchor similarity and reliability, without learned projections?
  2. How does GGA compare with Sim(3) plus lightweight bundle adjustment, with BA points used as control points to correct the dense pointmaps, in pose accuracy, rendering quality, and runtime?
  3. What does each learned module (A_geo, A_rel/A_corr, the attention projections) achieve that its training-free counterpart cannot?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.