acceptodds
Withdrawn from ICLR 2027

LonGaussian: Tracking Foundation Priors for Long-Context Feedforward Gaussian Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstructing high-quality, renderable 3D scenes from long videos is essential for robotic perception and embodied intelligence. Many feed-forward Gaussian reconstruction methods rapidly predict 3D scenes from a small set of images, but extending them to longer sequences presents two bottlenecks: jointly processing all views increases GPU memory requirements, while accumulating pixel-aligned predictions redundantly represents the same surfaces and increases storage and rendering costs. We present **LonGaussian**, a streaming feed-forward framework that reconstructs compact, globally renderable Gaussian scenes from long image sequences with known intrinsics and unknown camera poses. Built on a pretrained streaming 3D reconstruction backbone, LonGaussian decodes long-context geometric features into dense Gaussians. Within each chunk, multi-frame point tracks link repeated observations of the same surface and guide a learned merger to consolidate the corresponding Gaussian predictions. Rendered-depth consistency filtering further suppresses Gaussians from other chunks that incorrectly occlude locally reconstructed surfaces. Across long-sequence evaluations, LonGaussian produces high-quality global Gaussian scenes using substantially fewer Gaussians than the evaluated feed-forward baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Reconstruction, 3D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7br2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LonGaussian is a streaming feed-forward framework for reconstructing compact, renderable 3D Gaussian scenes from long image sequences with known intrinsics and unknown camera poses. It builds on a frozen LoGeR backbone for geometry and long-context features, predicts dense pixel-aligned Gaussians, and uses tracks from frozen AllTracker to guide a learned within-chunk merger. Rendered-depth filtering and a far-field Gaussian shell address additional cross-chunk and sky artifacts. Experiments on held-out DL3DV and RealEstate10K sequences compare rendering quality and Gaussian count at 64, 96, and 128 context frames. A separate single-scene study examines construction scalability and local rendering quality up to 3,768 input frames.

Critical strengths
  1. An important problem addressed through a well-motivated design. The paper addresses both the memory cost of processing long sequences and the redundant scene representations produced by accumulating pixel-aligned Gaussians. Using multi-frame tracks to identify repeated observations provides a clear rationale for consolidation. The overall presentation is clear, and the distinction between streaming geometry estimation and Gaussian scene construction makes the approach easy to follow.

  2. Impressive rendering-quality and compactness results against the evaluated streaming baselines. At 128 context frames on DL3DV, LonGaussian achieves 18.31 dB PSNR with 4.80 million Gaussians, compared with 14.83 dB and 13.67 million for OF3GS. On RealEstate10K, the corresponding results are 18.23 dB and 2.15 million versus 16.19 dB and 11.37 million. These are substantial gains in both quality and representation size. The component ablations also provide useful evidence that learned consolidation and artifact handling contribute to the final results.

Critical weaknesses
  1. The dependence on the two frozen foundation models is insufficiently quantified. LoGeR supplies the point maps, camera poses, and cross-chunk alignment underlying Gaussian prediction, while AllTracker determines which observations can be consolidated. Consequently, geometry errors, track coverage, and track fragmentation can affect both rendering quality and the number of retained Gaussians. The paper acknowledges these limitations in Section 5 and Appendix E, and Figure 8 illustrates severe global-rendering artifacts associated with long-range misalignment. However, the component ablations in Table 3 do not measure how the accuracy of LoGeR's predictions or the density and reliability of AllTracker's tracks affect the final quality-compactness trade-off. This is particularly relevant because the evaluation uses method-specific predicted target poses, so image metrics reflect pose-estimation error as well as scene quality (Appendix D.3). A more direct analysis of these dependencies is needed to establish when the proposed consolidation remains effective and where the frozen priors become the main bottleneck.

  2. The comparison and generalization evidence do not yet fully support the intended long-sequence setting. AnySplat and WorldMirror are included in Table 1, but every entry is OOM under the 24-GB memory budget. This establishes a useful resource limitation under the chosen setting, but leaves their rendering quality and Gaussian compactness unassessed. Both methods have reported feed-forward results with 200 input views on MatrixCity under their respective evaluation settings (AnySplat, Table 2; WorldMirror, Appendix B.4 and Table 10). A complementary comparison under feasible native inference settings, with the memory budget and other settings explicitly reported, would clarify the trade-off against these relevant offline methods.

    The systematic multi-scene evaluation is also restricted to held-out sequences from DL3DV and RealEstate10K, both of which are used during training or adaptation, and to at most 128 context frames. The train/test separation is documented; the remaining concern is generalization to different domains and substantially longer sequences. The fixed sampling protocol in Appendix D.1 does not establish robustness to different input-view intervals or reduced overlap, which directly affect tracking and consolidation. The additional 3,768-frame study is valuable, but covers only one scene and evaluates quality using seven neighboring chunks rather than the complete accumulated representation. As the authors acknowledge, this reduces interference from distant misregistered chunks and does not establish full-scene global rendering fidelity. Multi-scene evaluation at greater lengths and on out-of-domain data, for example suitable sequences from MatrixCity or ScanNet++, would materially strengthen the central claim.

Questions for authors
  1. Dependence on frozen priors. How do LoGeR's geometry and pose errors correlate with final rendering quality, and how do AllTracker's track density, coverage, and reliability affect Gaussian count and rendering quality? Can controlled measurements separate these effects from those of the learned merger, including separating target-pose error from scene-rendering error where feasible?

  2. Offline baseline comparison. Can AnySplat and WorldMirror be evaluated under feasible native inference settings, such as shorter sequences or a larger explicitly stated memory budget, while retaining the current 24-GB comparison? Rendering metrics and Gaussian counts would help assess the quality-efficiency trade-off beyond whether inference fits in memory.

  3. Long-sequence and domain generalization. What evidence is available for different input-view sampling intervals and for longer sequences across multiple unseen-domain scenes, such as MatrixCity or ScanNet++? In those experiments, can full-scene rendering quality be reported separately from local-chunk rendering to show how global consistency changes with sequence length?

Reviewer_oG45OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper concentrates on the challenges of memory usage and rendering quality for long-sequence 3D reconstruction. Although conventional methods apply chunk-wise reconstruction to resolve the conflict, the authors reveal severe Gaussian redundancy from repeated observations, which increases both storage and rendering cost, and harms the rendered-depth consistency and rendered-texture quality.

Critical strengths

The main contribution of the paper is to propose a novel Gaussian merge strategy based on multi-view point tracks. To effectively tackle this challenge, the pipeline starts from the neural priors of a streaming feed-forward reconstruction model, LoGeR, following a tracker-guided Gaussian merger to produce compact, renderable scene representations.

Critical weaknesses
  • The tracker-guided consolidation is a core contribution of the paper. However, its necessity should be clarified more explicitly. For instance, why is such a strategy the most suitable choice in this case? What are the limitations of other merge approaches?

  • In Table 1, the WorldMirror and AnySplat are mentioned to illustrate that advanced feed-forward Gaussian reconstruction methods fail on ultra-long sequences, marked as “OOM”. Figure 4 excludes their visualization results. Perhaps it is better to compare their performance on shorter image sequences with the proposed LonGaussian both qualitatively and quantitatively. This would provide a more balanced and comprehensive evaluation, clarifying the trade-off between reconstruction quality and input scalability.

  • The contributions of the individual modules are not sufficiently disentangled. In particular, the ablation studies suggest that depth filtering contributes more to rendering quality than tracker-guided merging, despite the latter being emphasized as the central technical contribution. The authors should clarify the respective roles of these modules in improving rendering quality and reducing Gaussian redundancy.

Questions for authors
  • Table 3 suggests that depth filtering contributes more to rendering quality than track-guided merging. Could the authors clarify the respective contributions of these modules to rendering quality and compactness? What evidence establishes the advantage of multi-frame tracking over simpler correspondence-based or spatial merging strategies?

  • Could the authors provide quantitative and qualitative comparisons with WorldMirror and AnySplat at shorter sequence lengths that fit within the memory budget? The OOM results establish a scalability limitation but leave their reconstruction quality relative to LonGaussian unclear.

Reviewer_C1ZeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

LonGaussian extends LoGeR with Gaussian prediction, within-chunk visual tracking guided Gaussian aggregation, and cross-chunk rendered-depth filtering. The problem of redundant Gaussians in pixel-aligned reconstruction is worthwhile.

Critical strengths

Table 1 reports better rendering quality with fewer Gaussians than the evaluated streaming baselines.

Critical weaknesses
  1. The central design choice is not convincingly validated. The long-context backbone and alignment are inherited without modification from LoGeR, while a frozen AllTracker provides multi-frame correspondences for the merger. The paper motivates tracks as preferable to voxel or spatial aggregation, but provides no direct comparison with those approaches. Table 3 compares against dense accumulation and deterministic track merging; neither establishes the advantage of track-based grouping. A comparison with the same backbone, Gaussian budget, and filtering is required.
  2. The ablation leaves the methodological contribution unclear. In table 3, learning merging improves PSNR by only 0.25dB over deterministic tracking merging (18.88 versus 18.63 dB). Removing the rule-based depth filter produces a much larger 1.52 dB drop. The filter may be useful, but these results do not show that the learned track merger is the main source of the quality improvement.
  3. The comparative evaluation is narrow for the paper's claims. Multi-scene quantitative results cover only DL3DV and RealState10K, and stop at 128 context frames. WorldMirror and AnySplat are out of memory at every reported length, leaving numerical quality comparisons with only ReCoSplat and OF3GS. Moreover, each method estimates its own target poses, such that rendering scores combine pose and scene errors. Results with common target poses would better isolate reconstruction quality.
  4. Long-range global rendering remains unproven. In table 1, LonGaussian's PSNR falls from 19.57 to 18.31 dB on DL3DV and from 19.73 to 18.23dB on RealEstate10K between 64 and 128 context frames. The single-scene 3678 frame study reports a further decline. More concerning, those long-sequence scores render from only seven nearby chunks, while Figure 8 show severe ghosting when the complete accumulated scene is rendered on a revisit. Thus the longest experiment does not substantiate the claimed globally renderable scene. Full-scene metrics and end-to-end costs on several long sequences, including revisits, are needed.
Questions for authors

see the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.