acceptodds
Under review as a conference paper at ICLR 2027

ZipSplat: Observation-Grounded Scene Tokens for Feed-Forward Gaussian Splatting

16 views

Abstract

Feed-forward 3D Gaussian Splatting reconstructs scenes from images in seconds. Pixel-aligned methods keep every Gaussian grounded in an observation, but produce redundant, potentially misaligned Gaussians across views. Token-based methods instead aggregate all views into a fixed set of learned tokens but lack explicit observation support and use a fixed reconstruction budget. We introduce ZipSplat, a method that predicts a compact, observation-grounded scene representation whose size is set at inference. It derives each scene token from an image patch rather than a learned query, and decodes it into freely placed 3D Gaussians. Tokens across views are merged into a compact set whose size is adjustable at inference. However, without geometric supervision, decoded Gaussians can drift from their source patch, and merging can mix tokens from different surfaces. We therefore introduce two geometric losses to ground and shape this representation. A cone loss ties each token's Gaussians to its source patch, and GeoCE aligns token similarity with 3D proximity so that tokens observing the same surface merge together. Building on these observation–geometry associations, we further introduce TokenBA, a token-level bundle adjustment that refines scene tokens and camera poses in a few seconds. ZipSplat requires neither camera poses nor intrinsics, yet outperforms pixel-aligned and token-based baselines on DL3DV and RealEstate10K, even those given ground-truth cameras. It further generalizes zero-shot to Mip-NeRF360 and ScanNet++. Code and trained models will be made public.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Feed-Forward Reconstruction, Novel View Synthesis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_r43XOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ZipSplat, a feed-forward model that reconstructs 3D scenes from unposed images as a compact set of observation-grounded scene tokens, each of which is decoded into a set of freely positioned Gaussian primitives. Unlike previous fixed-query representations, the proposed formulation supports an adjustable reconstruction budget at inference time without requiring retraining. In addition, the paper introduces TokenBA, an efficient token-level bundle adjustment algorithm that exploits the resulting 2D–3D correspondences to jointly refine scene tokens and camera poses using reprojection and photometric consistency, requiring only a few seconds of optimization.

Critical strengths
  • The paper presents an effective approach for predicting compact Gaussian primitives. Compared with previous fixed-query methods, the proposed representation maintains an explicit connection to the input observations while allowing the reconstruction budget to be flexibly adjusted according to the complexity of the scene.
  • ZipSplat consistently outperforms competing methods while using significantly fewer Gaussian primitives, demonstrating a favorable trade-off between reconstruction quality and representation efficiency.
Critical weaknesses
  • The method description could be made clearer in several places. For example, in Lines 178–179 (“These are merged into geometry tokens g\mathbf{g}”), it is unclear how the multi-layer visual tokens are merged into the geometry tokens. Furthermore, in Lines 180 and 182, the paper states that “Each geometry token serves as a scene token” and later refers to “each scene token zk\mathbf{z}_k.” Are geometry tokens and scene tokens the same concept? If so, using different terminology and notation is confusing. However, in Line 213, the statement “Each scene token is initialized as the mean of its cluster members” suggests a different definition of scene tokens, making the relationship between these concepts even less clear. Moreover, in Line 234, the term color features is introduced without explanation. Clarifying these definitions and maintaining consistent terminology throughout the paper would significantly improve readability.
  • The comparisons in Tables 1 and 2 could be made fairer. The reported results include TokenBA, an additional optimization step that improves performance but also incurs extra computation time. To better separate the contributions of the feed-forward model and the subsequent optimization, the authors should report the performance and runtime of the proposed method both with and without TokenBA, and compare them against the corresponding performance and runtime of all baseline methods.
  • The backbone comparison also raises a fairness concern. The proposed method is built upon the pretrained DA3-Giant backbone, whereas TokenGS is not initialized from a pretrained 3D foundation model. Since the pretrained backbone likely provides a substantial advantage, it would be helpful to either discuss this limitation explicitly or include comparisons under more comparable initialization settings.
  • The paper should include a comparison with a simple k-means baseline without token fusion to validate the effectiveness of the proposed token fusion strategy.
Questions for authors

The current clustering step is non-differentiable. Have the authors explored differentiable alternatives that would make the entire pipeline end-to-end trainable? A discussion or ablation on this design choice would be valuable.

4 comments

Thank you for the detailed review. We are glad that the explicit link to the input observations and the adjustable budget came across. Below we address each point:

  1. Terminology: Thank you for pointing these out. We will make the definitions more clear in Sec. 3.1 and 3.2 of the revision. To clarify:

    • Geometry token g_i: one per image patch. It combines backbone features from three layers with an embedding of the raw RGB patch, each passed through an MLP and summed with learnable scales (Eq. 11).
    • Scene token z_k: one per cluster of geometry tokens. It starts as the mean of the geometry tokens in its cluster (line 213). The fusion blocks then refine it (Eq. 4 and 5). Each scene token is decoded into Gaussians.
    • Color features: the RGB patch embedding above. The final cross-attention uses it as values (Eq. 5).

    Appendix A.1 gives the full construction, and we are happy to clarify further if anything remains unclear to improve our submission.

  2. TokenBA Results and Timings: Thank you for this feedback. The matched comparisons are reported in App. F (Tables 12 to 15). Without TokenBA, ZipSplat still leads all feed-forward baselines in PSNR and LPIPS on every benchmark. At 16 views it takes 0.19 s feed-forward and 1.8 s with TokenBA, against 6.3 s for TokenGS with its optimization (Fig. 10, App. C.1, App. E). To improve clarity in the main paper, we will add a column to Tables 1 and 2 that marks which methods use test-time refinement. We will also add a ZipSplat row without TokenBA.

  3. Backbone: Table 4 should address this already: Its “Fixed queries” row retrains the TokenGS formulation on the same pre-trained backbone as ours (DA3-Base). It uses learned queries, their frustum loss and the same training budget. On DL3DV at 6 views it reaches 23.02 dB with 262K Gaussians, while ZipSplat without TokenBA reaches 24.34 dB with 62K and still 23.25 dB with 31K. The gain therefore does not come from the backbone but can be attributed to our formulation. We will make this more explicit for the revision.

  4. Fusion Ablation: Token fusion gives each merged token access to its individual observations. Without it, which we read as decoding the cluster means directly (without Eq. 4 and 5), each scene token is a plain average of its member patches. As a first check, we removed fusion from the trained model at inference. On DL3DV at 6 views, PSNR drops from 24.3 to 14.0 dB and LPIPS rises from 0.17 to 0.76. This inference-only test shows that the decoder relies on the fused tokens. Please let us know if this is the comparison you have in mind.

  5. Differentiable Clustering: Yes, during development we tested soft k-means on an earlier, smaller model. It was slightly below hard assignment at full capacity and under compression. A soft assignment blends tokens even when each cluster is a single patch, so each token is always a mixture. Our pipeline is still trained end to end, since each scene token is the differentiable mean of its members. Our GeoCE loss reaches the same goal, a token space suited for clustering, as it shapes the token space directly (App. D.1).

Thank you again for the constructive feedback. We are happy to discuss any of these points further.

AuthorsOpenReview ↗

Fusion ablation (follow-up)

Following up on the fusion comparison: we have now retrained the model without token fusion, i.e. each scene token is the plain cluster mean of its member geometry tokens and is decoded directly, without Eqs. (4) and (5). All other components and the training budget are unchanged. The result is included as the last row of Table 4 in the revision. On DL3DV at 6 views, removing fusion reduces PSNR from 24.34 to 22.59 dB at r=1 and from 23.25 to 21.22 dB at r=0.5. The retrained number supersedes the inference-only test from our earlier reply (14.0 dB), which removed fusion from a model trained with it. The retrained variant confirms that fusion contributes about 1.8 to 2.0 dB, with the larger loss under compression, where each merged token must represent several patches.

We hope this is the comparison you had in mind and are happy to extend it further.

Thanks for the authors' detailed explanations. The new experiments have carefully addressed my concerns. I have just one remaining question for clarification: Does the method support training from scratch without relying on pretrained priors?

Thank you for the question. In principle yes: no component of ZipSplat assumes a pretrained backbone, and the encoder is trained end-to-end with the rest of the model. In practice, pose-free reconstruction requires learning multi-view correspondence and camera estimation from images alone, which is why all pose-free baselines (AnySplat, NoPoSplat, YoNoSplat) also start from a 3D foundation model. Posed methods such as TokenGS are given ground-truth cameras, so they skip this part and can train from scratch. Training ZipSplat from scratch would mean re-learning that geometric prior at foundation-model scale in compute and data. Table 4 confirms the gains do not come from the backbone, as all rows share it.

Reviewer_PKMuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

For feed-forward Gaussian splatting methods, an important question is how image observations are modeled as Gaussians: how many are predicted and where they are located in 3D space. As the paper discusses, most methods predict a single Gaussian per input pixel, so each prediction is grounded in a single pixel of the image observation. This produces many cross-view redundancies, as there would be repeated observations of the same surface, leading to separate view-specific questions rather than a shared prediction.

Other methods have attempted to remove this redundancy and cross-view misalignment by learning tokens with no explicit link to the observations, which can produce blurry reconstructions. SceneTok, on the other hand, derives each scene token directly from the source image patch, so the token-to-observation link is explicit. These tokens are then decoded into 3D questions. To constrain them to the patches that produce them, the authors propose a novel loss for scenarios where many views overlap, and many patches observe the same surface. Instead of merging in feature space, the authors propose a new geometric-contrastive loss called GeoCE, which clusters tokens based on 3D proximity, so they cluster by surface rather than by viewpoint.

To correct the final residual camera and geometry errors, the authors leverage the 2D-3D correspondence between each token and its corresponding image patch via a token-BA module. That module performs token-level bundle adjustment to do PnP refinement of cameras and scene tokens through the frozen decoder.

Critical strengths
  1. The introduction of the paper is fairly well written, and I found it very easy to understand the problem as it builds the motivation step by step: citing prior work, mentioning the drawbacks, and explaining how each of those is addressed in the paper. Even though I'm familiar with the topic, I think someone less familiar would also find it easy to follow the paper's initial motivation and setup.

  2. The idea of having an adaptive token budget per scene, rather than producing the same number of goals regardless of scene content, appearance, or geometry complexity, is a timely contribution to the field.

Critical weaknesses
  1. Novelty relative to the recent 2026 token-based 3DGS paper: TokenSplat is incremental. The idea of merging per-view image tokens before decoding comes from TokenSplat, and the test-time token optimization comes from TokenGS. Decoding several Gaussians per anchor is closely related to scaffold-GS. What is genuinely novel is the key to clustering in learned token space and the cone loss. Table 11 suggests that clustering tokens by 3D position, which is in spirit similar to TokenSplat's voxel merging, performs about the same.

  2. The paper seems to ignore scene tokenizers outside of 3DGS that share its premise. Encoding a set of views into a complex set of tokens is demonstrated in SRT (CVPR 2022) and SceneTok (CVPR 2026), but neither is cited. The related work frames the alternative to ZipSplat as only 3DGS methods with learned queries, so the difference from this line of work is not well understood: tokens are derived from patches rather than from learned queries.

  3. Token BA, a per-scene test-time optimization strategy, seems to be behind most of the margin over baselines. The main tables mix it only with feed-forward baselines. Every ZipSplat entry in Tables 1 through 3 includes Token-BA, while 9 out of the 10 baselines run only feed-forward. TokenGS has its own test-time optimization. In the ablation study, we can see that Token-BA adds about 3.41 dB, compared to just 1.76 dB from the representation itself, as shown in Table 4. Without it, ZipSplat is still better than purely feed-forward baselines, but it trails TokenGS with test-time optimization on RealEstate10K and mipnerf360, as well as ScanNet

Questions for authors

N/A

3 comments

Thank you for the thorough review. We are glad that the motivation of our formulation and the adaptive budget came across. Below we address each point:

  1. Novelty: Our contribution has three parts: observation-grounded scene tokens (L93), two losses that ground and shape them (L96), and TokenBA, which refines them (L99). You identify the cone loss and the clustering in a learned token space as "genuinely novel”. These are what ground and shape the tokens. This grounding separates ZipSplat from the works you mention:
    • Merging (TokenSplat). TokenSplat merges tokens by their predicted 3D positions. ZipSplat clusters them in a learned token space, shaped by our GeoCE loss. This gives an exact budget at inference, and each merged token keeps its source patches.
    • Test-time refinement (TokenGS). The cone loss ties each token’s Gaussians to its source patches. TokenBA uses these 2D-3D correspondences to update the cameras with PnP. Without the cone loss, TokenBA collapses to 14.7 dB (Sec. 4.5). TokenGS has no such correspondences and needs ground-truth cameras.
    • Several Gaussians per token (Scaffold-GS). This design is shared by all token-based methods, including TokenGS. It is not among our claimed contributions.
    • Table 11. The 3D-position row uses ground-truth points. Our learned token space matches this row from images alone. This is what GeoCE is designed to achieve, and we will clarify it in the revision.
  2. Related Work: Thank you for pointing these out. Our related work focuses on feed-forward Gaussian splatting. Both works connect to the two lines discussed there: SRT relates one token with its input patch, similar to pixel-aligned methods, while a perceiver in SceneTok compresses the views into a learned set of queries, similar to TokenGS and its follow ups. ZipSplat sits between the two: its tokens keep the link to their source patches, yet are merged into a compact set whose size is set at inference. We will add both to the revision.
  3. TokenBA: As you note and we state in line 371, ZipSplat leads all feed-forward baselines without TokenBA. With refinement on both sides it also leads TokenGS with TTO in PSNR and LPIPS on every benchmark, including RealEstate10K, Mip-NeRF360 and ScanNet++ (Tables 1 and 2, full results in App. F). The representation alone accounts for the 1.76 dB as you point to, with six times fewer Gaussians and an adjustable budget. It is also what makes TokenBA effective: without the 2D-3D links from the cone loss, TokenBA collapses to 14.7 dB (Sec. 4.5). Gaussian-space refinement of the same prediction stays 0.6 to 2.4 dB lower (See Fig. 9). The refinement gain is therefore a result of the representation as well. To improve clarity in the main paper, we will add a column to Tables 1 and 2 that marks which methods use test-time refinement. We will also add a ZipSplat row without TokenBA.

Thank you again for the thoughtful review. We hope this clarifies our contribution and its relation to prior work, and we are happy to discuss any of these points further.

Thank you again for your review. We have uploaded a revision that includes the changes discussed in our response: a “Refined” column and a ZipSplat row without TokenBA in Tables 1 and 2, SRT and SceneTok in the related work (Sec. 2 and App. D, Table 11). As the discussion period closes soon, we would be grateful for your view on whether these changes and our earlier response address your concerns, and we are happy to clarify anything further.

Thank you for the detailed response, and apologies for not being able to engage during the discussion period. It resolves most of my concerns, and I am raising my score.

The positioning wrt SRT and SceneTok is clear. Please add it to the revision as promised.

Reviewer_DF7NOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3(+2)
Soundness
3(+1)
Read review
Paper summary

This is work on feedforward 3DGS reconstruction. Before reviewing this paper, I also used its released code for some of my projects.

My comments are based on the results I just obtained.

Critical strengths

I evaluated the official pretrained ZipSplat504 model on a single NVIDIA Tesla V100S GPU with 32 GB of memory. The inputs were several keyframes extracted from a synthetic camera-orbit video generated by Wan 3.0, conditioned on the original Fern scene image as the first frame and a target-view image previously generated by Seedance 2.5 as the last frame. I also directly used some image generation models to generate some photos based on the first view of Fern for testing. I evaluated both the feed-forward reconstruction and the result refined using the official default TokenBA settings. A nominal 0°–30° camera sweep was rendered directly at 504 × 280 pixels, producing a 5-second video with 150 frames at 30 FPS.

Based on my experiment, I am satisfied with the speed of this work.

Critical weaknesses

Our earlier experiments used two approaches to generate additional views: imagegen, which generated views through prompts and reference images without explicit cross-view geometric constraints, and SEVA, which jointly generated camera-conditioned views from a single Fern image. The later SEVA configuration generated 21 views using a CFG scale of 4 and a halved camera-translation magnitude. Reconstructions from these generated images exhibited noise, ghosting, and cloudy artifacts.

However, using frames extracted from a Wan 3.0-generated camera-orbit video produced visually cleaner reconstructions in our subsequent experiments. This suggests that more consistent input views can improve reconstruction quality, but the comparison does not isolate temporal continuity because the generation models, conditioning, and reconstruction settings also differed.

In our Fern experiments using generated multiview inputs, ZipSplat produced soft fine-leaf details, overlapping leaf structures, and stretching artifacts despite its observation-grounded representation. TokenBA slightly improved input-view PSNR in the SEVA-based experiment, but this did not translate into equally clear improvements in perceived sharpness or geometric quality; in the Wan-based experiment, refinement introduced more visible grain and transparent overlaps. These observations expose a gap between ZipSplat’s claimed reconstruction benefits and its performance on generated views of complex thin structures: source-patch grounding and token-level refinement were insufficient to preserve both fine detail and clean geometry in our setting.

ZipSplat’s broad claims of observation-grounded reconstruction and cross-dataset generalization need clearer qualification regarding generated inputs and complex thin structures. In our Fern experiments, the released model produced softened leaves, overlapping surfaces, and stretching artifacts, while TokenBA did not consistently improve visual quality despite improving input-view PSNR in the SEVA-based experiment.

Questions for authors

Just need to answer my concerns in Critical Weaknesses

2 comments

Thank you for the review. We are glad that you are satisfied with the quality and speed of our method.

ZipSplat assumes a static scene (see Limitations). The input images must therefore be consistent with one another, and generated views do not meet this assumption. Our tokens merge observations of the same surface across views, and TokenBA fits all input views jointly. Generated views often disagree on fine structures such as leaves. These disagreements then appear as blur, grain or semi-transparent overlaps.

We would also be grateful if you could tell us which parts of the presentation were hard to follow, such that we can improve our submission.

Thanks for the rebuttal. My rating of the clarity is just a typo. I have corrected that. Good Luck.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision uploaded

We have uploaded a revised version of the paper incorporating the discussion so far. Changes are marked in blue in the PDF and summarized here:

  • Tables 1 and 2: added a “Refined” column marking test-time refinement, a ZipSplat row without TokenBA in the feed-forward block, and TokenGS with and without TTO, so that feed-forward and refined results are compared separately (Reviewers PKMu, r43X).

  • Related work (Sec. 2): added SRT and SceneTok and positioned ZipSplat relative to them (Reviewer PKMu).

  • Sec. 3.1: unified the terminology and notation for geometry tokens, scene tokens, and the color embedding (Reviewer r43X).

  • Sec. 4.5 / Table 4: stated explicitly that every ablation row, including the fixed-queries (TokenGS-formulation) row, is retrained on the same DA3-Base backbone with the same budget, and added a retrained “w/o fusion” row (Reviewer r43X).

  • App. D: clarified that the ground-truth 3D-point row of Table 11 is an oracle for merging by 3D position (Reviewer PKMu).

We thank the reviewers for the feedback that led to these changes and remain available for further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.