- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ZipSplat, a feed-forward model that reconstructs 3D scenes from unposed images as a compact set of observation-grounded scene tokens, each of which is decoded into a set of freely positioned Gaussian primitives. Unlike previous fixed-query representations, the proposed formulation supports an adjustable reconstruction budget at inference time without requiring retraining. In addition, the paper introduces TokenBA, an efficient token-level bundle adjustment algorithm that exploits the resulting 2D–3D correspondences to jointly refine scene tokens and camera poses using reprojection and photometric consistency, requiring only a few seconds of optimization.
Critical strengths
- The paper presents an effective approach for predicting compact Gaussian primitives. Compared with previous fixed-query methods, the proposed representation maintains an explicit connection to the input observations while allowing the reconstruction budget to be flexibly adjusted according to the complexity of the scene.
- ZipSplat consistently outperforms competing methods while using significantly fewer Gaussian primitives, demonstrating a favorable trade-off between reconstruction quality and representation efficiency.
Critical weaknesses
- The method description could be made clearer in several places. For example, in Lines 178–179 (“These are merged into geometry tokens ”), it is unclear how the multi-layer visual tokens are merged into the geometry tokens. Furthermore, in Lines 180 and 182, the paper states that “Each geometry token serves as a scene token” and later refers to “each scene token .” Are geometry tokens and scene tokens the same concept? If so, using different terminology and notation is confusing. However, in Line 213, the statement “Each scene token is initialized as the mean of its cluster members” suggests a different definition of scene tokens, making the relationship between these concepts even less clear. Moreover, in Line 234, the term color features is introduced without explanation. Clarifying these definitions and maintaining consistent terminology throughout the paper would significantly improve readability.
- The comparisons in Tables 1 and 2 could be made fairer. The reported results include TokenBA, an additional optimization step that improves performance but also incurs extra computation time. To better separate the contributions of the feed-forward model and the subsequent optimization, the authors should report the performance and runtime of the proposed method both with and without TokenBA, and compare them against the corresponding performance and runtime of all baseline methods.
- The backbone comparison also raises a fairness concern. The proposed method is built upon the pretrained DA3-Giant backbone, whereas TokenGS is not initialized from a pretrained 3D foundation model. Since the pretrained backbone likely provides a substantial advantage, it would be helpful to either discuss this limitation explicitly or include comparisons under more comparable initialization settings.
- The paper should include a comparison with a simple k-means baseline without token fusion to validate the effectiveness of the proposed token fusion strategy.
Questions for authors
The current clustering step is non-differentiable. Have the authors explored differentiable alternatives that would make the entire pipeline end-to-end trainable? A discussion or ablation on this design choice would be valuable.
4 comments
Fusion ablation (follow-up)
Following up on the fusion comparison: we have now retrained the model without token fusion, i.e. each scene token is the plain cluster mean of its member geometry tokens and is decoded directly, without Eqs. (4) and (5). All other components and the training budget are unchanged. The result is included as the last row of Table 4 in the revision. On DL3DV at 6 views, removing fusion reduces PSNR from 24.34 to 22.59 dB at r=1 and from 23.25 to 21.22 dB at r=0.5. The retrained number supersedes the inference-only test from our earlier reply (14.0 dB), which removed fusion from a model trained with it. The retrained variant confirms that fusion contributes about 1.8 to 2.0 dB, with the larger loss under compression, where each merged token must represent several patches.
We hope this is the comparison you had in mind and are happy to extend it further.
Thanks for the authors' detailed explanations. The new experiments have carefully addressed my concerns. I have just one remaining question for clarification: Does the method support training from scratch without relying on pretrained priors?
Thank you for the question. In principle yes: no component of ZipSplat assumes a pretrained backbone, and the encoder is trained end-to-end with the rest of the model. In practice, pose-free reconstruction requires learning multi-view correspondence and camera estimation from images alone, which is why all pose-free baselines (AnySplat, NoPoSplat, YoNoSplat) also start from a 3D foundation model. Posed methods such as TokenGS are given ground-truth cameras, so they skip this part and can train from scratch. Training ZipSplat from scratch would mean re-learning that geometric prior at foundation-model scale in compute and data. Table 4 confirms the gains do not come from the backbone, as all rows share it.
Thank you for the detailed review. We are glad that the explicit link to the input observations and the adjustable budget came across. Below we address each point:
Terminology: Thank you for pointing these out. We will make the definitions more clear in Sec. 3.1 and 3.2 of the revision. To clarify:
Appendix A.1 gives the full construction, and we are happy to clarify further if anything remains unclear to improve our submission.
TokenBA Results and Timings: Thank you for this feedback. The matched comparisons are reported in App. F (Tables 12 to 15). Without TokenBA, ZipSplat still leads all feed-forward baselines in PSNR and LPIPS on every benchmark. At 16 views it takes 0.19 s feed-forward and 1.8 s with TokenBA, against 6.3 s for TokenGS with its optimization (Fig. 10, App. C.1, App. E). To improve clarity in the main paper, we will add a column to Tables 1 and 2 that marks which methods use test-time refinement. We will also add a ZipSplat row without TokenBA.
Backbone: Table 4 should address this already: Its “Fixed queries” row retrains the TokenGS formulation on the same pre-trained backbone as ours (DA3-Base). It uses learned queries, their frustum loss and the same training budget. On DL3DV at 6 views it reaches 23.02 dB with 262K Gaussians, while ZipSplat without TokenBA reaches 24.34 dB with 62K and still 23.25 dB with 31K. The gain therefore does not come from the backbone but can be attributed to our formulation. We will make this more explicit for the revision.
Fusion Ablation: Token fusion gives each merged token access to its individual observations. Without it, which we read as decoding the cluster means directly (without Eq. 4 and 5), each scene token is a plain average of its member patches. As a first check, we removed fusion from the trained model at inference. On DL3DV at 6 views, PSNR drops from 24.3 to 14.0 dB and LPIPS rises from 0.17 to 0.76. This inference-only test shows that the decoder relies on the fused tokens. Please let us know if this is the comparison you have in mind.
Differentiable Clustering: Yes, during development we tested soft k-means on an earlier, smaller model. It was slightly below hard assignment at full capacity and under compression. A soft assignment blends tokens even when each cluster is a single patch, so each token is always a mixture. Our pipeline is still trained end to end, since each scene token is the differentiable mean of its members. Our GeoCE loss reaches the same goal, a token space suited for clustering, as it shapes the token space directly (App. D.1).
Thank you again for the constructive feedback. We are happy to discuss any of these points further.