- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates predictive representation learning directly from raw 3D Gaussian scene parameters. A shared encoder learns contextual embeddings through masked latent prediction, distributional regularization, and image-feature distillation routed to appearance tokens. The paper studies the relationship between recognition and predictability, introduces predictor-side 3D rotary positional embeddings, and demonstrates retrieval-based filling and synthetic edit prediction. Experiments show that stronger image supervision improves semantic recognition while reducing latent predictability, whereas 3D rotary positional encoding substantially improves spatially sensitive prediction.
Critical strengths
- Studying whether Gaussian scene representations can be inferred from surrounding context provides a useful perspective.
- The authors provide substantial analysis, including reconstruction controls, fresh matched predictors, multiple null baselines, collapse diagnostics, and pretraining-exposure analyses.
Critical weaknesses
- Insufficient benchmarking against independent methods. The range of independent representation-learning baselines is limited, with SceneSplat serving as the main external learned-representation baseline. Much of the experimental comparison concerns variants of the proposed framework, including different tokenizations, regularizers, distillation configurations, token counts, and positional encodings. These experiments are primarily ablations: they help justify individual design choices but do not sufficiently establish the method’s advantage over other representation-learning approaches.
- Although the paper reports linear-probe evaluations of the encoder features, the practical utility of the predicted hidden-region embeddings remains insufficiently demonstrated. Additional evaluation on tasks such as semantic segmentation of masked regions, semantic prediction evaluated from held-out views, or scene completion would provide stronger evidence. For example, investigating whether the predicted embeddings can be decoded into Gaussian geometry and appearance attributes could help establish their usefulness for scene completion and extension.
- The results in Figure 2 remain visually fragmented even when retrieval uses the oracle hidden embeddings. Consequently, performance close to this oracle does not establish coherent scene completion. Group-level class agreement should be complemented by evaluations of geometric consistency and rendering quality.
Questions for authors
- Could the authors compare against additional independent representation-learning methods beyond SceneSplat, using matched inputs and supervision?
- How well do the predicted hidden-region embeddings support downstream tasks such as semantic segmentation and scene inpainting? How do they compare with relevant task-specific baselines under a common evaluation protocol?
- Could the authors provide clearer qualitative results and quantitative evaluations for downstream segmentation or inpainting? Such evidence would be more convincing than the fragmented retrieval-based examples in Figure 2.
Thank you for pressing on independent baselines and on what the predicted embeddings are good for.
On benchmarking against independent methods. Table 3 (bottom) trains a fresh predictor on each frozen encoder under one protocol. It scores the probe on 80 unseen rooms and the scene-mean margin on 20 unseen rooms. The encoders are Sonata, Concerto, Utonia, Chorus, SceneSplat and lifted DINOv2, plus Sonata's recipe retrained on our 1056 scans. Table 34 extends this to 14 rows, including the Point-MAE recipe retrained on the same scans.
Each row states its exposure. Sonata, Concerto and Utonia pretrained on scans that include our evaluation rooms. Chorus's recipe excludes them, and the retrained recipes saw neither.
Section 5 calls the fitted curve a weak bar, so we also read the retrained recipes against our trained points. The retrained Sonata recipe lies on the line between our α=0 and α=1 encoders (+0.012 [−0.061, +0.079], 3 encoder seeds). The retrained Point-MAE recipe reaches its high margin at untrained-level recognition (probe 0.120 ± 0.017, n=3; Table 34).
The Point-MAE and Gaussian-MAE objective also runs on our own tokenizer and encoder as the reconstruction control (Table 3, top). All four released scene encoders read their own Gaussians under member replacement (Table 5). Released encoders recognize better (Section 5). Our claim concerns predictability and how to account for it.
On the downstream use of predicted hidden-region embeddings. Section 7 evaluates semantic labelling of masked regions. A head refitted on predicted hidden tokens beats the model's own trained position-only predictor, which receives the same hidden positions (+0.101 mIoU, 3 of 3 seeds, Holm p=0.0003, 62 recipe-selection rooms; Table 6). Section 7 also states that supervised heads score higher. Against a nearest-visible label copy, the position-only predictor gains as much as the model, so this gain is measured over positions.
Inference beyond a copy is tested separately. A copy-plus-residual predictor starts from a label-free copy and learns a correction. It improves on that copy in 3 of 3 seeds on fresh synthetic rooms:
The gains are small. The scrambled-donor control in Section 9 is the next step.
The second use is the edit model of Section 8. Beyond a position-and-mask model, scene content adds +0.104 FVE, and prediction is 11.0× faster than re-encoding (178 edits, 15 rooms; Table 7).
On decoding, rendering quality and Figure 2. We agree that near-oracle retrieval does not establish coherent completion. The retrieval figure has left the main text, and Figure 2 now shows member replacement and the layout share. The model decodes nothing, so PSNR and geometric-consistency metrics do not apply to the current results (Section 5).
Appendix A.1 reports the attribute-level readout: predicted floor attributes are level with a copy of nearby floor. Section 9 specifies the next step, which decodes predicted tokens into Gaussians and scores the rendered hidden region with PSNR and SSIM.
Appendix Figure 8 gives a qualitative example. For a hidden bed (222 groups), the predicted embeddings reach a mean cosine of 0.87, and 203 of the 222 groups are closer to their own target than to any other. As the caption states, this room is in the model's checkpoint-selection split.