acceptodds
Under review as a conference paper at ICLR 2027

Learning Predictive Scene Representations from Raw 3D Gaussian Splat Parameters

Abstract

Gaussian splatting has become a common output of scene capture, and representation learning on splats has mainly distilled image features or reconstructed Gaussian parameters. Inferring what a scene hides, or anticipating what an edit will change, requires a representation that is predictable from context. We learn such representations directly from raw Gaussian parameters by predicting the latent representations of masked scene regions from their visible surroundings; images supply training targets only, so inference needs neither photographs nor rendering. Measuring recognition and prediction separately reveals that they are distinct and that image supervision trades one for the other: stronger supervision improves recognition while steadily lowering predictability, and features lifted from photographs, although better recognizers, are far less predictable than ours under matched predictors. Adding relative spatial position to the predictor makes its predictions depend on how a scene is arranged. These representations enable two capabilities: predicted embeddings retrieve real Gaussian groups from other scenes to fill hidden regions, and an action-conditioned latent world model, trained separately on synthesized edits, predicts how a scene's representation changes when objects are removed, moved, or added. Its removal predictions improve steadily with more training scenes, and searching over its predictions recovers the edits that produced a goal scene far more often than chance. Captured splats can thus be used to predict, not only to render.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
3D Gaussian splatting, self-supervised learning, joint-embedding predictive architecture, 3D scene representation learning, latent world models, action-conditioned prediction, 3D scene understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SMUVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates predictive representation learning directly from raw 3D Gaussian scene parameters. A shared encoder learns contextual embeddings through masked latent prediction, distributional regularization, and image-feature distillation routed to appearance tokens. The paper studies the relationship between recognition and predictability, introduces predictor-side 3D rotary positional embeddings, and demonstrates retrieval-based filling and synthetic edit prediction. Experiments show that stronger image supervision improves semantic recognition while reducing latent predictability, whereas 3D rotary positional encoding substantially improves spatially sensitive prediction.

Critical strengths
  1. Studying whether Gaussian scene representations can be inferred from surrounding context provides a useful perspective.
  2. The authors provide substantial analysis, including reconstruction controls, fresh matched predictors, multiple null baselines, collapse diagnostics, and pretraining-exposure analyses.
Critical weaknesses
  1. Insufficient benchmarking against independent methods. The range of independent representation-learning baselines is limited, with SceneSplat serving as the main external learned-representation baseline. Much of the experimental comparison concerns variants of the proposed framework, including different tokenizations, regularizers, distillation configurations, token counts, and positional encodings. These experiments are primarily ablations: they help justify individual design choices but do not sufficiently establish the method’s advantage over other representation-learning approaches.
  2. Although the paper reports linear-probe evaluations of the encoder features, the practical utility of the predicted hidden-region embeddings remains insufficiently demonstrated. Additional evaluation on tasks such as semantic segmentation of masked regions, semantic prediction evaluated from held-out views, or scene completion would provide stronger evidence. For example, investigating whether the predicted embeddings can be decoded into Gaussian geometry and appearance attributes could help establish their usefulness for scene completion and extension.
  3. The results in Figure 2 remain visually fragmented even when retrieval uses the oracle hidden embeddings. Consequently, performance close to this oracle does not establish coherent scene completion. Group-level class agreement should be complemented by evaluations of geometric consistency and rendering quality.
Questions for authors
  1. Could the authors compare against additional independent representation-learning methods beyond SceneSplat, using matched inputs and supervision?
  2. How well do the predicted hidden-region embeddings support downstream tasks such as semantic segmentation and scene inpainting? How do they compare with relevant task-specific baselines under a common evaluation protocol?
  3. Could the authors provide clearer qualitative results and quantitative evaluations for downstream segmentation or inpainting? Such evidence would be more convincing than the fragmented retrieval-based examples in Figure 2.
1 comment

Thank you for pressing on independent baselines and on what the predicted embeddings are good for.

On benchmarking against independent methods. Table 3 (bottom) trains a fresh predictor on each frozen encoder under one protocol. It scores the probe on 80 unseen rooms and the scene-mean margin on 20 unseen rooms. The encoders are Sonata, Concerto, Utonia, Chorus, SceneSplat and lifted DINOv2, plus Sonata's recipe retrained on our 1056 scans. Table 34 extends this to 14 rows, including the Point-MAE recipe retrained on the same scans.

Each row states its exposure. Sonata, Concerto and Utonia pretrained on scans that include our evaluation rooms. Chorus's recipe excludes them, and the retrained recipes saw neither.

  • No public checkpoint lies above the curve fitted through our sweep (0 of 5, pre-registered). Released Sonata lies on it (excess +0.013 [−0.059, +0.080], 3 predictor seeds), and Concerto lies below it (−0.178 [−0.234, −0.123]).
  • Utonia, Chorus, SceneSplat and lifted DINOv2 recognize more and predict less. This is the exchange our own sweep measures (Section 5).
  • Read per route, our image-taught appearance route (probe 0.466 ± 0.008, 3 encoder seeds) exceeds released Sonata's margin by +0.103 [+0.043, +0.169] (13 of 20 rooms; Table 34). Sonata saw these rooms in pretraining, and our encoder did not.

Section 5 calls the fitted curve a weak bar, so we also read the retrained recipes against our trained points. The retrained Sonata recipe lies on the line between our α=0 and α=1 encoders (+0.012 [−0.061, +0.079], 3 encoder seeds). The retrained Point-MAE recipe reaches its high margin at untrained-level recognition (probe 0.120 ± 0.017, n=3; Table 34).

The Point-MAE and Gaussian-MAE objective also runs on our own tokenizer and encoder as the reconstruction control (Table 3, top). All four released scene encoders read their own Gaussians under member replacement (Table 5). Released encoders recognize better (Section 5). Our claim concerns predictability and how to account for it.

On the downstream use of predicted hidden-region embeddings. Section 7 evaluates semantic labelling of masked regions. A head refitted on predicted hidden tokens beats the model's own trained position-only predictor, which receives the same hidden positions (+0.101 mIoU, 3 of 3 seeds, Holm p=0.0003, 62 recipe-selection rooms; Table 6). Section 7 also states that supervised heads score higher. Against a nearest-visible label copy, the position-only predictor gains as much as the model, so this gain is measured over positions.

Inference beyond a copy is tested separately. A copy-plus-residual predictor starts from a label-free copy and learns a correction. It improves on that copy in 3 of 3 seeds on fresh synthetic rooms:

  • +0.0069 (lower bound +0.0059) on 30 InteriorGS rooms;
  • +0.0032 (lower bound +0.0023) on 40 Structured3D rooms (Table 6).

The gains are small. The scrambled-donor control in Section 9 is the next step.

The second use is the edit model of Section 8. Beyond a position-and-mask model, scene content adds +0.104 FVE, and prediction is 11.0× faster than re-encoding (178 edits, 15 rooms; Table 7).

On decoding, rendering quality and Figure 2. We agree that near-oracle retrieval does not establish coherent completion. The retrieval figure has left the main text, and Figure 2 now shows member replacement and the layout share. The model decodes nothing, so PSNR and geometric-consistency metrics do not apply to the current results (Section 5).

Appendix A.1 reports the attribute-level readout: predicted floor attributes are level with a copy of nearby floor. Section 9 specifies the next step, which decodes predicted tokens into Gaussians and scores the rendered hidden region with PSNR and SSIM.

Appendix Figure 8 gives a qualitative example. For a hidden bed (222 groups), the predicted embeddings reach a mean cosine of 0.87, and 203 of the 222 groups are closer to their own target than to any other. As the caption states, this room is in the model's checkpoint-selection split.

Reviewer_FpkQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies predictive representation learning for scenes represented by 3D Gaussian splats. Local groups of raw Gaussian parameters are encoded into tokens, and a JEPA-style predictor estimates masked groups' embeddings from visible context at supplied query positions. The main model uses geometry/appearance token pairs, routes image-feature supervision to appearance tokens, and uses distributional regularization instead of a momentum teacher. A separate action-conditioned model learns to predict changes in the frozen representation under synthetic object edits. The experiments examine semantic recognition, contextual predictability, fragment retrieval, transplanted-object detection, and edit search.

The central empirical argument is that semantic recognition and contextual predictability are distinct properties: increasing image supervision improves recognition while reducing predictability in the tested configurations. Several controlled findings are supported, but their implications for meaningful applications and advantages over existing methods remain less clear.

Critical strengths
  1. The paper addresses a relevant question about the relationship between semantic recognition and contextual predictability in Gaussian-scene representations.

  2. The image-supervision sweeps, shuffled-teacher controls, and fresh predictors provide useful evidence about the recognition-predictability trade-off in the tested configurations.

  3. The context-shuffling tests reveal the reference predictor's limited sensitivity to arrangement and show that rotary position encoding improves sensitivity to this perturbation.

Critical weaknesses
  1. Image-feature distillation already provides semantic features for Gaussian scenes, and the paper does not establish which practical task improves by adding predictive training. Better prediction of the model's own embeddings does not answer this question. The contribution beyond earlier point-cloud and Gaussian-splat JEPAs also needs clarification.

  2. The missing-region predictor receives the hidden groups' actual centers, which may reveal the removed object's footprint, while overlapping groups can leave some constituent Gaussians visible. Predicting a chair near a desk could also rely on object co-occurrence. These cues make it difficult to determine how much spatial understanding the task requires.

  3. Retrieval copies separate Gaussian fragments into supplied locations and produces visibly fragmented fills. In the reference space, instance-level class agreement is 0.204 versus 0.188 for the permuted control (p=0.58; Appendix F). This provides limited evidence that the method can recover a missing object or complete a scene coherently.

  4. The detection experiment labels foreign objects as positives even when they are placed plausibly on the floor. Why should a chair imported from another office be anomalous? The experiment does not clearly separate inappropriate placement from donor reconstruction differences, artifacts from overwriting host groups, or room familiarity. This distinction should be explicit in the main text.

  5. Removing a bed's Gaussians does not reconstruct the floor that the bed concealed. The edit model predicts changes in encoder features, including substantial changes caused by coordinate normalization, rather than demonstrating this geometric outcome. Complete ground-truth geometry or consistent before/after captures would provide a more meaningful test of physical edit effects.

Questions for authors
  1. What downstream advantage over image-distilled or reconstruction-trained representations is supported by the reported results? Please clarify which contributions beyond existing 3D JEPA methods account for that advantage, distinguishing improved embedding predictability from demonstrated application benefits.

  2. To what extent do the reported controls distinguish contextual inference from information supplied by masked-group centers and overlapping Gaussian groups? What evidence supports sensitivity to spatial relationships beyond common object associations, such as desks and chairs?

  3. What does anomalous mean in the transplanted-object experiment, given that positives include plausibly placed objects? Please clarify how the existing results distinguish inappropriate placement from foreign reconstruction statistics, insertion artifacts, or familiarity with the host room, and which explanations remain unresolved.

  4. Is the edit model intended to predict the encoder's response to synthetic edits or the consequences of physical scene changes, such as revealing the floor beneath a removed bed? Please clarify how coordinate-normalization effects limit this interpretation and what the reported results establish about the benefit of latent edit search over direct editing and re-encoding.

1 comment

Thank you for a detailed review. Its questions drove the largest changes in the revision.

On the advantage over distillation, reconstruction and earlier 3D JEPAs. Recognition comes from the image teacher (Section 5): distillation alone probes 0.463 ± 0.013 (80 unseen rooms, n=3). Predictive training adds predictability from context. With tokenizer, encoder and fresh predictor held fixed, the scene-mean margin is 0.780 ± 0.002 for latent prediction. Reconstruction reaches 0.486 ± 0.007, distillation 0.380 ± 0.015, and distillation plus JEPA 0.504 ± 0.004 (3 predictor seeds, one encoder per recipe; Table 3, top).

Two uses carry this into applications. First, predicted tokens label hidden regions better than the model's own trained position-only predictor (+0.101 mIoU, 3 of 3 seeds, Holm p=0.0003, 62 recipe-selection rooms; Table 6). Second, distill-only features barely learn edits (removal FVE 0.024, 15 rooms). Their class-named utility (0.147) is below every space that combines masked prediction with an image teacher (30 rooms; Figure 6b).

Beyond earlier 3D JEPAs, we contribute the captured multi-room setting, together with the accounting and the member-replacement test that separate layout from content (Sections 2 and 6.3). The trained position-only null and member replacement on the split of Table 3 (top) are the next study (Section 9).

On supplied centres, overlap and co-occurrence. Section 7 states that every test supplies the footprint, and it scores each result against a baseline given the same information.

  • On 285 non-shell objects (20 rooms, 3 seeds), a trained position-only predictor carries 0.746 [0.703, 0.787] of the margin at α=0 and 0.5001 [0.442, 0.556] at α=2.5 (Section 6.2).
  • With the footprint removed (centroid query, 0.5 m lattice, occupied cells), the predictor beats a position-only predictor given the same query in 9 of 9 pairs (Section 7).
  • Overlapping groups share 4.4% of hidden member slots (0.1–9.3% per room). On hidden groups that share none, the target-recompute check is unchanged (−0.478 against −0.468; Table 29).
  • Re-arranging the visible objects, with class composition kept, lowers the prediction by 0.038–0.047 (75 rooms). A room's own context beats a donor room's by +0.170 to +0.240 (80 rooms). Both hold in 3 of 3 seeds (Appendix E.3).

The desk-to-chair partner test is unresolved because its control was mostly another chair (189 chairs; Appendix E.3). A control drawn from another object class is the fix. On this evidence we claim sensitivity to placement.

On retrieval. We agree, and retrieval has left the main text. Section 7 now tests inference beyond a copy directly. A copy-plus-residual predictor starts from a label-free copy and learns a correction. On fresh synthetic rooms it beats that copy in 3 of 3 seeds: +0.0069 (lower bound +0.0059, 30 rooms) and +0.0032 (lower bound +0.0023, 40 rooms; Table 6). The gains are small, and copies that use labels remain stronger. The scrambled-donor control (Section 9) is the next test.

On transplanted-object detection. Its positives included plausibly placed objects. The test therefore could not separate placement from donor reconstruction statistics, insertion artefacts or room familiarity. We removed it and make no anomaly claim. The controls above now test placement directly.

On what the edit model predicts. It predicts the encoder's response to a specified, located, synthesized edit. Section 8 states that a removal does not reveal the occluded surface. Fixed-frame targets remove the coordinate-normalization artefact (Appendix C.2).

Section 8 leads with a capacity-matched model given only positions and the edit mask. That model explains 0.464 of a removal's change, against 0.567 for the edit model, so scene content adds +0.104 FVE. This comparison uses one reference encoder, on rooms in the encoders' checkpoint-selection split (Table 7).

Compared with editing and re-encoding, the gain is cost. Prediction is 11.0× faster per edit (178 edits, 15 rooms) and 3.1× faster per search. A full 220-set search takes 7.7 s, against 23.8 s with re-encoding and about 41 minutes from raw Gaussians (Appendix G.1). Three-edit search recovers the exact set in 0.233 [0.183, 0.287] of 300 trials, against 0.043 for a heuristic (30 encoder-seen rooms). On encoders with no gradient on those rooms, it replicates at α=0 but not at α=1 (Table 7).

Physical consequences are the next step. Real re-captures have not yet passed the fidelity gate (1 of 86 pairs; Appendix A.1). Section 9 specifies edit models on real re-captures and on encoders unexposed to the edited rooms.

Reviewer_i5mtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper learns predictive representations directly from 3D Gaussian Splatting parameters using a JEPA-based framework. It studies the trade-off between semantic recognition and context prediction, introduces spatially aware prediction, and further applies the learned representations to action-conditioned latent world modeling.

Critical strengths
  1. The analysis of the trade-off between recognition and predictability is particularly valuable. The paper demonstrates that stronger image supervision can improve semantic recognition while reducing the predictability of the learned representations, providing useful insights into representation learning for 3D Gaussian scenes.
  2. The paper provides a broad range of experiments, including comparisons with parameter reconstruction, image-supervision ablations, different tokenization strategies, spatial prediction analysis, masked-region retrieval, and action-conditioned scene editing.
  3. The authors include several meaningful baselines and controlled evaluations. The discussion of limitations, particularly regarding coordinate normalization and real-scene transfer, also helps clarify the scope of the reported results.
Critical weaknesses
  1. JEPA-based prediction has already been explored for object-level Gaussian splats [43] and scene-level point clouds [40]. Merely extending it to scene-level Gaussian splats appears to offer limited methodological novelty.
  2. The masked JEPA predictor and the action-conditioned world model are trained separately. Although the world model uses JEPA-pretrained features, it remains unclear how masked prediction specifically benefits downstream world modeling. Comparisons with alternative pretrained representations or joint training with the encoder would help clarify this connection.
  3. The use of 3D rotary positional embeddings is a relatively standard design choice. Although the experiments demonstrate meaningful improvements in spatial sensitivity, this component appears more suitable as an effective architectural refinement than as an independent major methodological contribution.
Questions for authors
  1. Could the authors clarify the main methodological novelty compared with existing JEPA-based methods for point clouds and 3D Gaussian representations?

  2. How does masked JEPA pretraining specifically benefit the downstream world model, given that the two prediction tasks are trained separately?

  3. Is the recognition–predictability trade-off a general phenomenon, or does it mainly depend on the training objectives and supervision settings?

1 comment

Thank you for the careful reading and for three precise questions.

On novelty relative to point-cloud and Gaussian JEPAs. Section 2 now states plainly that latent prediction on Gaussian groups is not new, and cites Gaussian-JEPA, Point-JEPA and SR-JEPA. Our contribution lies in the setting of captured multi-room splats and in what that setting requires. A scene predictor is queried at hidden-group centres. Unlike positions on a 2D patch grid, these centres differ from scene to scene, so a high prediction score can come from layout alone (Section 1). We supply the evaluation that separates layout from content. Every score is read against a trained position-only predictor and a nearest-visible copy (Sections 4 and 6.2). Member replacement tests whether each token depends on its own Gaussians, read beside the same architecture at initialization (Section 6.3).

That test found that 20 of 24 checkpoints of our original image-free recipe ignore their Gaussians while passing rank screens. The same architecture at initialization does read them (centred cosine 0.365 ± 0.027, n=3; Table 5). Independent plain code reproduces the collapse on real Gaussian rooms when the predictor receives the exact target centres (3 of 3 seeds), and withholding the centres prevents it (3 of 3; Appendix F.2). Our remedy is a per-token content target (Section 3), and Table 4 is retrained with it. Section 9 turns this into a reporting recommendation for any model whose tokens or predictor receive positions.

On how masked pretraining benefits the separately trained edit model. The edit model amortizes a fixed map: the frozen encoder's re-encoding of an edited room (Appendix C.2). Joint training would move the target the model is learning. The pretrained representation then determines how useful the learned transitions are. Spaces that combine masked prediction with an image teacher give more class-named utility than either an untrained encoder or distillation alone (Section 8, Figure 6b; 30 rooms, n=3 seeds per space). Distill-only features barely learn edits (removal FVE 0.024) and deliver 0.147. The untrained encoder is the most learnable (0.932) but gives only 0.401.

At fixed content, utility rises with edit-model training (+0.326 [+0.264, +0.387], 28 of 30 rooms better, 3 seeds; Appendix E.1). Against independent representations, the reference space is more learnable than Sonata's and Concerto's under a like-for-like ceiling (15 of 15 rooms each, one run each; Appendix G.1). This comparison is a post-read design, disclosed in Table 12. Edit models on encoders unexposed to the edited rooms are the next step named in Section 9.

On 3D rotary position as a contribution. We agree. Section 2 claims no method contribution for it, and we report only its effect. The shuffled-context margin rises from 0.013 ± 0.002 to 0.624 ± 0.021 (20 unseen rooms, n=3). The rotary predictor beats the nearest-visible copy in every size bin and seed (+0.276 ± 0.009, n=3, recipe-selection rooms; Table 6). We read this as sensitivity to placement (Section 7).

On whether the trade-off is general. Its sign in the training losses follows from weighting two losses (Proposition 1, Appendix B.2). The empirical finding is that the trade-off appears in readouts that are not training losses, on rooms that none of our compared models trained or was selected on. The recognition side comes from the teacher's content rather than its weight: a teacher shuffled within each room keeps the real teacher's margins to within 0.005 at less than half its probe (Table 16).

The trade keeps its direction in three further settings:

  • across encoder sizes, with the exchange rate unchanged from ViT-S to ViT-L (ratio 0.92 [0.84, 1.01] and 1.03 [0.96, 1.09], n=3 per arm; Appendix E.2);
  • with a rotary predictor (Appendix E.2);
  • in both directions on ARKitScenes and ScanNet++ (Table 33).

Under trained nulls, what the teacher removes is the positional part. The margin beyond a trained position-only predictor is 0.055 at α=0 and 0.080 at α=2.5 (n=3, descriptive; Table 4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.