- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents GS-JEPA, a self-supervised pretraining method for Gaussian Splatting that replaces masked reconstruction of Gaussian attributes with JEPA-style latent prediction. Two variants of the method are presented.
(i) Point-GS-JEPA, where gaussian primitives are grouped into patches, and then passed to a PointNet-based tokenizer to produce patch embeddings. Then, a JEPA-style training is done by predicting representations of masked patches.
(ii) Graph-GS-JEPA, where each Gaussian is treated as a node in a KNN graph with geometric and appearance edge features and uses a Graphormer-style encoder.
The models are pretrained on synthetic objects from ShapeSplat and evaluated on the tasks of object classification and part segmentation.
Critical strengths
- The paper is clear, well-written, and easy to follow. The progression from a simpler baseline (Point-GS-JEPA) to the full method (Graph-GS-JEPA) makes the logical development of the method particularly clear.
- The paper presents an interesting application of JEPA-style objectives to Gaussian representations.
- The experiments are comprehensive and well structured.
- The results showcase the usefulness of the JEPA-style pretraining, particularly under distrubution shifts.
Critical weaknesses
- In table 2, Point-GS-JEPA beats Graph-GS-JEPA on MACGS data under all three protocols (linear evlauation, MLP evaluation, and full fine-tuning). This means that the graph-based method is less transferable than the point-based one.
- The ablation in Table 5 shows limited gains under full fine-tuning. Under linear probing on MACGS the configuration without any edge attributes is best.
- The segmentation advantage of Graph-GS-JEPA cannot be directly attributed to relational modeling, since the two variants also differ in the tokenization procedure, backbone, predictor and parameter count. Without this evidence, the contribution largely reduces to applying Point-JEPA to Gaussian attributes, and the abstract's claims about relational modeling can be seen as overstated.
- My points above lead to the fact that the benefit of graph-based relational modeling is not established.
Questions for authors
- Can the authors provide more intuition in regards to why Graph-GS-JEPA transfer worse than Point-GS-JEPA on MACGS?
- How can one explain the decrease in performance when adding edge attributes under linear probing on MACGS?
- Could the authors report the edge-attribute ablation for segmentation with the Graphormer backbone, as done in Figure 8 for GINE?
- Could the authors evaluate Point-GS-JEPA with smaller patches, to test whether the segmentation gains come from finer granularity rather than relational modeling?
3 comments
Addressing review (2/2)
Q1. "Can the authors provide more intuition in regards to why Graph-GS-JEPA transfer worse than Point-GS-JEPA on MACGS?"
Graph-GS-JEPA explicitly models local Gaussian neighborhoods, which may make its representations more sensitive to changes in Gaussian density, geometry, and reconstruction artifacts between synthetic ShapeSplat and real-world MACGS. Point-GS-JEPA's patch aggregation and global attention may be less sensitive to these local variations. This interpretation is consistent with Table 5: edge attributes are less beneficial on MACGS when the encoder is frozen, while fine-tuning allows their contribution to adapt. Nevertheless, Graph-GS-JEPA outperforms Gaussian-MAE on MACGS with substantially fewer trainable parameters, supporting its value as a compact alternative while highlighting robustness to distribution shift as an opportunity for improvement. We will present this interpretation as a hypothesis.
Q2. "How can one explain the decrease in performance when adding edge attributes under linear probing on MACGS?"
As discussed in W2, edge-dependent features learned on synthetic ShapeSplat may be less reliable on real-world MACGS because local Gaussian relationships differ between the datasets. A frozen encoder cannot adapt these features to the target distribution, whereas full fine-tuning permits adaptation and yields modest improvements. This suggests sensitivity to distribution shift, rather than a consistent benefit from adding edge attributes.
Q3. "Could the authors report the edge-attribute ablation for segmentation with the Graphormer backbone, as done in Figure 8 for GINE?"
The Graphormer segmentation ablation shows that adding distance, direction, and opacity relationships improves both category and instance mIoU. The table below reports the values from the chart; attributes are added cumulatively.
| Edge attributes | Category mIoU (%) | Instance mIoU (%) |
|---|---|---|
| None | 84.15 | 85.59 |
| Distance | 84.65 | 85.79 |
| + Direction | 84.87 | 85.98 |
| + Opacity (Vis) | 85.15 | 86.27 |
| + Scale | 85.00 | 86.11 |
| + Rotation | 84.97 | 86.00 |
| + Appearance | 84.99 | 86.03 |
Distance, direction, and opacity together improve category/instance mIoU by 1.00/0.68 percentage points over the configuration without edge attributes. Further attributes do not improve the result, suggesting that selected local relationships are more useful than including every descriptor. Because the backbone is held fixed, this comparison directly examines the contribution of edge attributes within Graphormer. We will include this table and the updated discussion in the revised manuscript.
Q4. "Could the authors evaluate Point-GS-JEPA with smaller patches, to test whether the segmentation gains come from finer granularity rather than relational modeling?"
We explored the finest-granularity setting, assigning one Gaussian to each Point-GS-JEPA token, but observed representation collapse during pretraining. Thus, increasing granularity alone did not yield a viable alternative under the tested setup. Graph-GS-JEPA retains Gaussian-level tokens while providing explicit neighborhood interactions. This motivates its design, although the collapse at single-Gaussian granularity does not rule out benefits from intermediate patch sizes or establish relational modeling as the sole source of segmentation gains.
Additionally, as requested, we conducted further experiments to investigate the influence of group size on segmentation and MACGS classification using linear probing, while keeping the number of groups fixed.
For MACGS classification, we observe a consistent decrease in accuracy as the group size decreases, potentially due to smaller local receptive fields capturing less geometric context and being more sensitive to noise.
In contrast, the effect on segmentation performance is less straightforward. Reducing the group size from 32 to 16 yields a slight improvement in both mIoU_I and mIoU_C, potentially because smaller local neighborhoods preserve finer-grained geometric details that benefit segmentation. However, further reducing the group size to 8 does not improve performance, suggesting that excessively small neighborhoods may provide insufficient local context. Having in mind all those observations and results we do not believe that high segmentation results on Graph GS-JEPA comes solely from the granular representations, but rather whole architecture and modeling approaches as the whole.
| Group size | mIoU_I | mIoU_C | MACGS-LINEAR-ACC |
|---|---|---|---|
| 8 | 83.60 | 85.44 | 88.06 |
| 16 | 83.97 | 85.71 | 89.98 |
| 32 | 83.85 | 85.56 | 90.52 |
I thank the authors for their detailed response and the additional experiments.
- The ablation and the smaller patch Point-GS-JEPA study are helpful. I also appreciate that the authors will distinguish the evidence for JEPA-style predictive pretraining from the less certain contribution of graph-based relational modeling.
- My mains concerns are reduced but not fully resolved.
- Point-GS-JEPA and Graph-GS-JEPA still differ in tokenization, backbone, and parameter count. As such, the current experiments do not isolate the benefit of relational modeling.
- The provided explanation for the weaker transfer is a plausible hypothesis rather than a controlled finding.
Overall, the rebuttal improves the paper and appropriately narrows its claims, but I would maintain my current assessment.
Addressing review (1/2)
Thank you for your careful analysis of transferability, edge attributes, and the segmentation comparison.
W1. "In table 2, Point-GS-JEPA beats Graph-GS-JEPA on MACGS data under all three protocols (linear evlauation, MLP evaluation, and full fine-tuning). This means that the graph-based method is less transferable than the point-based one."
Both Point-GS-JEPA and Graph-GS-JEPA (Graphormer) outperform Gaussian-MAE on both classification datasets across all three evaluation protocols. With frozen backbones, MACGS linear-probing accuracy improves from 81.55% to 90.52% and 86.67%, respectively. We acknowledge that Point-GS-JEPA performs better on MACGS and position the graph formulation as a parameter-efficient alternative, rather than a uniformly superior replacement. It uses approximately 45% fewer trainable pretraining parameters and achieves stronger segmentation results. We will make this tradeoff explicit in the revised manuscript.
W2. "The ablation in Table 5 shows limited gains under full fine-tuning. Under linear probing on MACGS the configuration without any edge attributes is best."
Table 5 suggests that the benefits of edge attributes depend on the transfer setting and the characteristics of the downstream dataset. Under frozen linear probing, edge-dependent features learned on synthetic ShapeSplat may transfer less effectively to real-world MACGS, explaining why the configuration without edge attributes achieves the best performance.
We hypothesize that this reflects differences in local geometric consistency. Gaussian reconstruction is primarily optimized for rendering quality, so Gaussians representing the same local object structure need not be spatially close. Such relationships may be less consistent in MACGS than in the synthetic pretraining data, making learned edge attributes less reliable under transfer.
Full fine-tuning allows the model to adapt these representations to the target distribution, resulting in modest improvements from edge attributes. In contrast, on ModelSplat40, which is closer to the pretraining distribution, using all edge attributes achieves the best linear-probing performance. We will include this discussion in the revised manuscript, presenting the explanation as a hypothesis and clarifying that removing edge attributes still retains the KNN topology.
W3. "The segmentation advantage of Graph-GS-JEPA cannot be directly attributed to relational modeling, since the two variants also differ in the tokenization procedure, backbone, predictor and parameter count. (...) the abstract's claims about relational modeling can be seen as overstated."
Point-GS-JEPA serves as an architecture-matched baseline for comparison with Gaussian-MAE, while the graph formulation focuses on local relationships between individual Gaussians. Our initial approach used Point-GS-JEPA with one Gaussian per patch. Although this preserved primitive-level granularity, it was computationally expensive and led to representation collapse during pretraining. This motivated Graph-GS-JEPA, which combines fine-grained Gaussian representations with local graph interactions and a cross-attention predictor to reduce predictor attention complexity while preserving primitive-level information.
Our intuition is that Gaussians belonging to different semantic parts may exhibit differences in geometry and appearance. Encoding pairwise attribute differences along graph edges could help identify potential part boundaries and distinguish interactions within a part from those across parts. These cues may benefit segmentation. However, the two formulations also differ in tokenization, backbone, predictor, and parameter count. We will revise the Abstract and segmentation discussion to distinguish this architectural motivation from a causal attribution of the gains solely to relational modeling.
W4. "My points above lead to the fact that the benefit of graph-based relational modeling is not established."
Our results support a compact graph formulation with competitive classification and stronger segmentation among the evaluated Gaussian-based methods. The architecture-matched Point-GS-JEPA/Gaussian-MAE comparison provides clearer evidence for predictive pretraining. We will distinguish these findings from the unresolved contribution of individual graph components.