- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OneField, a framework that aligns NeRF and 3D Gaussian Splatting (3DGS) into a shared embedding space. It employs two format-specific encoders, SigLIP and VICReg alignment losses, and a shared reconstruction decoder. Experiments on approximately 300K paired objects demonstrate effective cross-format transfer in classification, segmentation, retrieval, and language-based tasks.
Critical strengths
Strong experimental results. OneField demonstrates consistent cross-format transfer across multiple downstream tasks, with minimal performance degradation. The mixed-format retrieval results are particularly convincing. Large-scale dataset and useful analysis. The paired NeRF–3DGS dataset and the robustness experiments provide valuable resources and evidence for cross-format alignment.
Critical weaknesses
Missing unified representation baseline. Both NeRF and 3DGS describe spatially queryable fields. A simpler alternative would sample comparable spatial attributes from both representations and process them using a shared Point Transformer. Without this baseline, the necessity of separate encoders remains unclear. Lack of parameterization invariance analysis. Function-preserving permutations of NeRF hidden neurons can change the input weight coordinates without changing the underlying radiance field. The proposed encoder does not guarantee invariance to these transformations, raising concerns about whether it learns field properties or parameter-specific patterns. Potential bias in reconstruction supervision. Both branches use NeRF-rendered images as reconstruction targets. This may bias the shared embedding toward NeRF-specific information. The effect of symmetric or ground-truth reconstruction supervision remains unexplored.
Questions for authors
- How does OneField compare with a unified field-sampling baseline using a shared 3D encoder?
- Does the NeRF embedding remain stable under function-preserving hidden-neuron permutations?
- How does using format-specific or ground-truth reconstruction targets affect cross-format alignment?
2 comments
Update on Q3.
Training of the new baseline with ground-truth reconstruction targets is now complete, and we evaluated both models on the test set with the protocols of the paper. The final results confirm what we observed during training: the effect on cross-format alignment is negligible. The cosine similarity between the NeRF and 3DGS embeddings of the same object is practically unchanged (0.961 for OneField, 0.962 with ground-truth targets). Cross-format retrieval R@1 changes slightly, from 92.3 to 92.9 (NeRF→3DGS) and from 91.6 to 92.5 (3DGS→NeRF). Classification accuracy changes by at most 0.5 points. With the head trained on NeRF embeddings, OneField scores 93.6 on NeRF and 93.5 on 3DGS, against 93.5 and 94.0 with ground-truth targets. With the head trained on 3DGS embeddings, OneField scores 93.6 and 93.7, against 93.6 and 94.0. Ground-truth images are therefore not needed to reach the same alignment, and OneField can be trained directly on existing NeRFs and 3DGS.
Q1: Unified field-sampling baseline
We thank the reviewer for this suggestion. We trained a new field-sampling baseline that uses a shared 3D encoder for both NeRF and 3DGS. Both representations are queried as spatial fields, in the same frame, on a shared 16×16×16 grid covering the object, for a total of 4,096 cells. Each cell is subdivided into a regular 4×4×4 sub-grid, and both the NeRF and the 3DGS are queried at these same 64 locations. This gives one point per cell. Both point clouds are encoded by a single shared Point Transformer [1]. We treat the output of this Transformer as the output of our Attentive Poolers. Everything else is identical to our method: data and splits, the SigLIP and VICReg losses, the decoder with its rendering target, and the training schedule.
Appendix 14 of the revised paper discusses the results of this experiment.
The field-sampling baseline performs worse than OneField, across all the evaluated tasks (retrieval, classification, and segmentation). While on classification and segmentation, the gap with OneField is around 5 points, the baseline fails on retrieval, exhibiting a performance degradation of about 39 points.
The baseline never accesses the source NeRF and 3DGS representation, relying instead on a coarse point cloud sampled from each of the input formats. The single 3D encoder fails to produce informative embeddings of the input object, proving the necessity of the separate encoders adopted by OneField.
[1] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 16259–16268, 2021
Q2: Function-preserving hidden-neuron permutations
Like nf2vec, our encoder processes the raw weights of a fixed MLP. The NeRFs we use were all trained from one shared random initialization. Hidden-neuron permutations do not occur in the training or test data.
To measure the sensitivity of OneField to hidden-neuron permutations, we apply random function-preserving permutations to the NeRFs of the ShapeNet test set. Permuting a single hidden layer gives an average cosine similarity of 0.950.05 to the clean embedding, and classification drops slightly, from 93.56% to 92.80±0.1%. Permuting the first two layers gives a cosine of 0.860.10 and 89.70±0.2% classification accuracy. Permuting all three hidden layers gives a cosine similarity of 0.640.18 and 88.6±0.3% classification accuracy.
Retrieval is more sensitive than classification: NeRF to 3DGS R@1 drops from 92.3 to 81.8 (one layer), 60.0 (two layers) and 20.1 (three layers).
Appendix A.15 of the revised paper provides this analysis.
Q3: Format-specific vs ground-truth reconstruction targets
We thank the reviewer for this question. Our work assumes that the original training views of the NeRF and 3DGS are not available: OneField is meant to encode NeRFs and 3DGS as they are, without relying on any external data. For this reason, we take the reconstruction target from the input representation rather than from the original ground-truth images. Nonetheless, we conducted an additional experiment to test how ground-truth reconstruction targets affect cross-format alignment. We are training OneField from scratch using the original ground-truth images as reconstruction targets for both branches, instead of the NeRF renderings. Everything else is kept identical: architecture, losses, data and schedule. Ground-truth images are not specific to either format, so this setting removes the potential bias of the reconstruction supervision toward NeRF-specific information.
The current model is still training, and the results below compare OneField with this new baseline, evaluating them on the validation set and considering the same epoch for both (current epoch: 165/200). For this reason, the numbers for OneField are different from those in the paper.
So far, the effect on cross-format alignment is negligible. The cosine similarity between the NeRF and 3DGS embeddings of the same object is exactly the same, 0.963. Cross-format retrieval R@1 changes from 93.5 to 93.9 (NeRF→3DGS) and 93.3 to 93.7 (3DGS→NeRF). This is because our shared decoder is not intended to provide high-fidelity reconstructions, but rather to make the shared embedding more informative for all downstream tasks, as demonstrated by the new ablation in Section 5.1 of the revised manuscript. Differences in fitting fidelity between the NeRF renderings and the ground-truth images therefore do not bias the shared embedding toward either format or impact significantly on the cross-format alignment.
As soon as the training of the new baseline is completed, we will provide the final results on the test set.