acceptodds
Under review as a conference paper at ICLR 2027

OneField: A Shared Embedding Space for NeRFs and 3D Gaussian Splatting

Abstract

Radiance fields are a data modality where an object is stored as a function of space. Recent encoders read that function directly to perform classification, retrieval, segmentation, and language-based tasks. A radiance field is commonly stored in one of two formats: Neural Radiance Fields (NeRFs) or 3D Gaussian Splatting (3DGS). These formats exhibit a different trade-off between memory footprint and rendering speed: NeRFs are memory-efficient but slow to render, whereas 3DGS enable fast rendering at a substantially higher memory cost. Existing encoders can process only one format, i.e., a model trained on NeRF weights does not read Gaussian primitives, and vice versa. Yet, the two formats can be trained on the same rendered views of an object. Thus, they differ in the representation used to model the object, not in the object they encode. OneField is an encoder that maps both formats into a common embedding space, aligned across formats by a contrastive objective, and with a shared decoder enforcing the object's field reconstruction. In this space, the format of the input no longer matters: in classification, part segmentation, and language tasks, a head trained on one format transfers to the other one, with little to no performance degradation. OneField remains competitive with format-specific encoders in all tasks. In addition, it enables retrieval from a unified gallery containing both formats, a task that no format-specific encoder can do, and retrieves the correct object at –% recall@1, – points above a CLIP baseline processing rendered views.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
nerf, 3dgs, encoder, latent space, shared embedding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fm7POpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OneField, a framework that aligns NeRF and 3D Gaussian Splatting (3DGS) into a shared embedding space. It employs two format-specific encoders, SigLIP and VICReg alignment losses, and a shared reconstruction decoder. Experiments on approximately 300K paired objects demonstrate effective cross-format transfer in classification, segmentation, retrieval, and language-based tasks.

Critical strengths

Strong experimental results. OneField demonstrates consistent cross-format transfer across multiple downstream tasks, with minimal performance degradation. The mixed-format retrieval results are particularly convincing. Large-scale dataset and useful analysis. The paired NeRF–3DGS dataset and the robustness experiments provide valuable resources and evidence for cross-format alignment.

Critical weaknesses

Missing unified representation baseline. Both NeRF and 3DGS describe spatially queryable fields. A simpler alternative would sample comparable spatial attributes from both representations and process them using a shared Point Transformer. Without this baseline, the necessity of separate encoders remains unclear. Lack of parameterization invariance analysis. Function-preserving permutations of NeRF hidden neurons can change the input weight coordinates without changing the underlying radiance field. The proposed encoder does not guarantee invariance to these transformations, raising concerns about whether it learns field properties or parameter-specific patterns. Potential bias in reconstruction supervision. Both branches use NeRF-rendered images as reconstruction targets. This may bias the shared embedding toward NeRF-specific information. The effect of symmetric or ground-truth reconstruction supervision remains unexplored.

Questions for authors
  1. How does OneField compare with a unified field-sampling baseline using a shared 3D encoder?
  2. Does the NeRF embedding remain stable under function-preserving hidden-neuron permutations?
  3. How does using format-specific or ground-truth reconstruction targets affect cross-format alignment?
2 comments

Q1: Unified field-sampling baseline

We thank the reviewer for this suggestion. We trained a new field-sampling baseline that uses a shared 3D encoder for both NeRF and 3DGS. Both representations are queried as spatial fields, in the same frame, on a shared 16×16×16 grid covering the object, for a total of 4,096 cells. Each cell is subdivided into a regular 4×4×4 sub-grid, and both the NeRF and the 3DGS are queried at these same 64 locations. This gives one point per cell. Both point clouds are encoded by a single shared Point Transformer [1]. We treat the output of this Transformer as the output of our Attentive Poolers. Everything else is identical to our method: data and splits, the SigLIP and VICReg losses, the decoder with its rendering target, and the training schedule.

Appendix 14 of the revised paper discusses the results of this experiment.

The field-sampling baseline performs worse than OneField, across all the evaluated tasks (retrieval, classification, and segmentation). While on classification and segmentation, the gap with OneField is around 5 points, the baseline fails on retrieval, exhibiting a performance degradation of about 39 points.

The baseline never accesses the source NeRF and 3DGS representation, relying instead on a coarse point cloud sampled from each of the input formats. The single 3D encoder fails to produce informative embeddings of the input object, proving the necessity of the separate encoders adopted by OneField.

[1] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 16259–16268, 2021

Q2: Function-preserving hidden-neuron permutations

Like nf2vec, our encoder processes the raw weights of a fixed MLP. The NeRFs we use were all trained from one shared random initialization. Hidden-neuron permutations do not occur in the training or test data.

To measure the sensitivity of OneField to hidden-neuron permutations, we apply random function-preserving permutations to the NeRFs of the ShapeNet test set. Permuting a single hidden layer gives an average cosine similarity of 0.950.05 to the clean embedding, and classification drops slightly, from 93.56%93.56\% to 92.80±0.1%92.80 \pm 0.1\%. Permuting the first two layers gives a cosine of 0.860.10 and 89.70±0.2%89.70 \pm 0.2\% classification accuracy. Permuting all three hidden layers gives a cosine similarity of 0.640.18 and 88.6±0.3%88.6 \pm 0.3\% classification accuracy.

Retrieval is more sensitive than classification: NeRF to 3DGS R@1 drops from 92.3 to 81.8 (one layer), 60.0 (two layers) and 20.1 (three layers).

Appendix A.15 of the revised paper provides this analysis.

Q3: Format-specific vs ground-truth reconstruction targets

We thank the reviewer for this question. Our work assumes that the original training views of the NeRF and 3DGS are not available: OneField is meant to encode NeRFs and 3DGS as they are, without relying on any external data. For this reason, we take the reconstruction target from the input representation rather than from the original ground-truth images. Nonetheless, we conducted an additional experiment to test how ground-truth reconstruction targets affect cross-format alignment. We are training OneField from scratch using the original ground-truth images as reconstruction targets for both branches, instead of the NeRF renderings. Everything else is kept identical: architecture, losses, data and schedule. Ground-truth images are not specific to either format, so this setting removes the potential bias of the reconstruction supervision toward NeRF-specific information.

The current model is still training, and the results below compare OneField with this new baseline, evaluating them on the validation set and considering the same epoch for both (current epoch: 165/200). For this reason, the numbers for OneField are different from those in the paper.

So far, the effect on cross-format alignment is negligible. The cosine similarity between the NeRF and 3DGS embeddings of the same object is exactly the same, 0.963. Cross-format retrieval R@1 changes from 93.5 to 93.9 (NeRF→3DGS) and 93.3 to 93.7 (3DGS→NeRF). This is because our shared decoder is not intended to provide high-fidelity reconstructions, but rather to make the shared embedding more informative for all downstream tasks, as demonstrated by the new ablation in Section 5.1 of the revised manuscript. Differences in fitting fidelity between the NeRF renderings and the ground-truth images therefore do not bias the shared embedding toward either format or impact significantly on the cross-format alignment.

As soon as the training of the new baseline is completed, we will provide the final results on the test set.

Update on Q3.

Training of the new baseline with ground-truth reconstruction targets is now complete, and we evaluated both models on the test set with the protocols of the paper. The final results confirm what we observed during training: the effect on cross-format alignment is negligible. The cosine similarity between the NeRF and 3DGS embeddings of the same object is practically unchanged (0.961 for OneField, 0.962 with ground-truth targets). Cross-format retrieval R@1 changes slightly, from 92.3 to 92.9 (NeRF→3DGS) and from 91.6 to 92.5 (3DGS→NeRF). Classification accuracy changes by at most 0.5 points. With the head trained on NeRF embeddings, OneField scores 93.6 on NeRF and 93.5 on 3DGS, against 93.5 and 94.0 with ground-truth targets. With the head trained on 3DGS embeddings, OneField scores 93.6 and 93.7, against 93.6 and 94.0. Ground-truth images are therefore not needed to reach the same alignment, and OneField can be trained directly on existing NeRFs and 3DGS.

Reviewer_zQvXOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OneField to learn unified 3D object representations by separately encoding NeRF weights and 3DGS parameters, optimized via cross-format alignment losses and shared-decoder reconstruction supervision. Trained on ~300k paired objects, the framework demonstrates strong cross-format transferability for downstream models trained on a single modality across classification, retrieval, part segmentation, and language tasks.

Critical strengths

The paper tackles the representation incompatibility between NeRF and 3DGS, enabling shared downstream models and mixed-gallery retrieval. This effectively eliminates the overhead of training separate format-specific models in practice. Moreover, Bidirectional cross-format transfer is thoroughly evaluated across classification, part segmentation, and language tasks. The marginal drop in cross-format classification accuracy (only 0.08% and 0.10%) and consistent performance in segmentation and language tasks strongly validate the reusability of the learned shared embedding space.

Critical weaknesses
  1. Although the shared decoder and reconstruction supervision are highlighted as essential for preserving geometry and appearance, existing ablations only examine SigLIP and VICReg, omitting a direct ablation on the reconstruction objective itself. Moreover, neither quantitative metrics nor qualitative visualizations of the shared decoder's reconstructions are provided.

  2. Fitting to identical views does not ensure identical geometry and appearance across NeRF and 3DGS. Using NeRF renderings as the sole reconstruction target for both branches introduces an asymmetric supervision signal whose impact is under-explored. Specifically, if 3DGS captures high-frequency details missed by NeRF, it is unclear whether the shared representation biases toward NeRF and discards those details.

  3. The NeRF encoder relies on a fixed MLP architecture, as both the neuron tokenization and layer-wise projection networks are hard-coded to specific depths and widths. This constraint may restrict the method's generalizability to heterogeneous NeRF variants in the wild.

Questions for authors
  1. Could you provide comparisons or at least quantitative (e.g., PSNR/SSIM) comparisons between the images decoded from the NeRF/3DGS embeddings (via the shared decoder) and the direct renderings from the original NeRF/3DGS under identical views? Clarifying the preservation of geometric and appearance details would directly reveal any information loss during the encode-decode process.

  2. Why did you choose NeRF renderings over the original ground-truth images as the reconstruction target for both branches? How does this design affect the information retained in the shared embedding when the two modalities exhibit different fitting fidelities?

2 comments

W1: Ablation on the reconstruction objective

We thank the reviewer for this observation. We added a direct ablation on the reconstruction objective and reported its results in Section 5.1 of the revised paper. Removing the reconstruction term from the training loss leads to a performance degradation, across all metrics and tasks, showing how the reconstruction objective makes the shared embedding more informative for every downstream task.

W1/Q1: Comparison between shared decoder's reconstructions, NeRF/3DGS renderings, and GT views

In our architecture, the decoder is a training device and not a reconstruction module. During training, a reconstruction loss forces the embedding to retain the geometry and coarse appearance of the object, which the alignment objective alone does not require. The ablation in Section 5.1 of the revised paper shows that removing the decoder lowers every downstream metric, which supports this role. At inference, the decoder is dropped and every task runs on the embedding alone. Rendering its output is therefore not an alternative to the original NeRF or 3DGS, which already provide high-quality views.

We report the renders of the shared decoder in Appendix A.13 of the revised manuscript. The decoded fields reach 22.82 dB (NeRF embedding) and 23.55 dB (3DGS embedding) PSNR on ShapeNet, against 35.51 dB and 31.05 dB for the original fits. As the qualitative results in Figure 9 show, the decoder recovers the geometry and the coarse appearance of the object from a single embedding, while fine details are blurred. This gap is expected, since the original fits are optimized for a single object and stored in their full set of parameters, whereas the embedding is one compact vector produced by an encoder shared across all objects. What matters for our method is that the embedding carries the information that downstream tasks need. The results in the main paper show that it does. The shared embedding performs strongly across all tasks, including the fine-grained ones, such as part segmentation and language tasks. The decoder is not needed for any of them.

W3: The NeRF encoder relies on a fixed MLP architecture

We acknowledge the point raised by the reviewer. The NeRF encoder is built for the MLP architecture of our dataset, which follows nf2vec (three hidden layers of 64 units).

The underlying design, however, is not tied to that architecture. The neuron tokenizer produces one token per neuron, built from that neuron's incoming and outgoing weights, so the number of tokens grows with the network, and the attentive pooler aggregates a set of tokens of any size into one embedding of fixed dimension. Extending the encoder to MLPs of other widths and depths requires instantiating the per-layer projections and layer embeddings for each layer of the target architecture, while the tokenizer, the pooler and the shared decoder remain unchanged.

Recent work has already shown that weight-space encoders can process NeRFs with different architectures by treating the network as a graph [1]. Bringing this kind of architecture-agnostic encoding into OneField is a natural next step, and we plan it as future work.

[1] Francesco Ballerini, Pierluigi Zama Ramirez, Luigi Di Stefano, and Samuele Salti. Weight space representation learning on diverse nerf architectures. In International Conference on Learning Representations, volume 2026, pp. 122085–122108, 2026

W2/Q2: Choosing NeRF renderings over GT images as reconstruction target

We thank the reviewer for this question. Our work assumes that the original training views of the NeRF and 3DGS are not available: OneField is meant to encode NeRFs and 3DGS as they are, without relying on any external data. For this reason, we take the reconstruction target from the input representation rather than from the original ground-truth images.

We use a single reconstruction target so that both branches are trained with the same objective. If the 3DGS branch were supervised by renderings of its own 3DGS, the shared decoder would get two reconstruction losses whose targets differ for the same object. This would result in pulling toward two slightly different objects, and that difference would conflict with the alignment we want between the two embeddings. We chose NeRF over 3DGS because our decoder is itself a NeRF-like implicit field. Its prediction and the NeRF target are rendered by the same ray marcher, while a 3DGS target would come from a rasteriser with a different image-formation model.

Regarding how this design affects the information retained in the shared embedding when the two modalities exhibit different fitting fidelities, we added a dedicated analysis in Appendix A.12 of the revised paper, described in the authors’response to Reviewer wxDc.

We thank the reviewer zQvX for raising the score. Their comments helped us improve the paper. In particular, the ablation on the reconstruction objective is now part of the revised manuscript

Reviewer_wxDcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to encode two different 3D representations, neural fields (NeRF) and 3D Gaussian Splatting (3DGS), in the same embedding space. To this end, it first proposes NeRF and 3DGS encoders designed for this task. In addition, the paper proposes several alignment losses so that the embeddings store representation-independent 3D information, and so that different modalities do not result in different modes in the embedding space, which are known to be detrimental to downstream applications. Through various experimental settings, it shows that the shared embedding space can be used for several downstream applications without performance degradation.

Critical strengths

The experiments were designed to show various aspects of the proposed method on multiple downstream applications, including retrieval, segmentation, and captioning. In addition, ablation studies show why the proposed losses on the embedding space are needed.

Critical weaknesses

Motivation for the unified embedding space The paper does a good job of showing how to build a shared embedding space for both NeRF and 3DGS representations. However, it does not sufficiently explain why this is needed, and focuses more on evaluating downstream applications. It would be helpful to explain more clearly why this embedding space is needed.

Robustness This is a common issue for data-driven networks, but it is still unclear whether the method can work robustly in diverse situations. For instance, would it work robustly if a rigid transformation is applied, or should all objects be in the same normalized space? It is also often the case that NeRF and 3DGS converge to slightly different representations from the same set of images. I would like to know whether the method performs robustly in these cases.

Questions for authors

Would it be possible to explain why a joint embedding space for NeRF and 3DGS is needed? As a lower priority, additional robustness experiments, if possible, could strengthen the paper.

3 comments

W1/Q1: Motivation for the unified embedding space

NeRF and 3DGS are both common formats for 3D assets, and public datasets are released in one or the other. For example, ShapeNeRF-Text and ObjaNeRF-Text release NeRF weights, while ShapeSplat releases 3DGS. Existing encoders read only one format: nf2vec reads NeRFs, while ShapeSplat-GMAE and UniGS read 3DGS. With these encoders, anyone who holds assets in both formats has two options: discard the assets in the other format, or convert them. Converting means rendering views and fitting each object again, which is expensive at scale.

Moreover, when the original images are not available, it is cumbersome to determine a set of viewpoints to render good training images that allow to effectively fit the object and/or the scene.

OneField removes this trade-off. Each asset is encoded from its native format into a shared space, so whatever is trained on top of that space (e.g., a classifier, a segmentation head or a language model) works regardless of the input format. We conducted the following additional experiment to test this capability.

Mixing datasets released in different formats. A classifier is trained on 3D shapes, some available as NeRFs and some as 3DGS. With OneField, all of them can be pooled into one training set, with nothing discarded or converted. We conducted this experiment on ShapeNet by giving the classifier each training object in only one format (either NeRF or 3DGS, never both), at NeRF:3DGS percentage ratios from 50:50 to 99:1 and the reverse. A single-format encoder can only use part of the data in its own format. On this dataset, 1% of the samples correspond to 317 training objects. For ShapeSplat-GMAE the classification accuracy drops by 27.5 points from 50:50 to 99:1 training sample ratios (i.e., from 89.9% to 62.4%). Similarly, nf2vec exhibits a drop of about 12 points.

The accuracy of OneField ranges between 93.3% and 94.2% on both formats, across all ratios, thanks to the ability to train on all the data regardless of the available format.

Therefore, thanks to the shared space of OneField, the mix of formats barely matters.

This additional experiment is reported and discussed in Appendix A.11 of the revised paper.

W2: Robustness to a rigid transformation

None of the considered encoders (OneField, nf2vec, ShapeSplat-GMAE and UniGS) is robust to rigid transformations. They all assume objects are in the same normalized space.

W3: NeRF and 3DGS converging to slightly different representations from the same set of images

We conducted an additional experiment on this point. For every test object, we measure the PSNR of its NeRF and 3DGS fits against the training views, and we select the 100 pairs with the largest PSNR gap between the two formats. In these pairs, one format reconstructs the object much better than the other (average gap of 14.0 dB). Even so, the average cosine similarity between the NeRF and 3DGS embeddings stays high (96.3%) and cross-format retrieval remains accurate (R@1 of 96%). This shows that OneField is robust to differences in fit quality between the two formats.

The full experiment, including the comparison with the 100 pairs with the smallest PSNR gap, is discussed in Appendix A.12 of the revised paper.

Q2: Additional robustness experiments

The original submission also includes further robustness experiments:

  • Disjoint training views (App. A.10). We refit the NeRF and 3DGS of 100 ShapeNet test objects, so that the two formats share no views: the NeRF on the 18 even views, the 3DGS on the 18 odd ones. The cosine similarity between the embeddings matches the one obtained when both formats are fitted on the same 18 views.
  • Fewer Gaussians (App. A.2). We randomly subsample each 3DGS to fewer Gaussians and re-encode it, while the NeRF embedding is kept fixed. With 512 Gaussians, almost ten times fewer than the 5,000 used in training, the similarity to the NeRF embedding is still 98.5% of the one obtained with all 5,000 Gaussians. It drops only below about 128 Gaussians, where the splat can no longer describe the object.

I would like to thank the authors for providing detailed experiments and explanations.

I originally had two concerns: the motivation and the robustness.

The motivation concern is mostly resolved, but it raises another question. This does not affect my score since it was not part of my initial review, but I would like to make it clear as a remaining concern. If this is the storyline, why does the decoder only test NeRF-style reconstruction? The current pipeline seems to just map both NeRF and 3DGS into a NeRF-focused space.

More importantly, the robustness issue is not resolved, and it was raised by all reviewers. I asked about rigid transformations, reviewer zQvX asked about fixed MLP architectures, and reviewer fm7P asked about function-preserving hidden-neuron permutations. Most of these remain unresolved. I appreciate that the authors ran the permutation experiment and shared them, but unfortunately, the results clearly show that permutation matters.

As the authors pointed out, none of the considered encoders address robustness thoroughly. However, I think it would strengthen the paper if it demonstrated robustness, showing that the method can be used beyond the specific dataset and settings used in the paper. Currently, it feels too limited to a specific architecture and settings.

Therefore, I would like to keep my initial score.

We thank the reviewer for the follow-up and for acknowledging that the motivation concern is largely resolved. Below, we address the questions regarding robustness and the decoder.

1. Scope of the contribution

The claim of the paper is that NeRF and 3DGS fits of the same object can be mapped to a single shared embedding space, so that a head trained on one format works on the other. OneField’s main contribution is the alignment framework—the cross-format objectives and the resulting single space. For the encoding part, we consider the state-of-the-art models for each format (nf2vec, ShapeSplat-GMAE, UniGS). Each of these encoders operates on assets in a canonical, normalized frame, fitted using the protocol of the public datasets (ShapeNeRF-Text, ObjaNeRF-Text). This is a property of the input data convention, not of our alignment framework, and it applies identically to every method we compare against. It therefore does not affect any comparison, ablation, or conclusion in the paper.

2. Robustness (function-preserving permutations)

Following nf2vec/inr2vec, all our NeRFs are fitted from a shared initialization. This is established practice. Papa et al. (CVPR 2024) show that it substantially improves downstream accuracy even for permutation-equivariant classifiers. Nonetheless, our measured sensitivity is modest rather than catastrophic: permuting one hidden layer leaves cosine similarity at 0.95 ± 0.05 and classification at 92.80 ± 0.1, compared with 93.56. With all hidden layers permuted, classification degrades moderately (93.6 → 88.6).

3. Is the shared space NeRF-focused due to supervision?

The decoder is “NeRF-like” only in that it renders radiance fields by volume rendering. The results below argue against a NeRF bias:

(i) In response to reviewer fm7P, we retrained OneField from scratch using the ground-truth images as the reconstruction target. This supervision leaves alignment essentially unchanged (cosine 0.961 vs. 0.962; R@1 92.3 → 92.9 and 91.6 → 92.5). Ground-truth images are not specific to either format, so this removes any NeRF-specific bias in the supervision.

(ii) Decoding from 3DGS embeddings yields higher PSNR than decoding from NeRF embeddings (23.55 vs. 22.82 dB, App. A.13). If the embedding were NeRF-centric, one would expect the opposite outcome.

(iii) Downstream heads transfer successfully in both directions.

Finally, the decoder is a training device, not a module of the method, and is discarded at inference. Its role is to force the embedding to retain geometry and coarse appearance. The new ablation in Section 5.1 shows that removing it lowers every downstream metric.

4. Other Reviewers

For completeness, since the reviewer refers to concerns raised across reviews:

  • Reviewer zQvX (W3: fixed-MLP architecture): We explained that the tokenizer and pooler are architecture-agnostic by design and that extension would only require instantiating per-layer projections. zQvX subsequently raised their score to 3.

  • Reviewer fm7P (Q2: permutation invariance): We conducted the full sensitivity analysis (now in Appendix A.15). fm7P had already listed our robustness experiments among the paper’s strengths and rated the paper a 3.

  • fm7P (Q1: unified field-sampling baseline): We trained this baseline (now in Appendix A.14), which scores ~5 points below OneField on classification and segmentation and ~39 points below on retrieval, proving the necessity of the separate encoders adopted.

  • fm7P and zQvX: Both raised the reconstruction-target bias, answered by the ground-truth-target retraining described above.

We will add an explicit paragraph to the main paper stating the assumptions of the setting: canonical object frame and shared-initialization fitting. We thank the reviewer for pushing us to make this explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.