acceptodds
Under review as a conference paper at ICLR 2027

FedFields: Learning Shared Priors for Fast Implicit Neural Representations

Abstract

Implicit Neural Representations (INRs) model signals as continuous coordinate-based functions, but typically require a separate network to be optimized for every signal. Amortized approaches avoid costly per-signal fitting by learning across many signals to directly predict signal-specific representations. Existing methods, however, assume centralized training on pooled data, limiting their applicability when signals are distributed across multiple clients. Federated learning provides a natural framework for learning across such decentralized data, yet existing federated neural field methods retain per-signal optimization: they either federate a single field for one scene or meta-learn an initialization that must still be optimized for each new signal. We introduce FedFields, a federated amortized neural field that learns a shared representation prior across decentralized signals. A shared encoder maps each signal to a latent representation, from which a lightweight per-point hypernetwork generates modulations for a shared INR decoder. Once trained, the model represents unseen signals in a single forward pass, with optional latent-only adaptation when higher reconstruction fidelity is desired. The shared model is learned collaboratively across clients through federated aggregation, while both the signals and their latent representations remain local to each client. Experiments show that FedFields achieves high-fidelity reconstruction and generalizes to unseen data distributions, while remaining effective across varying client configurations, data imbalance, and aggregation rules. The same formulation extends from images to videos, 3-D occupancy, and audio, demonstrating that amortized neural fields can be learned effectively across decentralized data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Implicit Neural Representations, Federated Learning, Amortized Neural Fields, Latent Representations, Hypernetworks.

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_CZw5OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a new federated learning for INF problem, which clearly separates client private information with shared parameters, and the zero-shot inference for new data outperforms the baseline models. Authors have conducted a very comprehensive evaluation of the proposed model using image, audio signals , video , and occupancy filed, and test various privacy attack scenario. Over all this is a very interesting and thorough study.

Critical strengths

This study has conducted thorough tests of the proposed model using various types of data modalities, different attacks for privacy .

Critical weaknesses

When comparing the proposed model with baseline models, the number of parameters of each model should be provided to give a fair comprehension of different models, as well as communication load during federated learning. While the related work is provided, direct comparison of the proposed model with most relevant models in details (e.g, encoder, hypernetwork, decoder architecture) will clarify the novelty of the model,.

Questions for authors

How is the model performance compared to centralized model with exact the same model configuration?

3 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback. To address these concerns, we now report the parameter counts and communication costs of each method, provide a component-wise comparison with closely related approaches, and present the model's performance under centralized training.

Q1)

We provide a detailed comparison of the trainable parameters and federated communication costs of FedFields and the relevant baselines in table below.

Model size and communication in federated training, and Kodak PSNR of the resulting model (dB).

MethodTrainable (exchanged)Per client and round, one wayTotal communicationKodak
Trans-INR44.5M178 MB1,067 GB25.63
ANR54.7M219 MB1,313 GB33.17
FedMeNF0.33M1.3 MB8 GB25.50
FedFields1.42M5.7 MB34 GB39.26

Architecture comparison and novelty. FedFields is built for a setting that the most related methods were not designed for: learning one amortized neural-field encoder across clients that cannot share their data. Trans-INR and ANR are amortized but centralized designs. A transformer trained from scratch encodes each signal into a set of tokens; from these tokens, Trans-INR generates the weights of a decoder for that signal (6-layer transformer of width 768, 5-layer MLP decoder), and ANR's decoder attends to them at every query (6 + 6-layer transformer of width 512, 5-layer MLP decoder). FedMeNF is federated but not amortized: it learns only the initialization of a SIREN and fits it to each signal by optimization. FedFields brings the two properties together through three design decisions.

  • A pretrained encoder, adapted rather than retrained. The encoder combines DINOv2 ViT-B/14, adapted through LoRA, with a small convolutional branch. Federation therefore trains and exchanges only small heads and LoRA weights, 38× fewer parameters than ANR and 31× fewer than Trans-INR (See the Table above), instead of a full model trained from scratch.

  • One shared decoder, modulated at every query point. A single SIREN (6 layers, width 256) is shared by all signals. A small hypernetwork predicts its neuron-wise shifts from the latent representation interpolated at each query point. No decoder is generated per signal and no query attends over all tokens, so the knowledge shared across clients sits in one compact network.

  • Separation of shared and signal-specific parts. The content of each signal stays in its latent representation on its client, and only the shared network is exchanged. In the paper's attacks (App. A.7), FedFields' exchanged updates did not reveal recognizable signal content, whereas those of federated per-signal networks did.

With these choices, FedFields reconstructs unseen signals zero-shot in a single forward pass. Under identical federated training, it is ahead of ANR and Trans-INR on Kodak (39.26 vs. 33.17 and 25.63 dB) while exchanging far fewer parameters. We will add this comparison to the paper.

Q2)

The results for centralized training are presented in table below.

Centralized training (PSNR, dB)

MethodKodakCLICDIV2K
ANR45.1244.8244.87
Trans-INR29.9632.7529.94
FedFields53.6453.9953.48

The authors have addressed my concerns sufficiently .

We sincerely thank the reviewer for confirming that our responses have addressed the concerns. We greatly appreciate the time and effort devoted to reviewing our work.

Reviewer_t8YxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper presents FedFields, a framework for learning implicit neural representations from data distributed across clients. An encoder maps each input signal to a spatial latent representation, and a hypernetwork generates local modulations for a shared coordinate decoder. The model is trained using FedAvg, while the signals and their latent representations remain on the clients. Once trained, it can reconstruct unseen signals without test-time optimization, with optional latent adaptation to improve reconstruction quality. The authors evaluate the approach on images, videos, 3D occupancy fields, and audio, and examine its performance under different federated settings. They also assess privacy through reconstruction and membership inference attacks on client updates.

The authors have partially addressed my first weakness and have better addressed my other concerns. I raised my score from 2 to 3.

Critical strengths
  1. Learning a shared neural-field model across clients is a useful direction. FedFields avoids per-signal optimization at inference, which is a practical advantage over methods that require fitting a network to each new signal.
  2. The image reconstruction results are promising, including on datasets outside the training corpus. The ablations also show clear benefits from spatial latent features and from combining convolutional features with DINOv2 features.
Critical weaknesses
  1. The comparison with existing amortized methods does not control for training data or pretrained features. Appendix A.1 states that Trans-INR and ANR are trained on CelebA, whereas FedFields uses a six-dataset corpus and a pretrained DINOv2 backbone. These differences make it difficult to attribute the reported gains to the proposed architecture. This is particularly relevant given that ANR already achieves nearly identical PSNR on Kodak (39.25 vs. 39.26 dB). A comparison using the same training corpus and comparable pretrained resources is needed to establish the advantage of FedFields.
  2. The main experiments use IID client partitions across all four modalities. Although Appendix A.8 includes an experiment labelled “Non-IID data,” the described partition changes client dataset sizes without clearly introducing differences in their content distributions. This leaves an important aspect of the federated setting untested. It is unclear whether the reported performance holds when clients contain different image domains, such as faces, textures, and natural scenes, rather than random subsets of the same pooled corpus.
Questions for authors
  1. Could the authors compare FedFields with ANR and Trans-INR trained on the same image corpus? How much of the improvement comes from the proposed architecture, rather than the broader training data and pretrained DINOv2 features?
  2. Tables 1 and 7 report identical occupancy results, but it is unclear whether these require test-time adaptation. Section 4.2.3 presents them as direct reconstruction of unseen shapes, whereas Appendix A.3 states “After adaptation.” Were these results obtained directly from the encoder-produced latents, or were the latents further optimized for each test shape? If adaptation was used, please report the zero-shot results separately and specify the adaptation steps.
  3. Does the Dirichlet partition in Appendix A.8 change only client dataset sizes, or also their content distributions? How does FedFields perform when clients hold different image domains, such as faces, textures, and natural scenes?
  4. The benefit of avoiding per-signal optimization is clear. However, the encoder already receives the complete signal, and the evaluation mainly measures reconstruction of that same input. Could the authors clarify the intended use of the resulting representation? In particular, what would it enable beyond retaining the original signal? A concrete use case would help connect the reported reconstruction results to the practical motivation.
2 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address the four main concerns through additional experiments and clarifications. Specifically, we retrain ANR and Trans-INR on our training corpus and isolate the contribution of DINOv2, evaluate content-based non-IID client splits across all four modalities, and demonstrate two applications of the learned representations beyond input reconstruction.

Q1)

We retrained ANR and Trans-INR on the same images used to train FedFields, under federated settings. To isolate the contribution of pretrained features, we also trained FedFields without DINOv2, using only the convolutional encoder and the following table presents the results.

Federation under same image corpus (PSNR, dB)

MethodKodakCLICDIV2K
ANR33.1735.4733.24
Trans-INR25.6328.0625.29
FedFields39.2640.6839.15
FedFields w/o DINOv234.0036.0333.80

Contribution of the architecture: with or without DINOv2. Table above presents the results for both settings separately. Even without DINOv2, FedFields' architecture is ahead of both baselines under identical federated training, on all three test sets, while exchanging 46× fewer parameters than ANR. Its design also matters for privacy: the content of each signal is held in a latent representation on its client, and only a small shared network is exchanged.

Q2)

The Dirichlet partition in Appendix A.8 varied client dataset sizes after shuffling the training corpus, rather than separating clients by content. It therefore evaluated data imbalance, not content heterogeneity. We will relabel this experiment “unequal client sizes” to clarify the distinction.

Content-based non-IID partition. To evaluate content heterogeneity, we partitioned the same training corpus across 10 clients according to image source. Nine clients each contain images from a single dataset: CelebA (1 client), DTD (1), LSDIR (4), and COCO (3). The remaining client contains DIV2K and Flickr2K images. Please see the table below for the results.

Content-based non-IID for images (PSNR, dB).

MethodKodakCLICDIV2K
FedFields36.4538.4936.56
ANR32.2234.6832.34
Trans-INR25.3827.7825.00

Q3)

We thank the reviewer for identifying this inconsistency. The occupancy results reported in Tables 1 and 7 correspond to test-time adaptation, not zero-shot reconstruction. Table below explicitly distinguishes zero-shot reconstruction from test-time adaptation.

Occupancy IoU.

MethodTest-time optimizationModelNet40GSOPix3D
ConvONetnone (zero-shot)0.7120.8410.725
GenSDFnone (zero-shot)0.3850.3860.327
GenSDFauthors' TTO (800 steps), all weights0.7970.6860.810
FedFieldsnone (zero-shot)0.98100.98480.9766
FedFields500 steps0.99500.99550.9945

In the revised version, we will explicitly label the occupancy results in Tables 1 and 7 as adapted, include the zero-shot results, and revise Section 4.2.3 to clarify the evaluation protocol consistently with Appendix A.3.

Q4)

A neural field can be queried at any coordinate, including between the input samples. We show two uses that rely on this: segmentation and super-resolution, both at twice the input resolution.

Tumor segmentation at a finer resolution than the input. We segment brain tumors in BraTS 2023 (T2-FLAIR, whole tumor; 60 patients, split by patient into 40 for training and 20 for testing). Each model receives the slice downsampled by 2×, its features are queried on the full-resolution grid, and the same per-pixel linear classifier is trained for every neural field method. None of the neural-field models was trained on MRI. Results are in table below.

Tumor segmentation at 2× the input resolution

RepresentationDice
ANR, federated0.710
Trans-INR, federated0.605
FedFields, federated0.815

Super-resolution. We downsample signals by 2×, encode them, and evaluate each field on the full-resolution grid: 512×512 Kodak crops (input 256×256). For both images and shapes, FedFields gives the most accurate full resolution reconstruction. The results for images and 3D shapes are presented in tables below.

2× Image super-resolution on Kodak (PSNR, dB).

MethodKodak
FedFields27.16
ANR26.24
Trans-INR14.82

2× 3D shape super-resolution (IoU).

MethodModelNet40GSOPix3D
FedFields0.7680.8560.729
ConvONet0.7140.8230.694

We sincerely thank the reviewer for kindly increasing the score. We greatly appreciate the time and effort devoted to reviewing our work.

Reviewer_nX3AOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose an amortized Neural Representation framework that allows zero-shot prediction, which is trained through FL. Their main contribution is to take the idea of amortized inference for INRs and adapt it to FL. Their amortized inference architecture is new and extends to a variety of modalities, re-using existing components.

Their experimental evaluation covers a single federated setting and several modalities.

Score raised 2->3 post-rebuttal

Critical strengths

The paper evaluates a lot of data modalities & datasets. Along that axis, the evaluation is extensive. The headline of zero-shot performance with optional adaptation is very strong. Good ablation on Dino and other architectural choices.

Critical weaknesses

The authors propose both: a new architecture as well as a new way to train it (FL). Please separate the two: What is the centralized training performance of the architecture on the non-federated data-set? How do established amortized architectures such as ANR or Trans-INR fair under federated training?

The privacy claims are very strong and focus on signal recovery, but critically membership inference attacks succeed as shown in the appendix. Not technically a "critical weakness", but being straightforward about membership inference attack in the main-paper along the signal recovery resilience would be appreciated.

Baseline confounders. FedFields, ANR and Trans-field were trained on different datasets. Noteably, FedFields had access to much more data, in addition to using features from a pretrained DinoV2. All methods are evaluated on the same datasets, which are more diverse than CelebA alone which was used for ANR and Trans-field. Arguably the baselines have a much larger domain shift to overcome. FedFields is trained on 256x256, while baselines are trained on lower resolutions. The gap to the higher eval-resolution is thus higher for baselines.

The optimization surface for FedField for the per-image fine-tuning results are mis-matched to baselines. The latent-space at 128x128x192 is huge compared to the MLPs for baselines. While the authors control for adaptation costs, the headline figure 1 is disadvantaging baselines for capacity.

The baseline federated setup is limited: IID distributed data, only 10 clients. It serves as a proof-of-concept, but the appendix is way more interesting. Crucially, the non-iid experiments are not actually non-iid in terms of data-distribution, just in terms of data-quantity per client. FedAvg weighs updates by local dataset size, so the fact that experiments show little impact as a consequence of data-quantity is not super surprising. A truly interesting non-iid study creates different data-distributions across clients, e.g. faces on one, landscapes on another, etc. A more extensive focus on the actual federated experiments of their method is required, not only for images but also other modalities. (I'm not suggesting to mix modalities in a single training run). Another missing angle is client-participation. I would have expected to see fractional participation. Another angle to FL that is often relevant is communication budget: How much GBs is communicated comparing to baselines? In summary: Their federated setup is too easy to show relevance to potentially "real" federated deployment scenarios such as the hospital setting the authors use as an example.

Questions for authors

I would like the authors to expand their federated setup. I would like the authors to explain if their methodological comparison is valid compared to baselines given huge differences in terms of modeling capacity, datasets used, etc as mentioned above - as well as distinguishing Federated aspects from architectural design choices. I believe crucial experimental evaluations to be missing.

I would like to understand the motivation for federated amortized inference better. The authors' headline is Test-time optimization steps wrt. PSNR. Especially in privacy-relevant applications, why would I not choose to "simply" optimize SIREN on individual data-point until I achieve the same desired PSNR as what FedFields or other amortized approaches might give me. It might be much more practical to pursue a non-federated approach for a desired target PSNR than try to "proof" privacy guarantees to gain legal approval for a medical deployment across a federated hospital, given data-privacy protection regulation. I might want to spend more GPU hours. While I can appreciate the paper's motivation from a scientific advancement perspective, the practical implications might be very limited.

5 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the thoughtful suggestions regarding separating the architectural contributions from federated training, controlling for training data, pretrained features, and model capacity, and evaluating more challenging federated settings.

Q1)

We trained all three architectures on the same training images under two settings: centralized training, where all images are available at a single site, and federated training. Please see the table below for the results.

Reconstruction under centralized and federated training (PSNR, dB).

MethodCentralized KodakCentralized CLICCentralized DIV2KFederated KodakFederated CLICFederated DIV2K
ANR45.1244.8244.8733.1735.4733.24
Trans-INR29.9632.7529.9425.6328.0625.29
FedFields (Matched to ANR size)53.6453.9953.4837.4739.3737.46

Q2)

We thank the reviewer for highlighting this limitation. While the evaluated gradient-inversion and weight-readout attacks did not recover recognizable image content from FedFields updates, membership inference remains effective for clients holding few images. We will explicitly report this limitation in the main privacy section of the revised paper and soften Contribution 3 and the conclusion to avoid overstating the privacy protection provided by FedFields.

Q3)

Training data. We now compare all the methods trained under the same training corpus as FedFields.

DINOv2.

FedField's reconstruction performance without DINOv2 (PSNR, dB)

Training SettingKodakCLICDIV2K
Centralized38.6540.2038.52
Federated34.0036.0333.80

Resolution.

FedFields, ANR, and Trans-INR were originally trained at their native input resolutions. As the reviewer suggested, we trained ANR, Trans-INR, FedFileds on the same resolution, and the following table presents the results.

Effect of training input resolution under centralized training on the same corpus. At inference, each model reconstructs full-resolution images.

MethodTraining inputKodakCLICDIV2K
ANRnative45.1244.8244.87
ANR256×25630.7934.3732.80
Trans-INRnative29.9632.7529.94
Trans-INR256×25626.0229.0726.40
FedFieldsnative53.6453.9953.48

Both baselines reconstruct less accurately when trained on 256×256 inputs. Each of them represents an input tile with one network: Trans-INR generates a single MLP for the whole tile, and ANR's decoder answers every query by attending over the tokens of the whole tile. A 256×256 tile contains 4× (ANR) and 2.1× (Trans-INR) more pixels than its native input, while this network keeps the size chosen for the native input (a 5-layer MLP of width 512 for ANR, a generated 5-layer MLP of width 256 for Trans-INR) and is trained for the same number of steps. The same network must then describe more image content, so reconstruction quality drops.

Q4)

We evaluated the FedFields under same capacity as the MLP. The following table shows the results.

Reconstruction quality at matched representation size (PSNR, dB).

Test setFedFields zero-shotFedFields 500 stepsSIREN 500 steps
BraTS MRI38.8343.9134.81
Kodak32.0934.4730.94

Q5)

Content-based non-IID. We thank the reviewer for pointing out that our original Dirichlet split (Appendix A.8) was applied to a shuffled dataset, resulting in differences in client dataset sizes rather than image content. To address this, we experimented a content-based non-IID split in which clients receive images from different source datasets. Specifically, four clients hold LSDIR images (high-resolution scenes), three hold COCO images (everyday scenes), one holds CelebA images (faces), one holds DTD images (textures), and the remaining client holds DIV2K and Flickr2K images. We further extended the content-based non-IID evaluation to the other three modalities: audio, video, and 3D shapes. The results for images, video, audio and 3D shapes are presented in tables below.

Content-based non-IID for images (PSNR, dB).

MethodKodakCLICDIV2K
FedFields36.4538.4936.56
ANR32.2234.6832.34
Trans-INR25.3827.7825.00

Content-based non-IID for video (PSNR, dB).

Test setContent split
Kinetics38.24
Sintel35.75
Xiph34.13

Content-based non-IID for audio (PSNR, dB).

Test setContent split
Unseen speakers27.58
Seen speakers29.43

Content-based non-IID for 3D shapes (IoU).

Test setContent split
ModelNet400.9785
GSO0.9830
Pix3D0.9738

Partial participation.

To evaluate the effect of partial client participation, we randomly select a subset of the 10 clients to participate in each training round. With all 10 clients participating in every round, FedFields achieves 39.26 dB on Kodak. When only 7 or 5 randomly selected clients participate per round, it achieves 36.08 and 35.50 dB, respectively. Under the content-based non-IID split, partial participation, FedFields achieves 36.45 dB with all 10 clients and 35.35 dB with 5 of 10 clients.

Communication. FedFields communicates only its trainable parameters, while the frozen DINOv2 backbone is never transmitted. Please refer to the table below.

MethodPer client per round (one way)Total
FedFields5.7 MB34 GB
ANR219 MB1,313 GB
Trans-INR178 MB1,067 GB

Number of clients.

We also evaluated the effect of increasing the number of clients from 10 to 100 and the results are reported in table below.

Effect of the number of clients (PSNR dB)

Clients50100
Kodak31.1330.98

Q6)

The additional controls confirm the comparison on each of the points raised. With the same training data (Q1, Q3), the matched size as per-signal MLPs (Q4) and as ANR (Q1), and the same federated protocol, including non-IID clients, partial participation, and communication cost (Q5), FedFields remains ahead of the baselines. Training every architecture both centrally and federated separates the architecture from federation (Q1), and removing DINOv2 isolates the contribution of the pretrained features (Q1, Q3).

Q7)

We thank the reviewer for raising this practical point. We agree that fitting a SIREN to each signal locally is a sound choice: it shares nothing and, with enough GPU time, reaches high fidelity. FedFields is complementary to it. Once trained, it reconstructs a new signal in a single forward pass. To reach the same quality, SIREN needs about 1,700 optimization steps on brain MRI, and on Kodak it does not reach it even after 5,000 steps. This benefit comes from amortization, and it holds however the model is trained: centrally when data can be pooled, on public data, or federated when they cannot. Federation extends it to institutions that cannot share their data: they train one model together, each contributing a small part of the training cost. We quantify the cost per signal at equal PSNR, including the additional GPU hours the reviewer mentions, and the one-time cost of training.

Cost per signal at the same PSNR. The paper model was trained only on photographs and is applied to brain MRI without retraining. On BraTS, its single forward pass reaches 43.75 dB in 0.02 s per slice. SIREN reaches the same PSNR after about 1,700–1,800 steps, in 18–19 s per slice, so FedFields needs about 900× less compute per signal; in the time of one FedFields pass, SIREN completes about two steps. At this cost, a full BraTS volume of 155 slices takes about 3.1 s with FedFields and about 48 minutes with SIREN. On Kodak, the gap is larger still: FedFields' single pass (39.26 dB) is above SIREN even after 5,000 steps per image.

One-time training cost. Training FedFields takes about 21 GPU hours on one RTX A5000. Compared with fitting SIREN to the same PSNR, this is recovered after about 4,000 MRI slices. In federated training the cost is divided among the institutions: each of the ten clients computes about 2 GPU hours, which it recovers after about 400 of its own slices, fewer than three BraTS volumes. A user who reuses an already trained model, as the paper model is reused on MRI, has no training cost. For a few signals and an unlimited per-signal budget, fitting SIREN remains a good choice; when many signals must be converted, FedFields is the more efficient one.

When federation helps. Federation is useful when institutions want a shared model but cannot pool their data. Each institution keeps its signals on site and contributes to one model that all of them can then use locally, and in our attacks, the exchanged updates did not reveal recognizable signal content. In the revision, we will state that FedFields can be trained centrally, on public data or federated, and add this cost comparison.

Comments on the paper

1 comment

Dear all,

We're the half-way through the discussion phase. The authors have provided their response to your comments. Some of you have started involving discussion, but all of us can take action now.

If the provided responses do not address (even part of) your concerns, this is your chance to raise it. Please engage with the authors as soon as possible, we need to keep the conversation going!

Thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.