acceptodds
Under review as a conference paper at ICLR 2027

Optimization Encoders: Rethinking Second-Order Meta-Learning for Neural Fields

Abstract

Conditional neural fields represent signals continuously, but their effectiveness depends on how the conditional latent representations are inferred from observed data. In meta-learning, this encoding occurs through gradient updates induced by the decoder, tying representation learning directly to decoder design. We formalize this connection by interpreting latent optimization as an optimization encoder, unifying the roles of second-order differentiation, latent parameterization, and task supervision. This concept enables second-order meta-learning for end-to-end training of the encoding procedure alongside the decoder, and clarifies which learning pathway first-order approximations discard. Guided by this view, we introduce Attentive Latent Fields (MetaLF), an equivariant transformer-based neural field that contextualizes a latent pointcloud through self-attention. These interactions shape both field predictions and the updates that construct their representation, allowing local observations to inform coherent non-local structure. Disentangling the inner encoding objective from outer task supervision unifies reconstruction, classification, and segmentation within an end-to-end meta-learning framework, using reconstruction-only latent adaptation at test time. Controlled experiments on polynomial fields link latent coordination to lower effective rank and stronger alignment with the underlying function space. Across image and 3D shape reconstruction, MetaLF improves fidelity within three to five gradient updates, while supporting semantic prediction across images, shapes, and volumes. Together, these findings position the optimization encoder perspective as a unified basis for designing neural fields around how representations are constructed, coordinated, and used.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Conditional neural fields, meta-learning, optimization encoders, implicit neural representations, equivariant transformers, representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3q3HOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper interprets the latent optimization process in conditional neural fields as an optimization encoder and shows how second-order meta-learning allows the outer objective to train this encoding process through the decoder. Motivated by the observation that the decoder determines both field predictions and the gradient updates used to construct latent representations, the authors propose Attentive Latent Fields (MetaLF), an equivariant transformer-based neural field decoder that introduces self-attention among spatial latent tokens. Experiments on polynomial fields, image and 3D shape reconstruction, and semantic prediction tasks (classification and segmentation) demonstrate the effectiveness of the proposed architecture.

Critical strengths

Strong empirical performance across diverse tasks. The authors conduct extensive experiments across image and 3D shape reconstruction, classification, and segmentation, demonstrating substantial improvements over existing neural-field baselines on multiple benchmarks with only a few latent optimization steps.

Critical weaknesses
  1. Limited novelty. The second-order differentiation analysis largely follows established results from MAML and related meta-learning methods, while the proposed architecture primarily integrates existing self-attention mechanisms into equivariant neural fields. The novelty therefore appears to lie mainly in combining existing techniques, rather than introducing new theoretical results or fundamentally new architectural components.

  2. Unclear benefits of the optimization encoder perspective. Although the authors reinterpret latent optimization as an encoding process, it is unclear what additional insights this perspective provides. In particular, the motivation for introducing self-attention does not seem to follow directly from the analysis. Other architectural modifications, such as increasing the depth of the existing cross-attention decoder, could also enable richer interactions among latent features and potentially improve performance without requiring explicit latent self-attention.

  3. Unclear distinction from prior work and mathematical presentation. The paper does not clearly distinguish its own contributions from established techniques and results, making it difficult to identify which insights are genuinely novel. Additionally, the mathematical exposition is unnecessarily complicated in places. For example, Section 3.2 introduces several auxiliary variables (PikP_i^k, AikA_i^k, and BikB_i^k) to derive the second-order meta-gradient, making the main argument harder to follow.

  4. Insufficient ablation of the proposed self-attention mechanism. Although the authors provide ablations of self-attention depth, it remains unclear whether the observed improvements arise specifically from self-attention or simply from increased decoder expressivity. For example, increasing the depth of the existing cross-attention decoder could also enable richer interactions among latent features. Comparisons with parameter-matched alternatives without self-attention would help establish whether self-attention provides benefits beyond increased decoder capacity.

Questions for authors
  1. What concrete insights does the optimization encoder perspective provide for decoder design? In particular, how does this perspective motivate the introduction of self-attention, rather than other architectural modifications that could also improve decoder expressivity?

  2. Does self-attention provide benefits beyond increased decoder expressivity? Could the authors compare MetaLF against parameter-matched alternatives without self-attention, such as deeper cross-attention decoders? Such comparisons would help clarify whether the improvements are specific to self-attention or can also be achieved through increased decoder capacity.

4 comments
AuthorsOpenReview ↗

Response to 3q3H, weakness 1

We thank the Reviewer for taking the time to provide a critical review of our work.

W1. Limited novelty

The second-order derivatives build on established MAML results, and we do not claim them as new. Our contribution is the optimization encoder perspective: it identifies how second-order differentiation trains the construction of a representation alongside its decoding, and thereby makes decoder architecture a design choice for the encoding procedure itself (Section 3, Eq. 2). This explains the benefit of second-order training and guides the architecture we propose. MetaLF introduces explicit communication among latent elements before decoding, which ENF (and other neural field methods) lacks. Its attention formulation couples content and pose: queries depend on the receiving latent’s content, while keys combine relative geometry with modulation by the source latent’s content, a strictly different interaction than ENF-style cross-attention (Section 4, Figure 4). These interactions preserve equivariance and shape both predictions and the gradient updates that construct representations. While this effect was already demonstrated in our polynomial fields experiments (Table 1), we now provide additional FOMAML vs. MAML experiments for image and shape reconstruction (Tables 2 and 3), highlighting how MetaLF loses much of its expressiveness advantage under first-order approximation. This further links decoder design to the encoding process and how MAML trains it.

ModelTrainingCIFAR10 PSNR (dB) ↑ShapeNet-Part IoU ↑ACDC Myo IoU ↑
ENFFOMAML23.736.726.4
ENFMAML42.465.537.3
MetaLFFOMAML29.435.132.8
MetaLFMAML47.976.777.9
Rows of Tables 2 and 3 in the revised manuscript.

Local latents trained with second-order MAML but without latent interaction (ENF) reach 37.3 IoU on the myocardium. Latent interaction without the second-order pathway that trains it (MetaLF, FOMAML) reaches 32.8 IoU and loses its advantage over ENF on ShapeNet. Together they reach 77.9 IoU. The same combination yields 42.6% top-1 accuracy on ImageNet-1K with reconstruction-only adaptation at test time, against 24.1% for the strongest prior end-to-end method (Section 5.2, Table 4). It is this combination and framing of existing techniques that leads to substantially better performance across benchmarks, and it opens a design space for future work in which latent geometry, communication among latents, and task supervision are explicit choices.

We have revised the start of the Section 1 contribution statements to distinguish the conceptual contribution and its consequences for design more clearly. We hope this makes the scope and substance of our novelty more clear.

AuthorsOpenReview ↗

Response to 3q3H, weakness 2

W2. Unclear benefits of the optimization encoder perspective

We have revised the manuscript to state the benefits of the optimization encoder perspective more clearly (see R3q3H-W1), and have added results that support them. First, the perspective makes decoder architecture a design choice for encoding: the newly added Appendix A.4.3 proves that self-attention reshapes the inner-loop update in a way that per-latent learning rates cannot. Second, it identifies second-order differentiation as the pathway that trains the encoder. On polynomial fields, self-attention adds 22.0 dB under MAML but only 5.1 dB under FOMAML (Table 1), and we now provide FOMAML counterparts for image and shape reconstruction (Tables 2 and 3), where removing this pathway reduces MetaLF from 47.9 to 29.4 dB on CIFAR10. Third, it separates the inner from the outer objective, which enables end-to-end classification and segmentation with reconstruction-only adaptation (Section 3.3, Tables 4 and 5). These three consequences are now summarized at the end of Section 3.

Regarding the motivation for self-attention, the encoder-decoder perspective motivates it as a natural step towards a neural field that allows for non-local interaction between latents (at both encoding and decoding stages) as regular vision models typically do (e.g. CNNs and ViTs). Nevertheless, we agree that MetaLF is only one design informed by this perspective, and it does not exhaust the design space, as mentioned in the Limitations and future work section. Following the Reviewer’s suggestion, we test both a wider decoder, as well as the proposed deeper cross-attention decoder.

We define this new decoder as follows: The first layer queries coordinates and returns a representation through cross-attention with the latent pointcloud. Subsequent layers use this representation as the new query, which once again attends to its nearest K latents in the pointcloud (repeated 4 times). A projection layer subsequently returns the field value. We match the number of parameters to MetaLF (NSA=4N_{\mathrm{SA}}=4, 1.9M parameters) (see Appendix A.4.4).

We run the new decoders on both the polynomial fields dataset and CIFAR10. On CIFAR10, where signals are largely explained locally, the deeper decoder essentially matches MetaLF (47.72 versus 47.89 dB), and the wider decoder (ENF + GeGLU) comes within 1.1 dB using four times the parameters (46.79 dB, 7.69M) (Table 7). On polynomial fields, where global coefficients must be inferred from sparse observations, neither closes the gap: the deeper and wider decoders reach 44.4 and 36.0 dB, versus 64.2 dB for MetaLF. Decoder capacity does not replace interaction between latents, as the latents remain fixed and are combined only within each query’s local neighbourhood. Importantly, the deeper decoder does not scale. Self-attention acts on the 25 latents once per signal, so its cost is independent of the number of query coordinates, whereas every cross-attention layer is evaluated for every coordinate and differentiated through all inner steps.

Memory added to one second-order training step1,024 coordinates4,096 coordinates16,384 coordinates
Four self-attention layers (MetaLF)0.3 GB0.3 GB0.3 GB
Four cross-attention layers (deeper decoder)8.6 GB34.6 GBexceeds a 94 GB GPU

On CIFAR10 this amounts to 4.1x the training time and 4.6x the memory of MetaLF, and the decoder cannot be trained at CelebA resolution, where MetaLF uses 35 GB (Appendix A.6.1). Matching MetaLF on CIFAR10 thus comes at a cost that grows with every query coordinate, whereas the cost of self-attention does not. We describe these new findings in Appendix A.4.4, which we refer to in Section 5.2 of the manuscript.

AuthorsOpenReview ↗

Response to 3q3H, weaknesses 3 & 4

W3. Unclear distinction from prior work and mathematical presentation

Our contribution is the optimization encoder interpretation of the inner-loop in meta-learning, and the insights it offers for neural field design: decoder architecture shapes both predictions and the gradient updates that construct representations, thus identifying second-order meta-learning as essential for end-to-end training (also see R3q3H-W1). The value of this simple encoder-decoder view is that it makes latent geometry, communication among latent variables, and task supervision explicit design choices. This reasoning motivates MetaLF’s interaction between latent content and relative pose, as well as end-to-end task learning through reconstruction-based adaptation. We have revised the start of the Section 1 contributions statement as well as the Conclusion to clarify this scope.

Regarding mathematical presentation, the auxiliary notation in Section 3.2 is mostly relevant for the full derivation in Appendix A.2. We have therefore removed the intermediate Jacobian recurrence and its auxiliary variables from Section 3.2, thus foregrounding the encoder-decoder interpretation more clearly while preserving the initial mathematical justification. We thank the Reviewer for this note on clarity.

W4. Insufficient ablation of the proposed self-attention mechanism

We now provide parameter-matched controls in the newly added Appendix A.4.4 (Table 7). We compare MetaLF against two alternatives without self-attention: the suggested deeper cross-attention decoder (R3q3H-W2), matched in parameter count (1.87M), and a wider decoder (ENF + GeGLU, 7.69M).

ModelNSAN_{\mathrm{SA}}Latents interactCIFAR10 Params (M)CIFAR10 PSNR (dB) ↑Polynomial Params (M)Polynomial PSNR (dB) ↑
ENF-no0.6142.680.1844.15 (0.56)
ENF + GeGLU (wider)-no7.6946.791.9535.96 (1.57)∗^*
ENF + GeGLU (block-matched)-no0.8145.05--
MetaLF0no0.3645.930.3542.28 (0.13)
MetaLF + deep cross-attention0no1.8747.721.8744.42 (1.98)
MetaLF4yes1.8847.891.8764.24 (0.82)
Table 7 of the revised manuscript, with a column added that marks whether latents exchange information. CIFAR10: single seed; polynomial fields: mean (SD) over three seeds. ∗^*: overfit.

On CIFAR10, where densely observed images are largely explained by local structure, added decoder capacity recovers most of the gain: the deeper decoder reaches 47.72 dB against 47.89 dB for MetaLF (45.93 dB without self-attention), and the wider decoder reaches 46.79 dB with four times the parameters. On polynomial fields, where global coefficients must be inferred from sparse observations, it does not: the deeper and wider decoders reach 44.4 and 36.0 dB (overfit), against 64.2 dB for MetaLF. Both controls leave the latents fixed and combine only the nearest latents of each query, so neither lets latents exchange information. Among these models, MetaLF with self-attention is the only one that reconstructs both local and non-local signals well, and it does so at a cost that does not grow with the number of query coordinates: the deeper decoder is evaluated for every coordinate, which costs 4.1x the training time and 4.6x the memory of MetaLF on CIFAR10 and prevents training at CelebA resolution (see 3q3H-W2).

Appendix A.4.3 now also analyses how self-attention couples the latent tokens in the inner-loop update. We test this directly on trained MetaLF models by blocking only the cross-token gradient paths while keeping the forward computation, and therefore also decoder capacity, unchanged (Table 6). Under MAML, PSNR drops from 64.2 to 36.2 dB on polynomial fields, below MetaLF without self-attention (42.3 dB), and from 47.9 to 29.9 dB on CIFAR10. Under FOMAML the drops are far smaller (38.3 to 31.2 dB and 29.1 to 23.4 dB). The benefit of self-attention is therefore tied to how latents are constructed during encoding, and to second-order training of that process, rather than to decoder expressivity alone.

AuthorsOpenReview ↗

Response to 3q3H, questions

Q1. Concrete insights of the optimization encoder perspective

The optimization encoder perspective shows that meta-learning a neural field can be interpreted as training a regular encoder-decoder architecture, in which the inner-loop updates form the encoder and are parameterized by the decoder itself. Discarding the second-order gradient through the decoder (FOMAML) then trains the decoder without training this encoding process directly (Eq. 2). For decoder design, this means a decoder should be judged by the latent updates its gradients produce, and not only by the fields it can represent (see R3q3H-W2 for the full list of insights).

Self-attention follows from this. In ENF, each latent is updated only from observations in its own neighbourhood, whereas standard vision encoders let local observations inform non-local structure. Self-attention brings this interaction into the encoder, in a way that per-latent learning rates cannot reproduce (Appendix A.4.3). Modifications that only increase decoder expressivity change how latents are read out, but leave their updates independent per latent; we compare against these in Appendix A.4.4 (see R3q3H-W2 and R3q3H-W4).

Q2. Self-attention benefits beyond increased decoder expressivity

We now compare MetaLF against parameter-matched alternatives without self-attention, including the suggested deeper cross-attention decoder, in Appendix A.4.4 (see R3q3H-W4). On CIFAR10, where images are largely explained by local structure, added decoder capacity recovers most of the gain (47.72 versus 47.89 dB), though at 4.1x the training time and 4.6x the memory. On polynomial fields, where global structure must be inferred from sparse observations, it does not close the gap.

The benefit specific to self-attention is a more expressive encoder. Blocking only the cross-token gradient paths in a trained MetaLF, with its forward computation and hence its capacity unchanged, reduces PSNR from 64.2 to 36.2 dB on polynomial fields, below the model without self-attention (Appendix A.4.3). Decoder capacity can therefore substitute for self-attention on local signals, but not for the interaction between latents during encoding.

Reviewer_HXLbOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper frames the inner loop of meta-learned conditional Neural Fields (cNFs) as an optimization encoder. The authors argue that this optimization procedure basically resembles an encoder that forms a latent from observations. Under this view, the second-order optimization that is widely applied in these settings backpropagates the outer-loop loss to the unrolled encoder, similar to end-to-end training in vanilla encoder-decoder architectures. The authors draw two mai conclusions from this observation: (1) the decoder architecture also influences the encoder capabilities, and (2) the inner and outer objectives can be decoupled, i.e., we can use different loss terms for the construction of the latent (encoder/inner loop) and the prediction (decoder/outer loop).

Based on this, the authors then propose MetaLF that extends previous Equivariant Neural Fields (ENFs) with a SE(n) invariant self-attention mechanism. The authors argue that this allows local observations to inform non-local structure thorught the gradient updates (the encoder). They evaluate their proposed architecture on image and shape reconstruction, as well as on meta-classification and -segmentation (using their decoupled objective).

Critical strengths

I generally like the way the authors frame the inner loop as an optimization encoder. While the observation that second-order optimization plays an important part in training these NFs is not new, framing it in this way is interesting and nicely motivates the design choices the authors make. I think this view on what actually happens in the inner-loop is helpful for the community in making more informed design decisions and provides interesting insights into these methods.

I especially like the way the authors decouple the inner- and outer-loop training objectives and demonstrate that additional outer-loop objectives improve downstream performance and allows to e.g. perform segmentation through image fitting only (not new, but nicely demonstrated within this framework). I do also like that the authors evaluate their method across a set of different tasks and modalities, demonstrating strong empirical performance across the majority of them.

Critical weaknesses

While I generally like this paper, I have some concerns:

  • The demonstrated gains are not clearly attributed to the proposed self-attention mechanism. The authors claim that MetaLF works better due to the latents exchanging information through self-attention. However, MetaLF without this mechanism already ouperforms the ENF baseline (on CIFAR10), which suggests that part of these gains stem from other design choices. This should clearly be mentioned and ideally disentangled.

  • It remains questionable wether the reported gains really stem from a better encoder or if a better decoder could also explain them. If the self-attention mechanism, as claimed by the authors, really improves the encoding, turning off second-order optimization should hurt MetaLF more then ENF. While this is demonstrated on the polynomial fields, I'd really like to see the effect on the main benchmarks.

  • Some baselines look really weak. Functa for example performs poorly on shape reconstruction and completely fails on some segmentation tasks. While I appreciate that the authors include other task specific methods, including stronger Functa-like methods (like the conceptually very similar MedFuncta) could strengthen the evaluation.

Questions for authors
  • Which design choices explain why MetaLF without self-attention already outperforms ENF on CIFAR10? Can you provide a step-by-step ablation from ENF to MetaLF?
  • The Appendix suggests that Functa performs poorly on shape reconstruction and fails on segmentation across several attempted settings. Is this failure inherent to global latents with few inner steps, or could Functa with more inner steps, or a scaled variant such as MedFuncta, produce better results?
3 comments
AuthorsOpenReview ↗

Response to HXLb, weaknesses 1 & 2

We appreciate the Reviewer’s remarks on our method framing, and address any weaknesses in a point-by-point manner here.

W1. Disentanglement of self-attention gains

We appreciate the comment, as it allowed us to design additional experiments that disentangle the gains of self-attention more clearly.

So far, we showed that the difference between ENF and MetaLF (NSA=0N_{\mathrm{SA}}=0) can largely be attributed to a GeGLU nonlinearity, vastly improving the expressivity of the network post cross-attention (Section 4 and Appendix A.3.1). We now extend this with a disentangling analysis (see below) and in Appendix A.4.4.

StepModelNSAN_{\mathrm{SA}}Params (M)CIFAR10 PSNR (dB) ↑GainPolynomial fields PSNR (dB) ↑
ENF-0.6142.6844.15 (0.56)
+ GeGLU block (block-matched)ENF + GeGLU-0.8145.05+2.37-
+ smaller cross-attention head, context projectionMetaLF00.3645.93+0.8842.28 (0.13)
+ four self-attention layersMetaLF41.8847.89+1.9664.24 (0.82)
Rows of Table 7 in the revised manuscript. CIFAR10: single seed; polynomial fields: mean (SD) over three seeds.

To summarize, adding a GeGLU block matched to MetaLF’s block size improves ENF by 2.37 dB, accounting for most of the gap to MetaLF without self-attention. Adding self-attention subsequently improves MetaLF by an additional 1.96 dB. We would like to note that this gain is specific to CIFAR10. On the more structurally coherent polynomial fields dataset in Table 1, MetaLF without self-attention offers no improvement over ENF, whereas adding self-attention yields substantial gains, further supporting its role in exploiting shared structure. We verify this additionally by similarly ablating MetaLF components on the polynomial fields dataset, as well as comparing to different parameter-matched models in terms of width and depth. These results further illustrate how non-local interactions in MetaLF vastly improve reconstruction on data with non-local structure (Appendix A.4.4).

W2. FOMAML on the main benchmarks

We agree that performing this same FOMAML vs MAML analysis on main benchmarks would offer additional insights on the encoding mechanism of MetaLF. We therefore rerun the ENF and MetaLF models in Table 2 and Table 3 with FOMAML, and now additionally report these numbers in said tables. As anticipated, MetaLF suffers from a higher drop in reconstruction quality when compared to ENF, and performance gain becomes much narrower or even comparable to ENF, thus further supporting the importance of training the encoding process. We discuss these findings at the end of the shape reconstruction paragraph (Section 5.2). Furthermore, as suggested by Reviewer hny8, we now also include a more in-depth analysis on how self-attention changes the inner-loop gradient field, which may be found in Appendix A.4.3.

ModelTrainingCIFAR10CelebAImageNet
ENFMAML42.4 (0.2)35.7 (0.1)27.6 (0.0)
ENFFOMAML23.7 (-)23.7 (-)23.6 (-)
change−18.7−12.0−4.0
MetaLFMAML47.9 (0.1)39.5 (0.1)31.9 (0.0)
MetaLFFOMAML29.4 (-)27.4 (-)25.7 (-)
change−18.5−12.1−6.2
Rows of Table 2 in the revised manuscript (SE(2)\mathrm{SE}(2) bi-invariant). Test PSNR (dB) ↑.
ModelTrainingShapeNet-PartShapeNet-CoreACDC LVACDC MyoACDC RV
ENFMAML65.5 (0.5)67.2 (1.9)77.1 (2.1)37.3 (2.6)64.7 (1.1)
ENFFOMAML36.7 (-)39.6 (-)59.2 (-)26.4 (-)49.1 (-)
change−28.8−27.6−17.9−10.9−15.6
MetaLFMAML76.7 (1.5)78.2 (4.0)89.3 (0.5)77.9 (0.4)79.5 (2.7)
MetaLFFOMAML35.1 (-)35.2 (-)70.2 (-)32.8 (-)40.8 (-)
change−41.6−43.0−19.1−45.1−38.7
Rows of Table 3 in the revised manuscript. IoU ↑.
AuthorsOpenReview ↗

Response to HXLb, weakness 3 + questions

W3. Weak baselines

We appreciate the Reviewer’s note on the Functa baselines, encouraging us to re-evaluate the optimization setting for this baseline. While we experimented extensively with different optimizer settings, mostly based around combinations of settings provided by Dupont et al. (Functa) and Bauer et al. (SpatialFuncta), we had not yet applied the more finetuned settings proposed in MedFuncta. We therefore re-optimize all Functa models using the optimization settings proposed in MedFuncta, significantly boosting performance for both shape reconstruction (Table 3) and meta-segmentation (Table 5). Additionally, we test the full MedFuncta setting, i.e. a 15-layer network with a 2-16x larger latent vector. We report the best-performing version in the updated tables (now denoting the use of MedFuncta with ‘M’), and reflect any changes to the final optimization setting in Appendix A.5. While these results significantly improve the Functa baselines, MetaLF still outperforms all baselines by a large margin.

ShapeNet-PartShapeNet-CoreACDC LVACDC MyoACDC RV
Functa (original submission)4.0 (0.1)5.0 (0.0)1.2 (0.0)1.1 (0.0)1.2 (0.0)
Functa (revised)71.1 (1.3)64.1 (7.0)71.9 (5.1)49.9 (4.0)53.0 (9.2)
MetaLF76.7 (1.5)78.2 (4.0)89.3 (0.5)77.9 (0.4)79.5 (2.7)
Shape reconstruction IoU ↑ (Table 3 of the revised manuscript, with the Functa row of the original submission added). Mean (SD) over three seeds.
OMBRIA IoU ↑ShapeNet-Part mIoU ↑OASIS Dice ↑
Functa (original submission)39.9FailedFailed
Functa (revised)39.682.1 (18.1)67.2 (1.2)M^{\mathrm{M}}
MetaLF72.383.4 (16.7)86.9 (0.9)
Meta-segmentation (few-step neural fields of Table 5 in the revised manuscript, with the Functa row of the original submission added).

Q1. Design choices explaining the difference between ENF and MetaLF

This can mostly be attributed to the additional GeGLU nonlinearity, as discussed more extensively in HXLb-W1. We now provide a step-by-step ablation in Appendix A.4.4 to disentangle this behaviour.

Q2. Poor performance of Functa

This failure was caused by the optimization recipe, and is not inherent to global latents with few inner steps. With the MedFuncta optimization settings and the same five inner steps, Functa now reaches 71.1/64.1 IoU on ShapeNet-Part/-Core and 82.1 mIoU on ShapeNet-Part segmentation (see HXLb-W3). The scaled MedFuncta model helps on image intensities (CIFAR10, ImageNet-1K, OASIS), but not on SDFs. Meta-classification on images is the exception: even the best Functa variant stay far below MetaLF in both accuracy and reconstruction quality (10.9% vs. 42.6% and 23.0 vs. 27.9 dB on ImageNet-1K; 60.9% vs. 83.3% and 28.5 vs. 37.8 dB on CIFAR10). Tables 3 to 5 have been updated to reflect these findings. Our intuition is that a single global latent must describe the entire signal in detail while also remaining abstract enough to separate classes, and that these two demands compete for the same vector. We now discuss these findings in Appendix A.5.3. We did not evaluate more inner steps, though we expect this to help further, which is likely why many-step auto-decoding remains standard for global-latent fields.

I'd like to thank the authors for their detailed response that clarified most of my questions (especially those regarding the baselines). I am in favor of accepting this paper and do think it is an interesting contribution. However, I also have some reservations regarding novelty and will therefore keep my score.

Reviewer_hny8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper reframes gradient-based latent meta-learning for conditional neural fields as an "optimization encoder": the K inner-loop latent updates constitute an encoder parameterized by the decoder, and full second-order (MAML) differentiation is the pathway by which the outer objective trains this encoding procedure — a pathway first-order approximations truncate. Motivated by this view, the authors propose MetaLF, an SE(n)-equivariant transformer that adds self-attention among latent pointcloud elements before ENF-style cross-attention decoding. On controlled polynomial fields, self-attention lowers latent effective rank toward the function family's intrinsic dimension and improves alignment/PSNR. MetaLF outperforms Functa, SpatialFuncta, and ENF on image reconstruction (CIFAR10, CelebA, ImageNet), SDF/shape reconstruction (ShapeNet, ACDC), and end-to-end meta-classification and meta-segmentation with reconstruction-only test-time adaptation.

Critical strengths
  1. Clean and useful conceptual framing. The optimization-encoder formalization (Eq. 2 decomposition into a direct decoder gradient and a gradient through the unrolled encoder) crisply explains what first-order methods discard and connects MAML to standard encoder–decoder backpropagation. The inner/outer objective disentanglement naturally justifies semantic outer supervision with reconstruction-only adaptation.

  2. Well-designed controlled study. The polynomial-field experiment is a nice mechanistic validation: effective rank converging toward the intrinsic dimension (≈28) and rising tangent fraction link the architectural change to encoding geometry, not just end metrics.

Critical weaknesses
  1. The theory–architecture link is partly suggestive. The optimization-encoder view is a reinterpretation (close to CAVIA/bilevel formulations and "optimization as encoding" of Xu et al.) rather than new theory; no formal result shows why latent self-attention improves the inner-loop gradient field specifically, beyond the empirical geometry metrics. MetaLF could be motivated purely as "ENF + latent self-attention" (a modest architectural delta over ENF and Zhang et al.'s 3DShape2VecSet-style attention).

  2. Fairness of some comparisons. The classification baselines (Functa/ENF at 4–7% on ImageNet) suggest they may be poorly suited or under-tuned for that setting; segmentation comparisons against NISF/MetaSeg use different protocols, as acknowledged. Compute cost of second-order training and KSA-neighborhood attention is deferred to the appendix.

  3. Limited analysis of failure modes. The limitations note weaker gains on heterogeneous data and oversmoothing with global attention, but these are not quantified in the main text.

Questions for authors

refer to weaknesses

5 comments
AuthorsOpenReview ↗

Response to hny8, weakness 1

We thank the Reviewer for their positive remarks and critical review.

W1. The theory-architecture link is partly suggestive

The optimization encoder perspective is our main contribution, and the updated revision supports it with a formal result and a direct test. Our perspective proposes that if the decoder parameterizes the encoder, then latent self-attention (SA) should change the inner-loop update itself, and only second-order training should make full use of this.

To formalize why latent self-attention (SA) changes the inner-loop gradient field, we have added a theoretical analysis in Appendix A.4.3. Here, we prove that SA can reshape the inner-loop update itself by sharing gradients written into one token with its neighbours, which no tuning of per-latent learning rates can reproduce. Importantly, as this effect acts through the encoder term described in Eq. 2, only second-order MAML trains it directly. We empirically verify this behaviour in a new experiment, for which we analyse the encoding behaviour of MetaLF for both polynomial fields and CIFAR10 (Table 6, see below).

DataTrainingPSNR fullPSNR localResidual fullResidual local
PolynomialMAML64.2 (0.8)36.2 (4.3)0.028%0.47%
PolynomialFOMAML38.3 (0.8)31.2 (0.4)1.0%10.1%
CIFAR10MAML47.9 (0.1)29.9 (0.3)0.31%26.2%
CIFAR10FOMAML29.1 (0.4)23.4 (1.1)2.7%3.1%
Table 6 of the revised manuscript. 'Full' uses the original gradients, 'local' removes token-to-token routing in the backward pass. PSNR in dB after K=3K=3 updates with the best of nine step-size scales; mean (SD) over three seeds.

We implement this by using the Table 1 and 2 checkpoints and blocking only the cross-token gradient paths, which reduces PSNR from 64.2 to 36.2 dB even after retuning the step size, below the model without SA (42.3 dB), whereas FOMAML loses far less (38.3 to 31.2 dB). The combination of these findings further supports that SA is more than an architectural delta over ENF here, as its benefit is specific to gradient-based encoding, which is why it adds 22.0 dB under MAML but only 5.1 dB under FOMAML (see Table 1). We now refer to this formal analysis in Section 5.1 of the updated manuscript (2nd paragraph).

AuthorsOpenReview ↗

Response to hny8, weakness 2

W2. Fairness of some comparisons

Classification baselines. As also suggested by Reviewer HXLb, we took this opportunity to re-evaluate our optimization of the Functa baseline for classification, and retrain it as proposed in MedFuncta. We now report the full MedFuncta model on images, which raises ImageNet-1K top-1 accuracy from 7.4% to 10.9% (Table 4, see below). Meta-classification thus remains the outlier among our baselines, with SpatialFuncta and ENF, which use localized latents, reaching ~4%. We argue that these methods fall short for different reasons. A global latent, as in (Med)Functa, must describe the entire signal in detail while also remaining abstract enough to separate classes, and these two demands compete for the same vector. Local representations offer better control of fine-grained detail, but nothing combines local evidence into a semantic representation. This is precisely what MetaLF addresses: self-attention lets localized latents interact, and second-order meta-learning trains this interaction end-to-end with the classification objective. MetaLF reaches 42.6% top-1 accuracy on ImageNet-1K, against 24.1% for the strongest prior end-to-end meta-classification method (MWT). We have updated our findings on meta-classification (Appendix A.5.3) to reflect this.

ImageNet PSNR ↑ImageNet Acc. ↑CIFAR10 PSNR ↑CIFAR10 Acc. ↑ShapeNet-Core IoU ↑ShapeNet-Core Acc. ↑
Functa (original submission)21.6 (2.5)7.4 (0.1)23.1 (1.9)57.7 (0.5)25.6 (8.0)1.8 (9.4)
Functa (revised)23.0 (2.4)M^{\mathrm{M}}10.9 (0.1)M^{\mathrm{M}}28.5 (1.7)M^{\mathrm{M}}60.9 (0.5)M^{\mathrm{M}}43.3 (15.2)69.0 (26.0)
MetaLF27.9 (2.9)42.6 (0.2)37.8 (2.4)83.3 (0.4)55.3 (16.9)70.8 (24.7)
Table 4 of the revised manuscript, with the Functa row of the original submission added. M^{\mathrm{M}}: full MedFuncta model.

Segmentation protocols. We agree that different protocols limit direct comparability. We selected tasks with existing published baselines and used the authors’ reported scores to preserve their original method designs and task-specific optimization. We now state this more directly in the updated manuscript (Section 5.2, meta-segmentation). However, since NISF utilizes a latent vector to represent its cohort, we are currently retraining the method ourselves in both the original auto-decoder fashion and using MAML to provide directly comparable numbers. We will update Table 5 when these results come in. As it stands, MetaLF is the best few-step neural field on all three datasets. On OASIS it exceeds the published NISF score by 5.9 Dice and comes within 4.1 Dice of MetaSeg while using five fitting steps instead of 100.

Computational costs: Regarding computational costs of second-order training and SA, this was mostly deferred to the Appendix due to a combination of space limitations and the relative similarity of computational costs amongst methods, with SpatialFuncta being the main outlier. To clearly communicate this, we now summarize computational findings after the sentence where we refer to this analysis in the main manuscript (Section 5.2, image reconstruction).

AuthorsOpenReview ↗

Response to hny8, weakness 3

W3. Limited analysis of failure modes

We appreciate this comment and strengthen our analysis with additional experiments and analyses of failure modes. Appendix A.6.3, and rewrite the Limitations paragraph in the updated manuscript around these measured numbers.

Experiment 1: Regarding heterogeneity, MetaLF is limited by the amount of non-local structure within a signal, rather than diversity across samples. We use Gaussian-process (GP) textures with controlled correlation length to measure this directly. We train on a mixture of GP lengths, for which self-attention adds 0.3 to 2.7 dB at individual GP lengths. The failure mode we found lies at the opposite end: on homogeneous, very smooth fields that every model fits above 58 dB, second-order training with self-attention becomes unstable. Its contribution is then unreliable (−0.3 ± 1.3 and −1.9 ± 2.1 dB over three seeds), and at the smoother length MetaLF falls below ENF, whereas MetaLF without self-attention stays above ENF. We update the Limitations and future work section with this finding, and add Table 16 in Appendix A.6.3:

ℓ\ellENFMetaLF (NSA=0N_{\mathrm{SA}}=0)MetaLF (NSA=4N_{\mathrm{SA}}=4)SA (single ℓ\ell)SA (mixture)MetaLF − ENF (mixture)
0.0325.59$^\dagger$26.49$^\dagger$26.67†^\dagger+0.18$^\dagger$+0.28 (0.04)+1.68$^\dagger$
0.0538.91$^\dagger$41.15 (0.23)42.70 (0.75)+1.55 (0.98)+1.04 (0.41)+4.41$^\dagger$
0.158.36$^\dagger$59.86 (0.41)59.58 (1.70)−0.28 (1.29)+1.96 (0.18)+5.27$^\dagger$
0.263.66 (1.17)64.04 (1.20)62.09 (0.91)−1.94 (2.09)+2.68 (0.35)+5.41$^\dagger$
0.461.83$^\dagger$63.32†^\dagger59.72$^\dagger$−3.59$^\dagger$+2.63 (0.30)+5.16$^\dagger$
Table 16 of the revised manuscript. PSNR (dB) on 1,024 held-out images per GP length; the three model columns are trained on a single ℓ\ell. Mean (SD) over three seeds, †^\dagger single seed. SA: MetaLF (NSA=4N_{\mathrm{SA}}=4) − MetaLF (NSA=0N_{\mathrm{SA}}=0).

Experiment 2: Regarding global attention, we train all four combinations of local and global self- and cross-attention on CIFAR10 (Table 17, see below). Both global self-attention and global cross-attention impact output reconstruction minimally at ~0.1dB difference. We find that it is the combination of global self- and cross-attention that results in a PSNR drop of ~2.7dB. Our original statement in the limitations section therefore holds for this combination only, and we have updated Limitations and future work in the updated manuscript to reflect this. We retain local neighborhoods for training stability and cost.

Self-attn.Cross-attn.EpochsPSNRLocalDifferenceHF error
locallocal (final MetaLF)130-15047.65--0.0081
globallocal130-15047.7847.65+0.13 (0.03)0.0078 / 0.0081
localglobal60-8046.1245.98+0.14 (0.22)0.0113 / 0.0112
globalglobaldiverged in epoch 6 (seed 42)
globalglobal (seed 7)110-13044.6247.26−2.65 (0.80)0.0142 / 0.0088
globalglobal (meta-SGD rate 0.02)30-5041.7143.59−1.87 (1.70)0.0250 / 0.0179
Table 17 of the revised manuscript. Test PSNR (dB) on 1,000 images, averaged over the listed checkpoint epochs, next to the local model at the same epochs; difference is mean (SD across checkpoints). HF error: fraction of high-frequency energy that is not reconstructed, model / local model.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to Reviewers

We would like to thank all Reviewers for reviewing our work in detail. To the best of our ability, we have addressed all remaining weaknesses and questions. We have further uploaded an updated manuscript in which all changes are highlighted in blue.

We refer to the individual official comments for each reviewer for a detailed answer to each concern, and look forward to interacting during the discussion period.

AuthorsOpenReview ↗

End of Discussion period

Dear Reviewers,

Thank you again for your consideration and constructive feedback. We are pleased to see that the follow-up comments recognize the value of the additional experiments and analyses, and that our revisions have addressed many of the concerns raised.

With the discussion period coming to an end, we would welcome any further feedback, especially if you have not yet had a chance to revisit our responses and the revised manuscript. Where you feel the changes warrant it, we would be grateful if you could update your assessments and scores.

We remain available to clarify any outstanding questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.