- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper interprets the latent optimization process in conditional neural fields as an optimization encoder and shows how second-order meta-learning allows the outer objective to train this encoding process through the decoder. Motivated by the observation that the decoder determines both field predictions and the gradient updates used to construct latent representations, the authors propose Attentive Latent Fields (MetaLF), an equivariant transformer-based neural field decoder that introduces self-attention among spatial latent tokens. Experiments on polynomial fields, image and 3D shape reconstruction, and semantic prediction tasks (classification and segmentation) demonstrate the effectiveness of the proposed architecture.
Critical strengths
Strong empirical performance across diverse tasks. The authors conduct extensive experiments across image and 3D shape reconstruction, classification, and segmentation, demonstrating substantial improvements over existing neural-field baselines on multiple benchmarks with only a few latent optimization steps.
Critical weaknesses
-
Limited novelty. The second-order differentiation analysis largely follows established results from MAML and related meta-learning methods, while the proposed architecture primarily integrates existing self-attention mechanisms into equivariant neural fields. The novelty therefore appears to lie mainly in combining existing techniques, rather than introducing new theoretical results or fundamentally new architectural components.
-
Unclear benefits of the optimization encoder perspective. Although the authors reinterpret latent optimization as an encoding process, it is unclear what additional insights this perspective provides. In particular, the motivation for introducing self-attention does not seem to follow directly from the analysis. Other architectural modifications, such as increasing the depth of the existing cross-attention decoder, could also enable richer interactions among latent features and potentially improve performance without requiring explicit latent self-attention.
-
Unclear distinction from prior work and mathematical presentation. The paper does not clearly distinguish its own contributions from established techniques and results, making it difficult to identify which insights are genuinely novel. Additionally, the mathematical exposition is unnecessarily complicated in places. For example, Section 3.2 introduces several auxiliary variables (, , and ) to derive the second-order meta-gradient, making the main argument harder to follow.
-
Insufficient ablation of the proposed self-attention mechanism. Although the authors provide ablations of self-attention depth, it remains unclear whether the observed improvements arise specifically from self-attention or simply from increased decoder expressivity. For example, increasing the depth of the existing cross-attention decoder could also enable richer interactions among latent features. Comparisons with parameter-matched alternatives without self-attention would help establish whether self-attention provides benefits beyond increased decoder capacity.
Questions for authors
-
What concrete insights does the optimization encoder perspective provide for decoder design? In particular, how does this perspective motivate the introduction of self-attention, rather than other architectural modifications that could also improve decoder expressivity?
-
Does self-attention provide benefits beyond increased decoder expressivity? Could the authors compare MetaLF against parameter-matched alternatives without self-attention, such as deeper cross-attention decoders? Such comparisons would help clarify whether the improvements are specific to self-attention or can also be achieved through increased decoder capacity.
4 comments
Response to 3q3H, weakness 2
W2. Unclear benefits of the optimization encoder perspective
We have revised the manuscript to state the benefits of the optimization encoder perspective more clearly (see R3q3H-W1), and have added results that support them. First, the perspective makes decoder architecture a design choice for encoding: the newly added Appendix A.4.3 proves that self-attention reshapes the inner-loop update in a way that per-latent learning rates cannot. Second, it identifies second-order differentiation as the pathway that trains the encoder. On polynomial fields, self-attention adds 22.0 dB under MAML but only 5.1 dB under FOMAML (Table 1), and we now provide FOMAML counterparts for image and shape reconstruction (Tables 2 and 3), where removing this pathway reduces MetaLF from 47.9 to 29.4 dB on CIFAR10. Third, it separates the inner from the outer objective, which enables end-to-end classification and segmentation with reconstruction-only adaptation (Section 3.3, Tables 4 and 5). These three consequences are now summarized at the end of Section 3.
Regarding the motivation for self-attention, the encoder-decoder perspective motivates it as a natural step towards a neural field that allows for non-local interaction between latents (at both encoding and decoding stages) as regular vision models typically do (e.g. CNNs and ViTs). Nevertheless, we agree that MetaLF is only one design informed by this perspective, and it does not exhaust the design space, as mentioned in the Limitations and future work section. Following the Reviewer’s suggestion, we test both a wider decoder, as well as the proposed deeper cross-attention decoder.
We define this new decoder as follows: The first layer queries coordinates and returns a representation through cross-attention with the latent pointcloud. Subsequent layers use this representation as the new query, which once again attends to its nearest K latents in the pointcloud (repeated 4 times). A projection layer subsequently returns the field value. We match the number of parameters to MetaLF (, 1.9M parameters) (see Appendix A.4.4).
We run the new decoders on both the polynomial fields dataset and CIFAR10. On CIFAR10, where signals are largely explained locally, the deeper decoder essentially matches MetaLF (47.72 versus 47.89 dB), and the wider decoder (ENF + GeGLU) comes within 1.1 dB using four times the parameters (46.79 dB, 7.69M) (Table 7). On polynomial fields, where global coefficients must be inferred from sparse observations, neither closes the gap: the deeper and wider decoders reach 44.4 and 36.0 dB, versus 64.2 dB for MetaLF. Decoder capacity does not replace interaction between latents, as the latents remain fixed and are combined only within each query’s local neighbourhood. Importantly, the deeper decoder does not scale. Self-attention acts on the 25 latents once per signal, so its cost is independent of the number of query coordinates, whereas every cross-attention layer is evaluated for every coordinate and differentiated through all inner steps.
| Memory added to one second-order training step | 1,024 coordinates | 4,096 coordinates | 16,384 coordinates |
|---|---|---|---|
| Four self-attention layers (MetaLF) | 0.3 GB | 0.3 GB | 0.3 GB |
| Four cross-attention layers (deeper decoder) | 8.6 GB | 34.6 GB | exceeds a 94 GB GPU |
On CIFAR10 this amounts to 4.1x the training time and 4.6x the memory of MetaLF, and the decoder cannot be trained at CelebA resolution, where MetaLF uses 35 GB (Appendix A.6.1). Matching MetaLF on CIFAR10 thus comes at a cost that grows with every query coordinate, whereas the cost of self-attention does not. We describe these new findings in Appendix A.4.4, which we refer to in Section 5.2 of the manuscript.
Response to 3q3H, weaknesses 3 & 4
W3. Unclear distinction from prior work and mathematical presentation
Our contribution is the optimization encoder interpretation of the inner-loop in meta-learning, and the insights it offers for neural field design: decoder architecture shapes both predictions and the gradient updates that construct representations, thus identifying second-order meta-learning as essential for end-to-end training (also see R3q3H-W1). The value of this simple encoder-decoder view is that it makes latent geometry, communication among latent variables, and task supervision explicit design choices. This reasoning motivates MetaLF’s interaction between latent content and relative pose, as well as end-to-end task learning through reconstruction-based adaptation. We have revised the start of the Section 1 contributions statement as well as the Conclusion to clarify this scope.
Regarding mathematical presentation, the auxiliary notation in Section 3.2 is mostly relevant for the full derivation in Appendix A.2. We have therefore removed the intermediate Jacobian recurrence and its auxiliary variables from Section 3.2, thus foregrounding the encoder-decoder interpretation more clearly while preserving the initial mathematical justification. We thank the Reviewer for this note on clarity.
W4. Insufficient ablation of the proposed self-attention mechanism
We now provide parameter-matched controls in the newly added Appendix A.4.4 (Table 7). We compare MetaLF against two alternatives without self-attention: the suggested deeper cross-attention decoder (R3q3H-W2), matched in parameter count (1.87M), and a wider decoder (ENF + GeGLU, 7.69M).
| Model | Latents interact | CIFAR10 Params (M) | CIFAR10 PSNR (dB) ↑ | Polynomial Params (M) | Polynomial PSNR (dB) ↑ | |
|---|---|---|---|---|---|---|
| ENF | - | no | 0.61 | 42.68 | 0.18 | 44.15 (0.56) |
| ENF + GeGLU (wider) | - | no | 7.69 | 46.79 | 1.95 | 35.96 (1.57) |
| ENF + GeGLU (block-matched) | - | no | 0.81 | 45.05 | - | - |
| MetaLF | 0 | no | 0.36 | 45.93 | 0.35 | 42.28 (0.13) |
| MetaLF + deep cross-attention | 0 | no | 1.87 | 47.72 | 1.87 | 44.42 (1.98) |
| MetaLF | 4 | yes | 1.88 | 47.89 | 1.87 | 64.24 (0.82) |
| Table 7 of the revised manuscript, with a column added that marks whether latents exchange information. CIFAR10: single seed; polynomial fields: mean (SD) over three seeds. : overfit. |
On CIFAR10, where densely observed images are largely explained by local structure, added decoder capacity recovers most of the gain: the deeper decoder reaches 47.72 dB against 47.89 dB for MetaLF (45.93 dB without self-attention), and the wider decoder reaches 46.79 dB with four times the parameters. On polynomial fields, where global coefficients must be inferred from sparse observations, it does not: the deeper and wider decoders reach 44.4 and 36.0 dB (overfit), against 64.2 dB for MetaLF. Both controls leave the latents fixed and combine only the nearest latents of each query, so neither lets latents exchange information. Among these models, MetaLF with self-attention is the only one that reconstructs both local and non-local signals well, and it does so at a cost that does not grow with the number of query coordinates: the deeper decoder is evaluated for every coordinate, which costs 4.1x the training time and 4.6x the memory of MetaLF on CIFAR10 and prevents training at CelebA resolution (see 3q3H-W2).
Appendix A.4.3 now also analyses how self-attention couples the latent tokens in the inner-loop update. We test this directly on trained MetaLF models by blocking only the cross-token gradient paths while keeping the forward computation, and therefore also decoder capacity, unchanged (Table 6). Under MAML, PSNR drops from 64.2 to 36.2 dB on polynomial fields, below MetaLF without self-attention (42.3 dB), and from 47.9 to 29.9 dB on CIFAR10. Under FOMAML the drops are far smaller (38.3 to 31.2 dB and 29.1 to 23.4 dB). The benefit of self-attention is therefore tied to how latents are constructed during encoding, and to second-order training of that process, rather than to decoder expressivity alone.
Response to 3q3H, questions
Q1. Concrete insights of the optimization encoder perspective
The optimization encoder perspective shows that meta-learning a neural field can be interpreted as training a regular encoder-decoder architecture, in which the inner-loop updates form the encoder and are parameterized by the decoder itself. Discarding the second-order gradient through the decoder (FOMAML) then trains the decoder without training this encoding process directly (Eq. 2). For decoder design, this means a decoder should be judged by the latent updates its gradients produce, and not only by the fields it can represent (see R3q3H-W2 for the full list of insights).
Self-attention follows from this. In ENF, each latent is updated only from observations in its own neighbourhood, whereas standard vision encoders let local observations inform non-local structure. Self-attention brings this interaction into the encoder, in a way that per-latent learning rates cannot reproduce (Appendix A.4.3). Modifications that only increase decoder expressivity change how latents are read out, but leave their updates independent per latent; we compare against these in Appendix A.4.4 (see R3q3H-W2 and R3q3H-W4).
Q2. Self-attention benefits beyond increased decoder expressivity
We now compare MetaLF against parameter-matched alternatives without self-attention, including the suggested deeper cross-attention decoder, in Appendix A.4.4 (see R3q3H-W4). On CIFAR10, where images are largely explained by local structure, added decoder capacity recovers most of the gain (47.72 versus 47.89 dB), though at 4.1x the training time and 4.6x the memory. On polynomial fields, where global structure must be inferred from sparse observations, it does not close the gap.
The benefit specific to self-attention is a more expressive encoder. Blocking only the cross-token gradient paths in a trained MetaLF, with its forward computation and hence its capacity unchanged, reduces PSNR from 64.2 to 36.2 dB on polynomial fields, below the model without self-attention (Appendix A.4.3). Decoder capacity can therefore substitute for self-attention on local signals, but not for the interaction between latents during encoding.
Response to 3q3H, weakness 1
We thank the Reviewer for taking the time to provide a critical review of our work.
W1. Limited novelty
The second-order derivatives build on established MAML results, and we do not claim them as new. Our contribution is the optimization encoder perspective: it identifies how second-order differentiation trains the construction of a representation alongside its decoding, and thereby makes decoder architecture a design choice for the encoding procedure itself (Section 3, Eq. 2). This explains the benefit of second-order training and guides the architecture we propose. MetaLF introduces explicit communication among latent elements before decoding, which ENF (and other neural field methods) lacks. Its attention formulation couples content and pose: queries depend on the receiving latent’s content, while keys combine relative geometry with modulation by the source latent’s content, a strictly different interaction than ENF-style cross-attention (Section 4, Figure 4). These interactions preserve equivariance and shape both predictions and the gradient updates that construct representations. While this effect was already demonstrated in our polynomial fields experiments (Table 1), we now provide additional FOMAML vs. MAML experiments for image and shape reconstruction (Tables 2 and 3), highlighting how MetaLF loses much of its expressiveness advantage under first-order approximation. This further links decoder design to the encoding process and how MAML trains it.
Local latents trained with second-order MAML but without latent interaction (ENF) reach 37.3 IoU on the myocardium. Latent interaction without the second-order pathway that trains it (MetaLF, FOMAML) reaches 32.8 IoU and loses its advantage over ENF on ShapeNet. Together they reach 77.9 IoU. The same combination yields 42.6% top-1 accuracy on ImageNet-1K with reconstruction-only adaptation at test time, against 24.1% for the strongest prior end-to-end method (Section 5.2, Table 4). It is this combination and framing of existing techniques that leads to substantially better performance across benchmarks, and it opens a design space for future work in which latent geometry, communication among latents, and task supervision are explicit choices.
We have revised the start of the Section 1 contribution statements to distinguish the conceptual contribution and its consequences for design more clearly. We hope this makes the scope and substance of our novelty more clear.