- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper considers the problem of neural field weight initializations and investigates a ‘sequential’ initialization scheme, i.e. initializing a neural field with the weights learned from a previous sample. The authors identify 1. adaptability of the network weights to new observations and 2. training-history-retention of the weights as important properties of a neural field. The paper proposes a method to quantify this adaptation cost (i.e. how much the weights need to change to reconstruct a new image) under the framework of ‘adaptation geometry’ and find that a mismatch between the learned geometry and a new sample increases the adaptation cost. The results also show that after sequential learning, weights contain information about the history of the previous samples, tested in a setting where the final samples were identical while previous samples were different. This history enables better regime classification using the sequentially learned weights. The authors suggest these identified properties are useful beyond simply considering the reconstruction quality as a metric to evaluate neural fields.
Critical strengths
- The introduction of the adaptation geometry as an interpretation of adaptation cost in sequentially trained neural fields.
- The introduction of a method to quantify retained-history after sequential training and its links to adaptation geometry.
- The code being shared in the supplementary materials eases reproducibility concerns.
Critical weaknesses
Main concerns: I don’t understand when sequential fitting is useful or can be used in practice. I also don’t understand what new insights can be gained from the results.
Motivation: I am not sure why one needs to adapt a neural field to new images? Why replace adaptability with reconstruction quality? For example the abstract states: “These results show that neural fields contain useful information beyond what they currently reconstruct: in how they can change and in how they got there.” I am not sure why or in what scenarios this could be useful. For example, why does it matter that the neural field weights need a larger adaptation to fit the next frame, as long as the reconstruction quality doesn’t decay? Depending on the learning rate, task complexity, etc. the adaptation cost may not even translate to training time.
Interpretation: Isn’t the problem considered in Table 1 just domain shift? Is it possible to test, or at least discuss, some domain adaptation methods for this task?
Related work: I feel like related work is missing some discussion on domain adaptation, meta-learning, transfer learning, etc. in the broader computer vision/machine learning domain, outside of the specific context of neural fields. How to optimally initialize neural networks is a well-studied topic and it would be helpful to discuss how the presented analysis using neural fields provides further or different insights.
On a related note, what if we replaced neural fields with (simple or variational) auto-encoders for reconstructing each frame in a video? Would we find that sequential initializations are more useful than independent initializations (as described in Fig. 1(c))? As an example, the latent space of autoencoders are also regularly used for classification tasks. What is the difference of neural fields in this context?
Results: What is the reconstruction quality of independent vs. sequential initialization? For what use-cases does it make sense to replace adaptability with reconstruction quality?
For example, there is no ‘independent’ setting baseline comparison in Table 3, e.g. it is not clear what the reconstruction MSE would be for the ‘independent weights’ in comparison to sequential training. This makes it hard to quantify the reconstruction quality trade-off.
Regarding Fig. 2 and Table 2, I don’t find it surprising that the weights retain information from the history of a wave that they’ve been trained on (e.g. at a level of 68%), as opposed to having chance level predictions (50%) without being trained on the history. What is the insight gained from this result?
Considering the experiment shown in Fig. 2: Can the authors quantify retained memory as a function of change in samples/domain shift? Maybe as a function of wave speed/sampling rate?
Would retained history be more useful (e.g. actually increase the reconstruction quality) under partial observation as is described in sections 1 and 2 (“this is why explicit memory helps learned PDE surrogates under partial observation”)? And if the retained memory about a system’s past is useful under certain settings, e.g. partial observation, low resolution or with noise, why not also test this using reconstruction error with/without the memory/sequential learning? If sequential training improves both reconstruction quality and other metrics (e.g. increased regime classification accuracy, relatively reduced adaptation cost or training time) at the same time under partial observation, this would strengthen the results considerably.
Minor:
Fig. 1(a) was quite unclear to me. The caption mentions K_c, but there is no K_c in the figure. Similarly, the figure includes r_c and x, which are not explained in the caption, therefore I couldn’t figure out the message behind this figure.
Questions for authors
- Why/when would one need to adapt a neural field to new images? Why replace adaptability with reconstruction quality? In what scenarios could it be useful?
- When is ‘adaptation cost’ informative? Would adaptation cost in general translate to reconstruction quality or training time?
- Is the problem considered in Table 1 just domain shift? Is it possible to test, or at least discuss, some domain adaptation methods for this task?
- What if we replaced neural fields with auto-encoders, sequentially trained to reconstruct each frame in a video? Would we find that sequential initializations are more useful than independent initializations? What is the difference to neural fields in this context?
- Is it possible to add an ‘independent’ setting reconstruction MSE baseline comparison to Table 3? This can help quantify the adaptability-reconstruction quality trade-off.
- Training sequentially, as opposed to independently, trivially retains memory. What is the insight gained from Fig. 2 and Table 2?
- In Fig. 2, would it be possible to quantify retained memory as a function of change in samples/domain shift? Maybe as a function of wave speed/sampling rate?
- Would retained history be more useful under certain settings, e.g. partial observation, low resolution or with noise? Can the authors test this in terms of both the proposed metrics and reconstruction quality?
4 comments
Q5 Independent MSE in Table 3. Added. The table is now Table 4; all ten budgets are in App. F.3. Sequential fits have lower MSE at every budget.
Q6 Isn't the memory trivial? That sequential and independent weights differ is trivial, and explicit history solves the wave task by construction (92–100%). What is not guaranteed is that a single fitted state, trained only to reconstruct, carries linearly decodable hidden state:
- It recovers the hidden velocity (R² = 0.22, up to 0.60; exactly 0 from the current observation).
- Its reconstruction carries less, by 4–30 pp in every wave condition.
- The amount depends systematically on the dynamics (Q7).
- Further fitting can overwrite it (see Allen–Cahn above).
Q7 Wave speed and sampling (Table 3). Current-observation controls remain exactly 50%.
| Sweep (baseline 68.6%, R² 0.22) | ×0.5 | ×2 |
|---|---|---|
| History duration (via Δt) | 58.0% (R² 0.05) | 89.5% (R² 0.60) |
| Wave speed (Δt fixed) | 66.7% (R² 0.16) | 79.9% (R² 0.36) |
- Duration dominates. Halving it costs 10.6 pp [8.5, 12.7], while halving the speed costs 1.9 pp [0.0, 3.8].
- Frame count matters little. At fixed duration and 1,600 total updates, 8, 16 and 32 frames give 70.9%, 68.6% and 67.5%.
- Adjacent-frame change does not order the results.
Q8 Partial observation and joint improvement (App. E.4). Across observation densities of 100%, 25% and 6.25% and noise of 0, 5% and 10%:
- Memory is robust: 66.1–68.7%, every interval excluding zero.
- Joint improvement holds with clean, dense observations: sequential fits reconstruct better and retain more (Allen–Cahn and Gray–Scott at ≥ 5 steps/frame; wave endpoint MSE 2.7e-5 vs 2.2e-4).
- With sparse or noisy observations the two diverge: warm-started fits recover the clean field worse (6.25% density: 0.168 vs 0.054) while still retaining history.
We hope this clarifies why reconstruction quality alone is not enough, and would be grateful if you would reconsider your score.
I thank the authors for the rebuttal. The responses address my technical questions sufficiently.
Having said that, I think a strong motivation is still missing. While the authors clarified that this is a diagnostic, and not a new method, it is still unclear to me when and under what settings one might want to use this diagnostic. The trade-off with the accuracy or reconstruction quality is not always clear, 'adaptation cost' in general does not translate to practical metrics like reconstruction quality or training time, the results seem to be dataset and evaluation metric dependent, making it hard to estimate how the diagnostics will generalize.
Taken together, I will maintain my current score for now.
Thanks again for the hard work and the effort.
Thank you for engaging with our rebuttal.
As for who would use this, we believe it would be useful for anyone who fits neural fields to a sequence (video or simulation snapshots) and then reuses the fitted weights as features, as in functa-style classification or weight-space learning. Our results tell them:
- Not to tune the fitting budget by reconstruction error and to choose the budget with a downstream validation probe instead.
- With sparse or noisy observations, a trade-off appears between warm-starting and independent fitting. At 6.25% density, warm-started fits recover the clean field worse but still retain history. The choice then depends on which of the two is needed.
We agree that adaptation cost is not a proxy for other useful metrics. Its role is to show that two networks with the same reconstruction can respond very differently to the next observation On dependence on dataset and metric, the variance is systematic rather than just being noise. History duration predicts how much information is retained: halving it costs 10.6 pp and doubling it adds 21.0 pp, while the number of frames doesn't matter much. This gives us a rule for when to expect a benefit.
Thank you for the detailed questions; several led to our most informative new experiments.
Q1 When is this useful in practice? This is a diagnostic, not a method. It matters whenever fitted weights are reused as features, as in functa, weight-space learning, or warm-started video/simulation compression (e.g., NIRVANA). There, reconstruction error can rank representations wrongly:
Matched error (validation; MSE in normalized field units):
Longer fitting: on Allen–Cahn, 200 → 1000 steps improves MSE (0.043 → 0.035), yet accuracy falls by 9.4 pp [2.4, 17.0].
Sparse observations: at 6.25% density, warm-started fits have 3× the clean-field error of independent fits, yet retain hidden velocity at 66–69% vs exactly 50%.
Q2 Is adaptation cost informative and does it signal faster training? It is not a measure of training time. Eq. 1 is fit error plus movement from the prior, and Sec. 4.1 uses it as a diagnostic of how a fitted network accommodates a new target:
Networks with identical reconstructions can therefore differ in how they will change.
Q3 Is Table 1 domain shift? No. Table 1 asks whether a model of each prior's local geometry predicts that prior's own adaptation, whereas domain adaptation aims to improve target accuracy. We added this contrast to Sec. 2.
Q4 Autoencoders instead of neural fields (App. E.5). A warm-started convolutional autoencoder shows the same effect:
Retention is therefore not unique to neural fields; neural fields are where the weights are the representation.