acceptodds
Under review as a conference paper at ICLR 2027

Neural Fields Encode Adaptation Geometry

Abstract

Neural fields are usually evaluated by how well they reconstruct an observation. We show that this misses two useful properties of a fitted network: how easily it can adapt to new observations, and what its weights retain from earlier ones. We study these properties as adaptation geometry. For images, we meta-learn class-specific initializations, adapt each one to a new image, and measure how much the network must change to fit it. A simple local linear model closely predicts this adaptation cost, while replacing one network's tangent kernel with another's substantially worsens the prediction. Adaptation thus depends on the local geometry of the fitted network, not only on its current reconstruction. For physical fields, we repeatedly fit the same network to observations from a sequence. Its weights then retain information about that history. When two wave histories end at exactly the same observation, the final weights recover the sign of the wave velocity with 68.6% accuracy, whereas the current observation alone contains no such information and gives 50%. These two phenomena are quantitatively linked: tangent-kernel eigenvalues predict both which changes are easy to learn and how quickly they are overwritten by later fitting. Together, these results show that neural fields contain useful information beyond what they currently reconstruct: in how they can change and in how they got there.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
neural fields, implicit neural representations, adaptation geometry, neural tangent kernel, meta-learned initializations, weight-space representations, memory, sequential fitting, physical dynamics, representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oWxmOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper considers the problem of neural field weight initializations and investigates a ‘sequential’ initialization scheme, i.e. initializing a neural field with the weights learned from a previous sample. The authors identify 1. adaptability of the network weights to new observations and 2. training-history-retention of the weights as important properties of a neural field. The paper proposes a method to quantify this adaptation cost (i.e. how much the weights need to change to reconstruct a new image) under the framework of ‘adaptation geometry’ and find that a mismatch between the learned geometry and a new sample increases the adaptation cost. The results also show that after sequential learning, weights contain information about the history of the previous samples, tested in a setting where the final samples were identical while previous samples were different. This history enables better regime classification using the sequentially learned weights. The authors suggest these identified properties are useful beyond simply considering the reconstruction quality as a metric to evaluate neural fields.

Critical strengths
  • The introduction of the adaptation geometry as an interpretation of adaptation cost in sequentially trained neural fields.
  • The introduction of a method to quantify retained-history after sequential training and its links to adaptation geometry.
  • The code being shared in the supplementary materials eases reproducibility concerns.
Critical weaknesses

Main concerns: I don’t understand when sequential fitting is useful or can be used in practice. I also don’t understand what new insights can be gained from the results.

Motivation: I am not sure why one needs to adapt a neural field to new images? Why replace adaptability with reconstruction quality? For example the abstract states: “These results show that neural fields contain useful information beyond what they currently reconstruct: in how they can change and in how they got there.” I am not sure why or in what scenarios this could be useful. For example, why does it matter that the neural field weights need a larger adaptation to fit the next frame, as long as the reconstruction quality doesn’t decay? Depending on the learning rate, task complexity, etc. the adaptation cost may not even translate to training time.

Interpretation: Isn’t the problem considered in Table 1 just domain shift? Is it possible to test, or at least discuss, some domain adaptation methods for this task?

Related work: I feel like related work is missing some discussion on domain adaptation, meta-learning, transfer learning, etc. in the broader computer vision/machine learning domain, outside of the specific context of neural fields. How to optimally initialize neural networks is a well-studied topic and it would be helpful to discuss how the presented analysis using neural fields provides further or different insights.

On a related note, what if we replaced neural fields with (simple or variational) auto-encoders for reconstructing each frame in a video? Would we find that sequential initializations are more useful than independent initializations (as described in Fig. 1(c))? As an example, the latent space of autoencoders are also regularly used for classification tasks. What is the difference of neural fields in this context?

Results: What is the reconstruction quality of independent vs. sequential initialization? For what use-cases does it make sense to replace adaptability with reconstruction quality?

For example, there is no ‘independent’ setting baseline comparison in Table 3, e.g. it is not clear what the reconstruction MSE would be for the ‘independent weights’ in comparison to sequential training. This makes it hard to quantify the reconstruction quality trade-off.

Regarding Fig. 2 and Table 2, I don’t find it surprising that the weights retain information from the history of a wave that they’ve been trained on (e.g. at a level of 68%), as opposed to having chance level predictions (50%) without being trained on the history. What is the insight gained from this result?

Considering the experiment shown in Fig. 2: Can the authors quantify retained memory as a function of change in samples/domain shift? Maybe as a function of wave speed/sampling rate?

Would retained history be more useful (e.g. actually increase the reconstruction quality) under partial observation as is described in sections 1 and 2 (“this is why explicit memory helps learned PDE surrogates under partial observation”)? And if the retained memory about a system’s past is useful under certain settings, e.g. partial observation, low resolution or with noise, why not also test this using reconstruction error with/without the memory/sequential learning? If sequential training improves both reconstruction quality and other metrics (e.g. increased regime classification accuracy, relatively reduced adaptation cost or training time) at the same time under partial observation, this would strengthen the results considerably.

Minor:

Fig. 1(a) was quite unclear to me. The caption mentions K_c, but there is no K_c in the figure. Similarly, the figure includes r_c and x, which are not explained in the caption, therefore I couldn’t figure out the message behind this figure.

Questions for authors
  • Why/when would one need to adapt a neural field to new images? Why replace adaptability with reconstruction quality? In what scenarios could it be useful?
  • When is ‘adaptation cost’ informative? Would adaptation cost in general translate to reconstruction quality or training time?
  • Is the problem considered in Table 1 just domain shift? Is it possible to test, or at least discuss, some domain adaptation methods for this task?
  • What if we replaced neural fields with auto-encoders, sequentially trained to reconstruct each frame in a video? Would we find that sequential initializations are more useful than independent initializations? What is the difference to neural fields in this context?
  • Is it possible to add an ‘independent’ setting reconstruction MSE baseline comparison to Table 3? This can help quantify the adaptability-reconstruction quality trade-off.
  • Training sequentially, as opposed to independently, trivially retains memory. What is the insight gained from Fig. 2 and Table 2?
  • In Fig. 2, would it be possible to quantify retained memory as a function of change in samples/domain shift? Maybe as a function of wave speed/sampling rate?
  • Would retained history be more useful under certain settings, e.g. partial observation, low resolution or with noise? Can the authors test this in terms of both the proposed metrics and reconstruction quality?
4 comments

Thank you for the detailed questions; several led to our most informative new experiments.

Q1 When is this useful in practice? This is a diagnostic, not a method. It matters whenever fitted weights are reused as features, as in functa, weight-space learning, or warm-started video/simulation compression (e.g., NIRVANA). There, reconstruction error can rank representations wrongly:

  • Matched error (validation; MSE in normalized field units):

    MSEAccuracy
    Allen–Cahn, independent, 1000 steps0.04545.1%
    Allen–Cahn, sequential, 200 steps0.04369.4% (+24.3 [16.3, 32.3])
    Gray–Scott, independent, 1000 steps0.55557.2%
    Gray–Scott, sequential, 200 steps0.52467.3% (+10.1 [7.9, 12.9])
  • Longer fitting: on Allen–Cahn, 200 → 1000 steps improves MSE (0.043 → 0.035), yet accuracy falls by 9.4 pp [2.4, 17.0].

  • Sparse observations: at 6.25% density, warm-started fits have 3× the clean-field error of independent fits, yet retain hidden velocity at 66–69% vs exactly 50%.

Q2 Is adaptation cost informative and does it signal faster training? It is not a measure of training time. Eq. 1 is fit error plus movement from the prior, and Sec. 4.1 uses it as a diagnostic of how a fitted network accommodates a new target:

  • A network's own tangent kernel predicts its nonlinear adaptation well (Spearman 0.79–0.96 across 8 settings).
  • Another class's kernel, paired with the same output, predicts it much worse (0.33–0.79), even after equalizing kernel trace.

Networks with identical reconstructions can therefore differ in how they will change.

Q3 Is Table 1 domain shift? No. Table 1 asks whether a model of each prior's local geometry predicts that prior's own adaptation, whereas domain adaptation aims to improve target accuracy. We added this contrast to Sec. 2.

Q4 Autoencoders instead of neural fields (App. E.5). A warm-started convolutional autoencoder shows the same effect:

  • Wave: 70.1% vs exactly 50%; its latent code reaches only 55.7%.
  • Gray–Scott: 66.4% vs 62.0% for frozen latents with explicit history.

Retention is therefore not unique to neural fields; neural fields are where the weights are the representation.

Q5 Independent MSE in Table 3. Added. The table is now Table 4; all ten budgets are in App. F.3. Sequential fits have lower MSE at every budget.

Q6 Isn't the memory trivial? That sequential and independent weights differ is trivial, and explicit history solves the wave task by construction (92–100%). What is not guaranteed is that a single fitted state, trained only to reconstruct, carries linearly decodable hidden state:

  1. It recovers the hidden velocity (R² = 0.22, up to 0.60; exactly 0 from the current observation).
  2. Its reconstruction carries less, by 4–30 pp in every wave condition.
  3. The amount depends systematically on the dynamics (Q7).
  4. Further fitting can overwrite it (see Allen–Cahn above).

Q7 Wave speed and sampling (Table 3). Current-observation controls remain exactly 50%.

Sweep (baseline 68.6%, R² 0.22)×0.5×2
History duration (via Δt)58.0% (R² 0.05)89.5% (R² 0.60)
Wave speed (Δt fixed)66.7% (R² 0.16)79.9% (R² 0.36)
  • Duration dominates. Halving it costs 10.6 pp [8.5, 12.7], while halving the speed costs 1.9 pp [0.0, 3.8].
  • Frame count matters little. At fixed duration and 1,600 total updates, 8, 16 and 32 frames give 70.9%, 68.6% and 67.5%.
  • Adjacent-frame change does not order the results.

Q8 Partial observation and joint improvement (App. E.4). Across observation densities of 100%, 25% and 6.25% and noise of 0, 5% and 10%:

  • Memory is robust: 66.1–68.7%, every interval excluding zero.
  • Joint improvement holds with clean, dense observations: sequential fits reconstruct better and retain more (Allen–Cahn and Gray–Scott at ≥ 5 steps/frame; wave endpoint MSE 2.7e-5 vs 2.2e-4).
  • With sparse or noisy observations the two diverge: warm-started fits recover the clean field worse (6.25% density: 0.168 vs 0.054) while still retaining history.

We hope this clarifies why reconstruction quality alone is not enough, and would be grateful if you would reconsider your score.

I thank the authors for the rebuttal. The responses address my technical questions sufficiently.

Having said that, I think a strong motivation is still missing. While the authors clarified that this is a diagnostic, and not a new method, it is still unclear to me when and under what settings one might want to use this diagnostic. The trade-off with the accuracy or reconstruction quality is not always clear, 'adaptation cost' in general does not translate to practical metrics like reconstruction quality or training time, the results seem to be dataset and evaluation metric dependent, making it hard to estimate how the diagnostics will generalize.

Taken together, I will maintain my current score for now.

Thanks again for the hard work and the effort.

Thank you for engaging with our rebuttal.

As for who would use this, we believe it would be useful for anyone who fits neural fields to a sequence (video or simulation snapshots) and then reuses the fitted weights as features, as in functa-style classification or weight-space learning. Our results tell them:

  • Not to tune the fitting budget by reconstruction error and to choose the budget with a downstream validation probe instead.
  • With sparse or noisy observations, a trade-off appears between warm-starting and independent fitting. At 6.25% density, warm-started fits recover the clean field worse but still retain history. The choice then depends on which of the two is needed.

We agree that adaptation cost is not a proxy for other useful metrics. Its role is to show that two networks with the same reconstruction can respond very differently to the next observation On dependence on dataset and metric, the variance is systematic rather than just being noise. History duration predicts how much information is retained: halving it costs 10.6 pp and doubling it adds 21.0 pp, while the number of frames doesn't matter much. This gives us a rule for when to expect a benefit.

Reviewer_9719OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates properties of neural field representations beyond reconstruction quality, focusing on local adaptation geometry and historical information retained through sequential fitting. The authors show that tangent kernels predict non-linear adaptation and that sequentially fitted weights retain information useful for physical-regime classification. They further connect adaptation and forgetting through local tangent-kernel dynamics.

Critical strengths
  1. The paper provides an interesting perspective on neural field weights by studying adaptation behavior and optimization history. In particular, analyzing historical information retained in the weights is meaningful for the considered physical dynamics tasks, where the current observation may not fully capture the underlying system state.

  2. The experiments are carefully designed, including kernel reassignment and identical endpoint wave experiments. The authors also provide extensive ablations and controls, transparently reporting weak results, which strengthens the crediblity of the empirical analysis.

Critical weaknesses
  1. While the retention rule works for a frozen Jacobian under linear gradient descent, practical neural field uses Adam optimization over non-linear spaces. Moreover, selective erasure of slow tangent modes does not consistently degrade classification performance more than fast modes. Thus, although the local retention dynamics are well validated under controlled updates, their connection to the mechanism underlying useful long-term memory in practical nonlinear fitting remains unclear.

  2. The empirical findings vary across tasks and configurations. Moreover, downstream performance is sensitive to the number of fitting steps. While the paper does not aim to propose a new method, these variations leave it unclear under what conditions sequential fitting provides an advantage and how broadly the findings generalize.

Questions for authors
  1. Given that selective erasure did not identify slow tangent modes as the causal carriers of useful memory, do the authors have a hypothesis about where this information is stored along nonlinear weight trajectories?

  2. Have the authors considered evaluating the proposed analysis in other neural field applications, such as video representation, to assess whether the observed properties generalize beyond the physical systems studied?

2 comments

Thank you for the careful review.

W1: Frozen-Jacobian rule vs nonlinear Adam (Sec. 4.4, App. F.6). We agree and tested it directly:

  • Gray–Scott: under the actual Adam updates, the frozen rule still ranks what stays in each tangent mode (Spearman 0.93 → 0.75 over 1–8 frames).
  • Allen–Cahn: perturbations leave the linear regime, and the eigenvalue ordering reverses.

The paper now states the retention rule only for the linearized update.

Q1: Where is useful memory stored?

  • In the weights more than in the reconstruction or a code. In all 15 wave conditions, the weights beat their own decoded field by 4–30 pp. A warm-started autoencoder's weights reach 70.1%, while its latent code reaches 55.7%.
  • Not yet localized. Our hypothesis (App. F.5) is that history lies in directions selected by the composition of fitting updates. Derivatives through the multi-frame Adam fits are now numerically valid (float64, Gray–Scott), but the erasure tests built on them were insensitive (exploratory at higher ranks), so the hypothesis remains untested.

W2: Dependence on task and fitting steps (Table 4, App. F.3). Across 1–1000 steps/frame, sequential beats independent in 18/20 system–budget settings and reconstructs better in all 20.

  • Allen–Cahn: budget sensitivity is substantial (69.4% at 200 steps, 60.1% at 1000).
  • Gray–Scott: mild (64.1–69.2%).
  • Wave: none (67.7–68.6% from 25 to 500 steps).

Shear flow does not replicate across splits and moves to App. D.5.

This helps us when:

  1. The current frame underdetermines the target: hidden velocity, an unknown reaction rate, or reduced fields.
  2. The observed history is long enough. Halving the wave history's duration costs 10.6 pp and doubling it adds 21.0 pp, while frame count matters little (Table 3).

The effect is robust to sparse and noisy observations (66–69%). It is not a general recipe: raw delays match it on Allen–Cahn, and supervised temporal models are stronger.

Q2: Video. The effect also appears in a convolutional autoencoder, so it is not tied to coordinate-based parameterizations. Video remains future work (Sec. 6).

We hope the narrower, better-supported claims address your concern about generality, and would be grateful if you would reconsider your score.

I appreciate the authors' efforts to address my concerns and revise their claims accordingly.

However, some of my concerns regarding the underlying mechanism and generalizability remain. While the revised claims are better supported, I still think the scope of the main findings rather limited. Therefore, I will maintain my original rating.

Reviewer_oJ8hOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates two properties of neural field representations beyond reconstruction accuracy: adaptation geometry and memory retained through sequential fitting. First, it shows that the tangent kernel of a fitted neural field can predict its nonlinear adaptation cost, with kernel-reassignment experiments demonstrating the importance of prior-specific local geometry. Second, it shows that sequentially fitted neural fields retain information about previous observations, even when different trajectories end at identical observed fields. Experiments on image datasets and physical systems, including wave propagation, Allen–Cahn, Gray–Scott, and shear flow, support these findings. A local tangent-kernel analysis further connects adaptation sensitivity with memory-retention timescales.

Critical strengths
  1. Interesting scientific insights supported by targeted experiments. Rather than proposing another neural-field architecture, the paper investigates fundamental properties of learned representations. The kernel-reassignment experiments provide convincing evidence that adaptation depends on prior-specific geometry, while the matched-endpoint wave experiments offer a particularly informative demonstration of historical information retained in network weights.

  2. Strong experimental design and mathematical analysis. The paper combines analytical derivations, controlled interventions, and evaluations across multiple architectures and physical systems. The authors provide detailed statistical analyses and explicitly report negative results and limitations, which strengthen the credibility of the findings.

Critical weaknesses
  1. Limited demonstration of practical advantages over alternative temporal representations. Sequential neural-field weights outperform several raw-field and independent-fitting baselines, but the experiments do not establish their advantages over more general temporal representation methods, such as recurrent encoders or learned history aggregation. The observed memory may partly reflect the implicit history retained by warm-started optimization rather than a distinctive property of neural fields. Furthermore, some downstream improvements are relatively modest, and the shear-flow evaluation uses only one data split, limiting confidence in the generality of the findings.

  2. Incomplete connection between adaptation geometry and practical memory. Although the tangent-kernel spectrum provides a mathematical explanation of adaptation and forgetting under a frozen linearized model, its role in practical sequential fitting with nonlinear Adam updates remains unclear. The matched-versus-mismatched Jacobian intervention helps on Gray–Scott but fails on shear flow, while selective erasure does not establish that the predicted slow modes carry useful classification information. Consequently, the proposed unified explanation of adaptation and memory is not fully supported by the practical experiments.

Questions for authors

NA

2 comments

Thank you for the positive assessment and precise criticisms. We ran an experiment for each weakness (details in the general comment).

Recurrent encoders (App. D.6).

  • Allen–Cahn: supervised CNN–GRU/TCN models are clearly stronger (99.3% vs 64.4%).
  • Gray–Scott: most methods, including a GRU on independently fitted weights, reach the same ~65% plateau.

We narrow the claim accordingly: sequential fitting makes history readable by a linear readout of the weights; it is not a competitive temporal classifier.

Warm-starting rather than a neural-field property (App. E.5). You were right. A warm-started convolutional autoencoder shows the same effect:

  • Wave: 70.1% vs exactly 50%.
  • Gray–Scott: 66.4% vs 62.0% for frozen latents with explicit history.

The paper now attributes retention to warm-started reconstruction fitting, studied in neural fields because there the weights are the representation.

Modest gains, one shear split (App. D.5). Shear flow does not replicate across four new splits (+1.4 pp [−1.4, 4.9]) and is removed from the main-text claims.

The remaining evidence holds under IC-clustered intervals:

  • Wave: +18.6 pp over the exact 50% bound [16.5, 20.6], positive in all 15 conditions (+8.0 to +39.5 pp).
  • Allen–Cahn: +22.4 pp over raw fields [15.7, 28.6].
  • Gray–Scott: +5.7 pp over validation-selected raw delays [2.0, 9.6].
  • Matched reconstruction error: +24.3 pp (Allen–Cahn) and +10.1 pp (Gray–Scott).

Adaptation geometry vs practical memory (Sec. 4.4, App. F.6). We tested this directly, and the link is partial under Adam:

  • Gray–Scott: the frozen retention rule still ranks what stays in each tangent mode (Spearman 0.93 → 0.75 over 1–8 frames).
  • Allen–Cahn: perturbations leave the linear regime.

The paper now claims the link only for the linearized update.

Our memory result is now stronger than in the initial submission: the memory result now holds across 15 wave conditions, in a second architecture, and at matched reconstruction error. We hope this addresses the concerns behind your score.

Thank you for the detailed response and additional experiments. The new experiments strengthen the empirical evidence. I also appreciate the authors' clarification of the claims and limitations.

My main concern remains the connection between tangent geometry and practical memory retention, which is still only partially supported. The autoencoder results also suggest that memory effect is not specific to neural fields.

Overall, the rebuttal addresses some of my concerns. I will maintain my rate.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment to all reviewers (1/2)

Summary of new experiments and revised claims

We thank the reviewers for careful, constructive reviews. With the new experiments, the paper's claims are:

  1. Prior-specific tangent geometry predicts nonlinear adaptation (Sec. 4.1).
  2. Warm-started reconstruction fitting stores decodable hidden-state information in a single fitted state, which reconstruction error does not reveal. This now holds across 15 wave conditions, in two architectures, and at matched reconstruction error.
  3. The link between tangent geometry and retention is exact for the linearized update and partial under practical Adam.

We report every condition we ran, including some results that narrow these claims. We've updated the PDF and code. Locations refer to the revised PDF.

1. What a single fitted state retains (Table 3, App. E.4). Two wave histories end at identical observations, so any function of the current observation scores exactly 50%. Explicit history solves the task by construction (92–100%); the question is how much one fitted state retains without being trained to.

Sweep (16 frames unless noted)SettingWeightsDecoded fieldvs baseline (pp)
BaselineΔt 0.05, speed ×168.6%55.6%—
History duration (via Δt)×0.5 / ×258.0 / 89.5%53.9 / 59.9%−10.6 [−12.7, −8.5] / +21.0 [19.0, 23.0]
Wave speed×0.5 / ×266.7 / 79.9%55.9 / 53.8%−1.9 [−3.8, 0.0] / +11.3 [9.3, 13.4]
Frames at fixed duration8 / 3270.9 / 67.5%54.9 / 53.5%+2.3 [0.4, 4.2] / −1.1 [−3.0, 0.8]
Density × noise9 settings, to 6.25% and 10%66.1–68.7%50.1–52.9%−2.5 to +0.1
  • Every condition exceeds 50%: (+8.0 to +39.5 pp, all intervals excluding zero).
  • Weights beat their own reconstruction in every condition, by 4 to 30 pp.
  • Duration dominates: at equal travel distance, the longer history wins by 9.6 [7.9, 11.3] and 8.7 [6.7, 10.6] pp. Frame count matters little.

2. Reconstruction does not explain it (Table 4, App. F.3). Independent-fit MSE is now reported at all ten budgets (1–1000 steps/frame; validation).

  • Reconstruction: sequential fits reconstruct better at every budget.
  • Classification: they classify better than independent fits in 18/20 system–budget settings.
  • Matched reconstruction error:
ValidationMSEAccuracy
Allen–Cahn, independent, 1000 steps0.04545.1%
Allen–Cahn, sequential, 200 steps0.04369.4% (+24.3 [16.3, 32.3])
Gray–Scott, independent, 1000 steps0.55557.2%
Gray–Scott, sequential, 200 steps0.52467.3% (+10.1 [7.9, 12.9])
AuthorsOpenReview ↗

General comment to all reviewers (2/2)

3. The effect is not specific to coordinate networks (App. E.5). We tested a convolutional autoencoder with no coordinate or temporal input, adapted per frame either from a shared initialization or warm-started. It has 9,569 parameters on wave and 2,626 on Gray–Scott (INRs had 2,241 and 354).

  • Wave: sequential AE weights reach 70.1%, vs exactly 50% for independent adaptation (+20.1 [18.1, 22.0]). The AE's latent code reaches 55.7%.
  • Gray–Scott: sequential AE weights reach 66.4%, vs 62.0% for frozen latents with explicit history (+4.4 [0.7, 8.1]).

Retention is therefore a property of warm-started reconstruction fitting. Neural fields are useful to study this phenomenon as the weights are the representation and the tangent kernel makes adaptation analyzable.

4. Learned temporal baselines (App. D.6). Each was validation-selected from 24 configurations × 3 seeds. Test accuracy (%):

Allen–CahnGray–ScottShear (5 splits)
Sequential weights + our readout (Eq. 4)64.464.928.3
Raw delays + our readout65.459.326.6
GRU on raw PCA / independent weights / sequential weights26.1 / 25.4 / 25.7†64.7 / 65.6 / 64.130.7 / 26.1 / 26.4†
CNN–GRU99.365.232.5
CNN–TCN97.267.543.9

† These heads overfit (Allen–Cahn: 67–80% train vs 31–34% validation accuracy, with 12 training sequences per class). An audit found no implementation error.

Supervised sequence models are the stronger classifiers. On Gray–Scott, most methods reach a common plateau near 65%, including a GRU on independent weights. Our claim is therefore narrower: sequential fitting puts history into a form that a linear readout of the weights can use.

5. Shear flow does not replicate (App. D.5). On four new IC-disjoint splits, sequential minus independent is +1.4 pp [−1.4, 4.9]. Shear flow is removed from the main-text claims.

6. Sparse and noisy observations (App. E.4). Retained history is robust: sequential weights reach 66.1–68.7% in all nine density × noise conditions, every interval excluding zero. Clean-field reconstruction is not robust: with degraded observations, warm-started fits track the degraded targets and recover the clean field worse than independent fits. At 6.25% density, endpoint clean MSE is 0.168 vs 0.054.

7. Tangent spectrum vs practical Adam (Sec. 4.4, App. F.5–F.6). We perturbed endpoint weights along tangent modes (1e-6 to 1e-4 of the weight norm) and continued fitting with the actual Adam updates.

  • Gray–Scott: the frozen retention formula still ranks what stays in each mode (Spearman 0.93–0.94 one frame later, 0.72–0.75 after eight).
  • Allen–Cahn: perturbations leave the linear regime, and the eigenvalue ordering reverses.
  • Erasure: derivatives through the multi-frame Adam fits are now numerically valid on Gray–Scott (float64). But erasure tests were insensitive at the pre-specified rank 8 and, exploratorily, at ranks 32 and 64: no intervention, random ones included, moved accuracy by more than 0.4 pp. We have therefore not localized the memory carrier.

Changes in the revised PDF

  • Abstract and Sec. 1: the geometry–retention link is now stated as exact in the linearization and partial under Adam, plus one sentence each on items 1–3.
  • Sec. 5–6: four sentences updated to match.
  • Figure 1: shear flow removed from panel (d); the caption now defines every symbol in panel (a).
  • Sec. 2: one sentence on transfer learning and domain adaptation.
  • Sec. 4.2: Table 3 and one paragraph (item 1).
  • Sec. 4.3:
    • shear flow moved to App. D.5;
    • two short paragraphs added (items 2 and 4).
  • Sec. 4.4:
    • "locally" added to the title;
    • Table 4 (formerly Table 3) adds independent fits;
    • the temporal-filter interpretation is marked as a hypothesis;
    • the final paragraph is replaced by item 7.
  • Appendices: new D.5, D.6, E.4, E.5 and F.6; F.3 and F.5 extended.
  • Supplementary code: updated with all new experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.