- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AugRelNet, a relation-structured neural dynamical model for forecasting and interaction discovery from limited observations. The framework learns a persistent relation matrix that represents interactions between variables, while augmented states capture nonlinear and history-dependent responses. The model is trained using prediction losses without requiring ground-truth interaction structures or system-specific physical constraints.
The authors evaluate AugRelNet on generalized Lotka–Volterra, Lorenz–96, FitzHugh–Nagumo, Game of Life, and stochastic Ising systems. On 40-dimensional Lorenz–96, the model recovers all 120 true directed interactions across five seeds using approximately 3,849 training states. It also substantially outperforms several forecasting and relation-learning baselines at short horizons. The spatial experiments further demonstrate recovery of local interaction structures across continuous, discrete, and stochastic dynamics.
Critical strengths
-
Clear separation between relational structure and dynamical responses. The proposed architecture explicitly separates persistent interaction pathways from their state-dependent realization. The use of augmented states, particularly the persistent response state, provides a flexible way to represent nonlinear and history-dependent interactions. The relation interventions and capacity-matched ablations help substantiate the importance of these components.
-
Strong forecasting and structure recovery under limited observations. The Lorenz–96 results are compelling. The model achieves exact recovery of all 120 directed interactions across five seeds without structural supervision, while substantially outperforming several baseline models under the same data budget. The additional experiments on Lotka–Volterra and spatial systems provide evidence that the relational representation can extend beyond a single dynamical system.
Critical weaknesses
-
Learned relations do not necessarily correspond to identifiable physical interactions. Although the recovery of Lorenz–96 interactions is convincing, the learned relation matrix represents predictive dependencies rather than uniquely identifiable physical or causal interactions. The nonlinear response functions and augmented states can potentially absorb or redistribute interaction effects. Moreover, the theoretical analysis of relation recovery relies on a frozen linearized model with unregularized gradient flow, rather than the actual nonlinear optimization procedure. I would like to see stronger evidence that the recovered structures remain meaningful under observational noise, partial observations, or different dynamical regimes.
-
The forecasting comparisons need stronger computational controls. The reported forecasting improvements are exceptionally large, particularly on Lorenz–96, where the model outperforms several established methods by more than an order of magnitude. However, the main AugRelNet checkpoints undergo 3,500 initial training epochs followed by approximately 13,000–16,000 additional epochs of optimization (Appendix D.2). While the baselines share the same training data, it is unclear whether they receive comparable optimization budgets, model capacities, or computational resources. A comparison under matched computational budgets would help establish how much of the improvement comes from the proposed architecture.
-
Strong short-term forecasting does not translate into accurate long-term dynamics. Despite substantial improvements in short-horizon RMSE, AugRelNet struggles to reproduce the invariant statistics of Lorenz–96. For example, its long-run mean energy is 12.68 compared with the reference value of 9.36, while DeepSkip remains considerably closer to the reference distribution (Appendix D.2.1, Table 9). The FitzHugh–Nagumo experiments also exhibit substantial degradation beyond the supervised forecast horizon. This raises concerns about whether the model learns the underlying dynamical system or primarily provides accurate finite-horizon approximations.
-
Limited evaluation of robustness and generalization. The experiments primarily involve synthetic dynamical systems with controlled interactions and fully observed states. Lorenz–96 is evaluated at a fixed forcing parameter and dimension, while the other systems also operate under relatively constrained settings. It is unclear whether the reported data efficiency and structure recovery extend to noisy, partially observed, or nonstationary systems. I would like to see experiments involving observation noise, missing variables, or parameter shifts, which are particularly relevant to real-world scientific dynamical forecasting.
-
The spatial experiments rely on relatively strong structural assumptions. While the method does not receive the exact interaction graph, its spatial extension assumes translation symmetry and restricts candidate interactions to predefined local neighborhoods. For example, FitzHugh–Nagumo and Game of Life use a radius-two stencil, while Ising considers only eight neighboring offsets. This substantially reduces the structure discovery problem. Moreover, the forecasting improvements are less consistent across these settings: PolyKAN performs better on Game of Life, while AugRelNet and CNN achieve nearly identical errors on Ising. Testing irregular interaction graphs or spatially heterogeneous systems would better establish the generality of the approach.
Questions for authors
See above.
5 comments
Updated experimental overview. We acknowledge insufficient training in some original runs: GraphODE-Oracle improved with extended optimization. We reran Lorenz-96 with 4,000 training states and five independent datasets. All five ARN runs separate every true edge from non-edges at main epochs 10-20 after conditioner warmup, while forecasts continue improving. ARN substantially outperforms the tested models without supplied graph, locality or matched-library priors, except CLS at H1, and also locality-informed LocalRC. GraphODE-Oracle and Local DeepSkip forecast better, within a factor of a few of ARN.
Five-instance mean physical RMSE; lower is better:
\text{Method / prior}&\text{H1}&\text{H8}&\text{H40}\\\\ \hline \mathbf{ARN}\\ \text{/ none}&\mathbf{0.00729}&\mathbf{0.0489}&\mathbf{0.572}\\\\ \text{NODE / none}&0.348&2.306&3.453\\\\ \text{NRI / none}&0.202&1.164&3.567\\\\ \text{cMLP / none}&0.870&2.419&3.705\\\\ \text{Global DeepSkip / none}&0.106&0.791&3.923\\\\ \text{CLS / none}&0.00164&0.154&\text{NF (2/5)}\\\\ \text{LocalRC / locality}&0.00897&0.146&1.479\\\\ \text{Local DeepSkip / locality}&0.00177&0.0131&0.117\\\\ \text{GraphODE-Oracle / true graph}&0.00380&0.0253&0.183\\\\ \text{SINDy-Poly2 / quadratic library}&0.0000395&0.000300&0.00174 \end{array}$$ NF denotes two nonfinite runs; no finite-subset mean is used. (SINDy-Poly2: second-order differences originally; fourth-order now.) **Q1 (last part): Structure under noise and partial observation.** Gaussian noise corrupts training inputs/targets, scaled by each variable's clean training SD; validation stays clean. AUROC measures how often true edges outrank non-edges; AP summarizes precision as recall increases (1 is ideal for both). ARN uses one fixed recipe for 300 epochs (n=3 per level): AUROC/AP=1 at eight tested levels (3-75%); at 200%/300%, AUROC remains 0.9997/0.9499 and AP 0.9960/0.7881. SINDy-Poly2 is validation-tuned separately per noise level (54 configs plus confirmation): its AP is 1 at 3-50%, exceeds ARN at 100%, and is 0.9617/0.8246/0.4403 at 150/200/300%. With its clean-data settings fixed, SINDy AP falls to 0.2396 at 50% and about 0.083 at 200-300%. We also retrained with 36/40 and 30/40 variables observed (one instance each). AUROC/AP are 0.99998/0.99978 and 0.99795/0.97292, respectively. Both retain strong agreement with the visible-variable direct subgraph, although neither achieves strict separation. For 30/40 observation, H1 RMSE is 0.0155 versus 0.0077 with full inputs on the same 30 targets; H40 rises to 2.978 versus 0.590. Meaningful structural ranking thus survives missing variables while long-horizon prediction deteriorates. Hidden variables can induce effective predictive dependencies, so this is not a direct causal-identification claim. Parameter-shift and nonstationary regimes remain untested; these results support robustness within the stated observation conditions. **Q2: Computational controls and training fairness.** Gradient fits complete their declared optimization/LR stages, then require two consecutive 100-epoch windows with <0.2% improvement in historical best validation score. LR changes reset patience; no hard epoch/time cap applies. Testing uses validation-best checkpoints. cMLP, cLSTM and SyCo-AE follow this protocol; regression methods complete native solves. Validation-only tuning includes 60/70/108 configurations for Local/Global DeepSkip/LocalRC, further DeepSkip ridge refinement, 15 each for SINDy Poly2/Poly3, and 24 for CLS. Selection records are retained per method. Many baselines without supplied structural priors peak very early under our common stopping controls and can overfit with further training. For example, NODE, MNO and NRI reach validation optima at epochs 5, 10 and 15-40. NODE's mean H8 training-probe RMSE falls from 2.065 to 0.067 while validation worsens from 2.317 to 2.415. Comparisons use their earlier validation-best checkpoints. **300-epoch control.** Across three paired instances, ARN restricted to 300 main epochs achieves H8 RMSE 0.1077 versus 2.2801 (NODE), 2.3991 (cMLP), 1.1314 (NRI) and 2.1792 (DySLIM). Their full-run validation-best checkpoints already lie within 300 epochs. These gaps already appear without prolonged refinement. Equal epochs do not equal compute: a separate same-GPU, two-hour-budget comparison (n=3) gives H8 0.1112 for ARN and 0.0343 for true-support Oracle. Width/depth searches span 12.6k trainable parameters (ALRNN) to 10.5M fitted readout coefficients (Global DeepSkip, plus fixed random features); ARN has 1.29M and NRI 1.07M. Parameter counts are not forced equal. Over the next few days, we will present the remaining baselines in the revised paper and supplementary materials, together with a detailed reproducibility report covering code/data identities, seeds, configurations, searches, per-run training/validation curves, checkpoint identities and measured costs.Q3: Long-horizon dynamics. ARN uses the same learned update to autonomously predict unseen test states over 40 steps, recovers the generating dependency pattern across independent datasets without structural supervision, and shows functional dependence on the learned relations in frozen-model interventions. Together, these results support learning reusable dynamics with meaningful interaction structure.
Five frozen 4k-data models give H40/H80/H160 RMSE of 0.581/2.985/21.597, versus Local RC's 1.498/3.331/4.773. RMSE is root mean squared prediction error; lower is better. H80 extends beyond ARN's H40 supervision, although GraphODE-Oracle and Local DeepSkip perform better throughout. Separately, the original 50k-data long-run experiment gives mean energy 12.68 versus the reference 9.36 and DeepSkip's 9.38.
ARN can learn where nonlinear interactions act and closely approximate finite-time evolution without enforcing Lorenz–96's exact energy balance:
E &= \frac{1}{2N}\sum_{i=1}^{N}x_i^2,\\\\ \frac{\mathrm{d}E}{\mathrm{d}t} &= -2E+\frac{F}{N}\sum_{i=1}^{N}x_i. \end{aligned}$$ The nonlinear terms redistribute energy and cancel in the total balance. Correct dependency locations alone do not enforce this quantitative cancellation. Systematic balance errors can shift long-run statistics, which our prediction loss does not explicitly constrain. For FHN, cross-site communication passes through the same offset weights $D$ across message channels. Making structure explicit thus constrains spatial mixing and may trade some predictive flexibility for a readable relation pattern. Correct neighbors do not fix reaction saturation, diffusion damping or the augmented states' behavior under repeated rollout. Errors in these quantities can amplify beyond the H16 training range; the new H32/H64 tests show substantial degradation, with ConvNODE performing better. The structural readout and finite-time accuracy therefore address different requirements from sustained, quantitatively faithful simulation. **Q4: Robustness and generalization.** We added Gaussian noise to training inputs and targets, scaled by each variable's clean training standard deviation, with clean validation. AUROC measures how consistently true interactions rank above noninteractions; AP (average precision) summarizes precision as more true edges are retrieved. Both are higher-is-better, with 1 ideal. With fixed hyperparameters and a 300-epoch budget, ARN achieves AUROC/AP=1 across three independent instances at each of eight noise levels from 3% to 75%. At 200%/300%, AUROC is 0.9997/0.9499 and AP is 0.9960/0.7881. Separately validation-tuned SINDy-Poly2 has AP 0.8246/0.4403 at these levels. We also retrained ARN under partial observation with the original validation-based stopping and checkpoint selection. With 36/40 and 30/40 variables observed, AUROC/AP are 0.99998/0.99978 and 0.99795/0.97292, respectively, with one instance per setting. Neither achieves strict separation. On the same 30 observed targets, H1 RMSE is 0.0155 versus 0.0077 with full inputs, while H40 is 2.978 versus 0.590. Strong agreement with the visible-variable direct subgraph persists despite larger forecasting errors: hidden variables remain dynamically active, adding influences that must be inferred from the observed history. These experiments address observation noise and missing variables; parameter shifts and nonstationarity remain untested.
Identifiability Limits and Structural Selection
Q1, part 1: Identifiability and structural selection
Our revised theory separates what finite observations identify about physical interactions from why learning can reproducibly select benchmark structure. We will revise the original theory accordingly.
Identifiability. For n≥3, every finite, uniformly sampled full-state dataset with unique observed successors and predecessors admits a smooth autonomous realization with any prescribed strongly connected dependency graph, fixed before observing the data. Self-dependence is allowed. The realization uses the original coordinates and no extra states; prescribed edges can have nonzero partial derivatives at every sampled state. No uniform function-complexity or Lipschitz bound is imposed.
Even exact prediction of finite noiseless data cannot certify the true physical graph or, without functional-form and normalization assumptions, GLV strength rankings.
Benchmark success may reflect a favorable alignment: inductive bias favors structures with properties common in benchmark generators, such as reusable interaction rules. When real systems share these properties and observations expose them, exploiting that bias can increase the probability of recovering their true physical structure, without guaranteeing recovery.
Structural selection. We develop a conditional principle for shared interaction representations under squared prediction loss. Under a product reference law, square-integrable channels are centered in each input and true dictionaries are orthonormal throughout learning. For Euclidean relation-gradient flow,
d˙e=Ck(t)be−Gk(t)de+ξe(t).
Here e indexes a candidate interaction channel of arity k, be its normalized generating coefficient, and de its learned coefficient. Gk is the learned-feature Gram operator, Ck the learned/true-feature overlap, and ξe the reference-gradient discrepancy. Relations of the same type share these evolving operators even as nonlinear features learn.
Present relations accumulate a common transformed signal, while absent relations lose initialization effects up to residuals. For nonempty true/non-edge sets and Gk⪰gI along the path, g>0, the score gap satisfies
Γ(t)≥a(t)bmin−2r0e−gt−2δ(t).
Scores group coefficient norms by participating variable; bmin is the smallest true-edge group norm, r0 the largest initial group norm, and a(t) the minimum gain of the shared accumulated transfer. The residual bound is
δ(t)=∫0te−g(t−u)∥ξ(u)∥2du,
with ξ stacking channel discrepancies. A positive lower bound guarantees strict separation before accurate response fitting. The product law is a theoretical reference, not an independence claim for Lorenz trajectories.
With common positive scalar gain, strength rankings are preserved once initialization effects vanish; small residuals preserve rankings when strength gaps dominate. In a fixed-feature shared-rule subcase, sharing can increase the probability of jointly positive structural margins without changing expected prediction risk.
Connection to ARN. The identifiability result makes structural selection dependent on inductive bias; the gradient principle explains how shared representations can favor recurring interaction rules.
Lorenz-96, GLV and reaction-diffusion systems reuse low-order mechanisms despite different graph densities. ARN's D organizes predicted dynamics through persistent named-variable routing. Shared source/receiver functions reuse mechanisms across relations; nonlinear responses and augmented states realize state- and history-dependent effects. Local-history initialization and a zero-aggregate response reference anchor this division.
ARN's contribution is a generative inductive bias: prediction is organized around reusable mechanisms and persistent relation coordinates. When this matches the generator and the data excite its mechanisms, shared learning can coordinate structural selection before detailed responses converge. The gradient principle motivates this relation–response–state design, while experiments test its realization in ARN. Reproducible selection does not certify unique physical coefficients.
Early structural selection. Across five independent runs, ARN fully separates true edges from non-edges at main epochs 10–20 (after conditioner warmup), while forecasting continues to improve. Five further runs without direct D shrinkage also fully separate by epoch 300.
Full theorem statements and proofs are too lengthy for this response and will appear in the appendix of our revised manuscript in a subsequent update. Noise, partial observation and other experimental concerns will be addressed in our second reply.