- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Observable Mapping (OM), which constructs low-dimensional search coordinates from structured responses to parameter perturbations. OM-EKI performs ensemble Kalman inversion in these coordinates, verifies updates through forward simulations, and periodically rebuilds the mapping. Experiments cover Neural ODE fitting, stiff chemical dynamics, and a hard-reset spiking network, with additional mechanism controls and PDE tests.
Critical strengths
-
Constructing search coordinates from observation-wise responses is interesting. The distinction between response compactness and residual reachability is useful and supported by Table 5.
-
The controlled experiments help explain the roles of coordinate alignment, renewal, and nonlinear verification.
Critical weaknesses
-
The practical experiments are too limited to establish broad utility. The primary tasks use a two-state ODE, a 20-state chemical system, and a small spiking network. Large parameter counts do not necessarily imply difficult optimization problems. The additional Burgers results remain below Adam, and the PDE-control comparison uses only one seed.
-
The performance advantage depends substantially on batched forward evaluations. POLLU requires approximately 100,000 evaluations, and the reported 7.72× gain concerns time to Adam’s endpoint quality, rather than full-budget runtime. DFBGN’s early termination further complicates the comparison.
The idea is promising, and the mechanism controls are informative. However, the practical tasks are insufficiently challenging to establish the method’s broader significance. Together with the substantial forward-evaluation budget and dependence on batching, this leaves me unconvinced of its advantage on demanding applications. A stronger application-level evaluation would be needed.
Questions for authors
-
How do the current experiments establish scalability beyond large, potentially redundant parameterizations?
-
What explains the remaining accuracy gap to Adam on Burgers?
-
Does restarting DFBGN or changing its stopping configuration materially improve its result?
-
With (K=r) in the primary ODE and POLLU experiments, which aspects of OM account for the gains when residual-aware truncation is inactive?
4 comments
Question 1 — Scalability beyond potentially redundant parameters
Response
We agree that a large nominal parameter count alone cannot establish that the parameters are nonredundant or identifiable, or that the method scales favorably with effective dimension. To broaden the parameter structures considered, Allen–Cahn directly estimates a 256-parameter spatial field from 192 observations and achieves competitive results, while the three-seed PDE-control experiment evaluates a neural feedback policy. The Burgers experiments also examine neural models with 28,930 and 113,154 parameters (Appendix D.7). These experiments broaden the representations and objectives studied and motivate further investigation into effective dimension and scalability.
Additional parameter structures (Table 21; Appendix D.7)
| Task | Parameterization | Parameters | Observations or objective |
|---|---|---|---|
| Allen–Cahn | Initial spatial field | 256 | 192 training observations |
| PDE control | Neural feedback policy | 17,153 | Closed-loop trajectory cost |
| Burgers, width 32 | Neural PDE vector field | 28,930 | One-step training and 16-step validation |
| Burgers, width 64 | Neural PDE vector field | 113,154 | One-step training and 16-step validation |
Question 2 — Burgers accuracy gap
To analyze this gap, we now report one-step and 16-step errors together with response diagnostics (Appendix D.7; Tables 18 and 19). Adam has lower 16-step validation and test errors; however, at width 32, OM-EKI with 192 coordinates has lower one-step validation error. The measured response energy is relatively concentrated, whereas the selected coordinates capture only a small fraction of the full one-step training residual. Larger capacity is associated with higher recorded residual capture, but does not consistently improve 16-step validation performance. These results are consistent with the paper's discussion of the local role and limitations of the observable geometry hypothesis, showing that local residual correction and longer-horizon trajectory prediction need to be evaluated separately. This also motivates us to further investigate representations and objectives that account for rollout behavior in future method design.
Burgers (seed 21; Tables 18–19)
| Width | Method | Coordinates | 1-step validation MSE ↓ | 16-step validation MSE ↓ | Residual capture |
|---|---|---|---|---|---|
| 32 | Adam | 28,930 | 0.009890 | 0.13076 | — |
| 32 | OM-EKI | 192 | 0.009277 | 0.15215 | 1.34% |
| 32 | OM-EKI | 256 | 0.010504 | 0.14968 | 2.01% |
| 64 | Adam | 113,154 | 0.009115 | 0.12459 | — |
| 64 | OM-EKI | 192 | 0.009726 | 0.13585 | 1.30% |
| 64 | OM-EKI | 256 | 0.009556 | 0.14063 | 2.04% |
Question 4 — OM with full probe capacity
Response
We thank the reviewer for raising this question. When all directions of the same orthonormal probes are retained, the SVD rotation leaves the parameter-space subspace they span unchanged. We therefore compare Q-128, Q-112, OM-128, and OM-auto-256 on POLLU (Appendix D.3), where Q-EKI directly uses the original probe coordinates. The validation log-MSEs of Q-128 and OM-128 are 2.43379e-5 and 2.41953e-5, respectively, and Q-112 is also competitive (Table 12, lines 1322–1343).
These results show that the probe space itself can contain useful search structure. This makes the role of OM clearer: it uses measured responses to organize directions within the space and to select combinations of directions when capacity is limited.
POLLU probe-coordinate comparison (seed 1; Table 12)
| Method | Probes | Realized dim. | Validation log-MSE ↓ | Test log-MSE ↓ | Evaluations |
|---|---|---|---|---|---|
| OM-auto-256 | 256 | 80–242 | 2.41432e-5 | 1.46306e-5 | 49,598 |
| OM-128 | 128 | 118–128 | 2.41953e-5 | 6.59857e-6 | 100,384 |
| Q-128 | 128 | 128 | 2.43379e-5 | 6.87078e-6 | 100,303 |
| Q-112 | 128 | 112 | 2.43336e-5 | 7.87728e-6 | 100,275 |
Thanks for the rebuttal. Unfortunately, I am not really convinced that 2D evaluation is sufficient to verify the proposed method. So, I will maintain my original score.
Thank you for the suggestion. To extend the evaluation beyond two dimensions, we added a 3D reaction–diffusion inverse problem in Appendix D.9. The task recovers a spatially varying coefficient from state observations, using direct grid parameters and a solver grid. The experimental setup and individual runs are provided in Appendices E.4 and F.1.
OM-EKI achieves validation and reconstruction quality close to that of Adam across three seeds. This result shows that response-derived coordinates can support spatial-coefficient recovery in a three-dimensional dynamical system. In this setting, however, OM-EKI does not show a time-to-quality advantage over gradient-based Adam.
| Method | Validation MSE ↓ | Final-field relative ↓ | Coefficient relative ↓ | Time to quality (s) ↓ |
|---|---|---|---|---|
| Adam | 0.003167 | 0.009398 | 1.088 (3/3) | |
| OM-EKI, cap 256 | 0.003343 | 0.009809 | 5.136 (1/3) | |
| OM-EKI, cap 324 | 0.003188 | 0.009257 | 4.626 (2/3) |
W1 — Application-level evaluation
Response
We thank the reviewer for suggesting that we broaden the empirical evaluation. We add a two-dimensional Allen–Cahn initial-field inversion task with a multiscale spatial field and sparse space-time observations, reporting results over four seeds (Section 5.1.4, lines 416–427; Table 4). The field has 256 parameters and 192 training observations; OM-EKI and Adam achieve similar validation and reconstruction errors. We also extend the closed-loop PDEControlGym experiment to three seeds, using separate validation and test episodes. In every run, OM-EKI has slightly lower test trajectory cost than Adam (Table 20). Together, these results extend response-driven search from trajectory fitting to spatial-field reconstruction and closed-loop policy optimization, showing its potential for different dynamical-system objectives.
Allen–Cahn (four-seed medians; Table 4)
Closed-loop PDE control (three-seed medians; Table 20)
W2, Part 1 — Forward-evaluation cost and batching
Response
We have explicitly updated the reporting criterion: time to quality is the first recorded validation checkpoint at which the validation score is no greater than 1.005 times the paired Adam endpoint, so as to show more clearly how quickly the methods reach the same quality. On POLLU, the median seed-wise paired speedup is 6.75×. The validation trajectories and full-run results show both early progress and subsequent computational cost (Figure 3(b) and Table 2): our method reaches the quality threshold earlier and continues to run through the full budget, accumulating approximately 100,000 objective evaluations. OM-EKI's many forward evaluations can be performed in parallel, whereas Adam's updates involve backpropagation, so the two reflect a tradeoff between forward and backward computation. Appendix D.1 analyzes this tradeoff: for a fixed OM-EKI workload, increasing the batch size from 8 to 64 reduces elapsed time by 6.93×; another comparison reports sampled CPU and GPU memory. In that test, OM-EKI has lower CPU RSS than Adam and similar GPU process memory. The results show the practical value of parallel forward evaluations, while the time benefit depends on the hardware and implementation.
POLLU fixed-workload batching (Table 10; Appendix D.1)
W2, Part 2 / Question 3 — DFBGN stopping and restarts
Response
We thank the reviewer for pointing out that reporting only native stopping cannot fully reflect DFBGN's performance. We now add a timed-restart comparison: after each termination, DFBGN restarts from the point with the best training residual until the paired OM-EKI time limit is reached. The POLLU comparison shows that OM-EKI achieves lower median validation error and reaches paired Adam quality earlier. We also add timed-restart DFBGN comparisons on ODE and SHD (Tables 1 and 3). OM-EKI's independent forward solves can be batched, whereas DFBGN uses sequential residual evaluations. The comparison between the two methods demonstrates OM-EKI's performance advantage in the current POLLU implementation.
POLLU (four-seed medians; Table 2)
Additional DFBGN comparisons (four-seed medians; Tables 1 and 3)