acceptodds
Under review as a conference paper at ICLR 2027

Observable Geometry for Derivative-Free Optimization of Dynamical Systems

Abstract

Derivative-free optimization of dynamical systems avoids differentiating through a simulator, yet it often reduces structured outputs to scalar losses and discards their observation-wise responses. These responses retain optimization-relevant structure that scalarization hides. We exploit this structure through observable mapping (OM), which constructs local response-informed coordinates, and instantiate it in Observable-Mapping Ensemble Kalman Inversion (OM-EKI) as a derivative-free optimizer for structured-response dynamical-system problems. We call the resulting task-relevant response structure observable geometry and hypothesize that useful coordinates exhibit three properties: compactness, residual reachability, and local persistence. Across settings with available, costly, and unavailable exact gradients, OM-EKI achieves competitive held-out accuracy and runtime on a deterministic Neural ODE, reduces median time-to-quality by relative to Adam on a 19,587-parameter stiff UDE, and approaches surrogate-gradient validation performance on a 50,176-parameter hard-reset recurrence for Spiking Heidelberg Digits (SHD). Mechanism-focused controls further support this hypothesis. Together, these results show that structured forward responses can be used not only to evaluate derivative-free search in dynamical systems, but also to organize it.

Primary area:
optimization
Keywords:
Derivative-Free Optimization, Ensemble Kalman Inversion, Dynamical Systems, Scientific Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_64KROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Observable Mapping (OM), which constructs low-dimensional search coordinates from structured responses to parameter perturbations. OM-EKI performs ensemble Kalman inversion in these coordinates, verifies updates through forward simulations, and periodically rebuilds the mapping. Experiments cover Neural ODE fitting, stiff chemical dynamics, and a hard-reset spiking network, with additional mechanism controls and PDE tests.

Critical strengths
  1. Constructing search coordinates from observation-wise responses is interesting. The distinction between response compactness and residual reachability is useful and supported by Table 5.

  2. The controlled experiments help explain the roles of coordinate alignment, renewal, and nonlinear verification.

Critical weaknesses
  1. The practical experiments are too limited to establish broad utility. The primary tasks use a two-state ODE, a 20-state chemical system, and a small spiking network. Large parameter counts do not necessarily imply difficult optimization problems. The additional Burgers results remain below Adam, and the PDE-control comparison uses only one seed.

  2. The performance advantage depends substantially on batched forward evaluations. POLLU requires approximately 100,000 evaluations, and the reported 7.72× gain concerns time to Adam’s endpoint quality, rather than full-budget runtime. DFBGN’s early termination further complicates the comparison.

The idea is promising, and the mechanism controls are informative. However, the practical tasks are insufficiently challenging to establish the method’s broader significance. Together with the substantial forward-evaluation budget and dependence on batching, this leaves me unconvinced of its advantage on demanding applications. A stronger application-level evaluation would be needed.

Questions for authors
  1. How do the current experiments establish scalability beyond large, potentially redundant parameterizations?

  2. What explains the remaining accuracy gap to Adam on Burgers?

  3. Does restarting DFBGN or changing its stopping configuration materially improve its result?

  4. With (K=r) in the primary ODE and POLLU experiments, which aspects of OM account for the gains when residual-aware truncation is inactive?

4 comments

W1 — Application-level evaluation

Response

We thank the reviewer for suggesting that we broaden the empirical evaluation. We add a two-dimensional Allen–Cahn initial-field inversion task with a multiscale spatial field and sparse space-time observations, reporting results over four seeds (Section 5.1.4, lines 416–427; Table 4). The field has 256 parameters and 192 training observations; OM-EKI and Adam achieve similar validation and reconstruction errors. We also extend the closed-loop PDEControlGym experiment to three seeds, using separate validation and test episodes. In every run, OM-EKI has slightly lower test trajectory cost than Adam (Table 20). Together, these results extend response-driven search from trajectory fitting to spatial-field reconstruction and closed-loop policy optimization, showing its potential for different dynamical-system objectives.

Allen–Cahn (four-seed medians; Table 4)

MethodValidation MSE ↓Initial-field RMSE ↓Final-field RMSE ↓
Adam0.038020.343210.11367
OM-GN0.067310.424740.15809
DFBGN0.190330.968080.21363
OM-EKI0.037250.328550.11635

Closed-loop PDE control (three-seed medians; Table 20)

MethodValidation cost ↓Test cost ↓Time to quality (s) ↓Full time (s)
Adam0.4623940.626097353.67507.1
OM-GN0.4590700.629771275.34881.7
OM-EKI0.4570580.62095429.93563.5

W2, Part 1 — Forward-evaluation cost and batching

Response

We have explicitly updated the reporting criterion: time to quality is the first recorded validation checkpoint at which the validation score is no greater than 1.005 times the paired Adam endpoint, so as to show more clearly how quickly the methods reach the same quality. On POLLU, the median seed-wise paired speedup is 6.75×. The validation trajectories and full-run results show both early progress and subsequent computational cost (Figure 3(b) and Table 2): our method reaches the quality threshold earlier and continues to run through the full budget, accumulating approximately 100,000 objective evaluations. OM-EKI's many forward evaluations can be performed in parallel, whereas Adam's updates involve backpropagation, so the two reflect a tradeoff between forward and backward computation. Appendix D.1 analyzes this tradeoff: for a fixed OM-EKI workload, increasing the batch size from 8 to 64 reduces elapsed time by 6.93×; another comparison reports sampled CPU and GPU memory. In that test, OM-EKI has lower CPU RSS than Adam and similar GPU process memory. The results show the practical value of parallel forward evaluations, while the time benefit depends on the hardware and implementation.

POLLU fixed-workload batching (Table 10; Appendix D.1)

Batch sizeTime (s) ↓Validation log-MSE ↓Peak CUDA allocated (MiB)
8724.240.00179972.57
16334.490.00181972.65
32167.310.00181972.65
64104.510.00181472.65

W2, Part 2 / Question 3 — DFBGN stopping and restarts

Response

We thank the reviewer for pointing out that reporting only native stopping cannot fully reflect DFBGN's performance. We now add a timed-restart comparison: after each termination, DFBGN restarts from the point with the best training residual until the paired OM-EKI time limit is reached. The POLLU comparison shows that OM-EKI achieves lower median validation error and reaches paired Adam quality earlier. We also add timed-restart DFBGN comparisons on ODE and SHD (Tables 1 and 3). OM-EKI's independent forward solves can be batched, whereas DFBGN uses sequential residual evaluations. The comparison between the two methods demonstrates OM-EKI's performance advantage in the current POLLU implementation.

POLLU (four-seed medians; Table 2)

MethodValidation log-MSE ↓Endpoint time (s)Time to quality (s) ↓
Adam2.7382e-59,119.297,973.48
DFBGN (restart)2.50096e-58,907.281,786.01
OM-GN2.48481e-51,501.551,071.80
OM-EKI2.4054e-59,115.151,182.56

Additional DFBGN comparisons (four-seed medians; Tables 1 and 3)

TaskMethodValidation metric ↓Secondary metricTime (s)
ODEDFBGN (restart)MAE: 6.37668e-5Test MAE: 7.24218e-5192.20
ODEOM-EKIMAE: 0.72985e-5Test MAE: 0.98288e-5154.71
SHDDFBGN (restart)Brier: 0.046197Accuracy: 17.17%2,207.83
SHDOM-EKI (cap 256)Brier: 0.042218Accuracy: 29.83%1,401.56

Question 1 — Scalability beyond potentially redundant parameters

Response

We agree that a large nominal parameter count alone cannot establish that the parameters are nonredundant or identifiable, or that the method scales favorably with effective dimension. To broaden the parameter structures considered, Allen–Cahn directly estimates a 256-parameter spatial field from 192 observations and achieves competitive results, while the three-seed PDE-control experiment evaluates a neural feedback policy. The Burgers experiments also examine neural models with 28,930 and 113,154 parameters (Appendix D.7). These experiments broaden the representations and objectives studied and motivate further investigation into effective dimension and scalability.

Additional parameter structures (Table 21; Appendix D.7)

TaskParameterizationParametersObservations or objective
Allen–CahnInitial spatial field256192 training observations
PDE controlNeural feedback policy17,153Closed-loop trajectory cost
Burgers, width 32Neural PDE vector field28,930One-step training and 16-step validation
Burgers, width 64Neural PDE vector field113,154One-step training and 16-step validation

Question 2 — Burgers accuracy gap

To analyze this gap, we now report one-step and 16-step errors together with response diagnostics (Appendix D.7; Tables 18 and 19). Adam has lower 16-step validation and test errors; however, at width 32, OM-EKI with 192 coordinates has lower one-step validation error. The measured response energy is relatively concentrated, whereas the selected coordinates capture only a small fraction of the full one-step training residual. Larger capacity is associated with higher recorded residual capture, but does not consistently improve 16-step validation performance. These results are consistent with the paper's discussion of the local role and limitations of the observable geometry hypothesis, showing that local residual correction and longer-horizon trajectory prediction need to be evaluated separately. This also motivates us to further investigate representations and objectives that account for rollout behavior in future method design.

Burgers (seed 21; Tables 18–19)

WidthMethodCoordinates1-step validation MSE ↓16-step validation MSE ↓Residual capture
32Adam28,9300.0098900.13076—
32OM-EKI1920.0092770.152151.34%
32OM-EKI2560.0105040.149682.01%
64Adam113,1540.0091150.12459—
64OM-EKI1920.0097260.135851.30%
64OM-EKI2560.0095560.140632.04%

Question 4 — OM with full probe capacity

Response

We thank the reviewer for raising this question. When all directions of the same orthonormal probes are retained, the SVD rotation leaves the parameter-space subspace they span unchanged. We therefore compare Q-128, Q-112, OM-128, and OM-auto-256 on POLLU (Appendix D.3), where Q-EKI directly uses the original probe coordinates. The validation log-MSEs of Q-128 and OM-128 are 2.43379e-5 and 2.41953e-5, respectively, and Q-112 is also competitive (Table 12, lines 1322–1343).

These results show that the probe space itself can contain useful search structure. This makes the role of OM clearer: it uses measured responses to organize directions within the space and to select combinations of directions when capacity is limited.

POLLU probe-coordinate comparison (seed 1; Table 12)

MethodProbesRealized dim.Validation log-MSE ↓Test log-MSE ↓Evaluations
OM-auto-25625680–2422.41432e-51.46306e-549,598
OM-128128118–1282.41953e-56.59857e-6100,384
Q-1281281282.43379e-56.87078e-6100,303
Q-1121281122.43336e-57.87728e-6100,275

Thanks for the rebuttal. Unfortunately, I am not really convinced that 2D evaluation is sufficient to verify the proposed method. So, I will maintain my original score.

Thank you for the suggestion. To extend the evaluation beyond two dimensions, we added a 3D reaction–diffusion inverse problem in Appendix D.9. The task recovers a spatially varying coefficient from state observations, using 83=5128^3=512 direct grid parameters and a 24324^3 solver grid. The experimental setup and individual runs are provided in Appendices E.4 and F.1.

OM-EKI achieves validation and reconstruction quality close to that of Adam across three seeds. This result shows that response-derived coordinates can support spatial-coefficient recovery in a three-dimensional dynamical system. In this setting, however, OM-EKI does not show a time-to-quality advantage over gradient-based Adam.

MethodValidation MSE ↓Final-field relative L2L^2 ↓Coefficient relative L2L^2 ↓Time to quality (s) ↓
Adam3.063×10−63.063 \times 10^{-6}0.0031670.0093981.088 (3/3)
OM-EKI, cap 2563.275×10−63.275 \times 10^{-6}0.0033430.0098095.136 (1/3)
OM-EKI, cap 3243.033×10−63.033 \times 10^{-6}0.0031880.0092574.626 (2/3)
Reviewer_iSwEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OM-EKI for derivative-free optimization of dynamical systems. It uses structured forward responses to construct search coordinates, followed by ensemble Kalman updates, verification, and coordinate renewal. Three main experiments and several controlled studies evaluate the method.

Critical strengths
  1. The problem is meaningful when gradients are expensive or unavailable. The POLLU results show promising improvement in time to reach a given accuracy.

  2. The paper includes useful experiments on residual reachability and coordinate renewal. Reporting individual results for four seeds also helps assess the consistency of the main results.

Critical weaknesses
  1. The technical innovation needs to be clearer. Finite differences, SVD, EKI, and the verification rule are established techniques. What new capability does their combination provide? In Table 16, the deterministic ODE and POLLU retain all probe coordinates, so the SVD rotation does not change the search subspace. The current comparisons do not clearly separate its benefit from scaling, probe reuse, and coordinate renewal.

  2. The theoretical and experimental support needs strengthening. The theory explains local approximation properties but does not clearly establish progress under the actual regularization and step constraints. For POLLU, DFBGN stops after 221–278 evaluations, while OM-EKI uses about 100,000. Although the paper explains this early termination, a stronger comparison is needed to support the claimed advantage over residual-based optimization.

Questions for authors
  1. When all probe coordinates are retained, what does the SVD rotation add? Can you compare against EKI using the same original probe basis, with the same scale, renewal, and verification?

  2. Why use EKI rather than a regularized reduced Gauss–Newton update based on the same measured response operator? Can you provide a direct comparison?

2 comments

W1, Part 1 — Clarifying the technical contribution

Response

We thank the reviewer for the suggestion. We have reorganized the Introduction, contributions, and method description to highlight the paper's central contribution: using structured forward responses to organize the search representation for derivative-free optimization. Observable mapping (OM) puts this idea into a concrete method: it measures the effects of parameter perturbations on individual observations, constructs response-aligned coordinates, selects capacity in conjunction with the current residual, and updates the representation during optimization (Section 4; Figure 2).

This construction allows the observable behavior of a dynamical system to guide search directions and their adjustment. We develop OM-EKI in these coordinates and introduce OM-GN, showing that the same response-driven approach can support ensemble residual correction and regularized Gauss–Newton updates. The control experiments in Section 5.2 further connect response alignment, residual-aware capacity, coordinate renewal, and verification with optimization performance. Together, these results demonstrate the value of structured responses as an optimization resource and how OM puts this idea into practice.


W1, Part 2 / Question 1 — Full probe capacity and Q-EKI

Response

We thank the reviewer for pointing out this distinction. When the full probe space is retained, the SVD rotation does not change its span, and we do not attribute a subspace-selection benefit to the rotation itself. The Q-EKI comparison confirms that the original probe coordinates can already support effective optimization. The additional role of OM lies in response-informed capacity selection and coordinate adaptation, particularly when the retained dimension is restricted.

MethodProbesRealized dimensionValidation log-MSE ↓Test log-MSE ↓Evaluations
OM-auto-25625680–2422.41432e-51.46306e-549,598
OM-128128118–1282.41953e-56.59857e-6100,384
Q-1281281282.43379e-56.87078e-6100,303
Q-1121281122.43336e-57.87728e-6100,275

The capacity, drift, and verification controls in Section 5.2 examine these mechanisms from different perspectives (Figure 4; Tables 6–7). These results also motivate us to further investigate how to construct and adapt informative directions beyond the current probe space.


W2, Part 1 — Conditional progress with regularization and step constraints

Response

We thank the reviewer for the suggestion. We have added a conditional descent analysis for OM-GN and OM-EKI in Appendix C.5 (lines 1189–1247). For a fixed map, let HH denote the whitened measured response and ee denote the residual, with local objective m(d)=12∥e−Hd∥2m(d)=\frac12\|e-Hd\|^2. For the regularized GN direction u=(H⊤H+λI)−1H⊤eu=(H^\top H+\lambda I)^{-1}H^\top e and clipped step d=τud=\tau u, when λ>0\lambda>0 and 0<τ≤10<\tau\leq1, we obtain the local decrease lower bound shown below:

m(0)−m(d)≥τ∥H⊤e∥22(∥H∥2+λ).m(0)-m(d)\geq\frac{\tau\|H^\top e\|^2}{2(\|H\|^2+\lambda)}.

For EKI, we establish a group-mean local decrease bound weighted by the ensemble covariance. These results explain how measured response sensitivity, regularization, clipping, and ensemble covariance determine the available decrease; residual reachability can further strengthen this bound when the corresponding response directions are sufficiently strong.

We then connect the local model to the simulator through verification: when prediction and the acceptance test use the same objective and decision statistic, if the predicted decrease is positive and the acceptance ratio satisfies ρ>η>0\rho>\eta>0, then the actual decrease satisfies Δact>ηΔpred>0\Delta_{\mathrm{act}}>\eta\Delta_{\mathrm{pred}}>0. Under the uniform boundedness conditions stated in the paper, summing the decreases of accepted steps also gives finite-sum bounds on the GN response-gradient signal and the covariance-weighted EKI signal. Thus, the theory connects measured responses with regularized finite-step progress for both update mechanisms.


W2, Part 2 — Stronger POLLU comparison with DFBGN

Response

We strengthen the POLLU comparison by reporting both native stopping and timed-restart DFBGN. After each termination, DFBGN restarts from the point with the best training residual until the paired OM-EKI time limit is reached (Appendix E.7). The results show that OM-EKI achieves lower median validation error and a shorter median time to quality; OM-GN reaches the quality threshold earliest among the reported methods. We also add DFBGN comparisons on ODE, SHD, and Allen–Cahn (Tables 1, 3, and 4).

MethodValidation log-MSE ↓Endpoint time (s)Time to quality (s) ↓
Adam2.7382e-59,119.297,973.48
DFBGN (restart)2.50096e-58,907.281,786.01
OM-GN2.48481e-51,501.551,071.80
OM-EKI2.4054e-59,115.151,182.56

The values in the table above are four-seed medians from Table 2. Time to quality is the first recorded validation checkpoint whose score is no greater than 1.005 times the paired Adam endpoint. Complete POLLU runs of OM-EKI use 100,385–106,852 objective evaluations. The batching and resource controls in Appendix D.1 show how independent forward evaluations can be performed in parallel, connecting optimization performance with computational work.

Additional DFBGN comparisons (four-seed medians; Tables 1 and 3)

TaskMethodValidation metric ↓Secondary metricTime (s)
ODEDFBGN (restart)MAE: 6.37668e-5Test MAE: 7.24218e-5192.20
ODEOM-EKIMAE: 0.72985e-5Test MAE: 0.98288e-5154.71
SHDDFBGN (restart)Brier: 0.046197Accuracy: 17.17%2,207.83
SHDOM-EKI (cap 256)Brier: 0.042218Accuracy: 29.83%1,401.56

Question 2 — Comparison with reduced Gauss–Newton

Response

Following this suggestion, we introduce OM-GN, which uses the measured response in observable coordinates to perform regularized Gauss–Newton correction (Section 4, Eq. 18; Appendix B.1). We compare the two updates on the deterministic ODE, POLLU, Allen–Cahn, and PDE-control tasks. In the ODE and POLLU comparisons, OM-GN resets regularization and reconstructs the map once after consecutive numerical stalls; if the stalls recur, the run stops (Appendix E.7). Their complementary results show that OM is a reusable response-driven representation that can connect different optimization mechanisms.

Task and metricOM-EKIOM-GN
ODE test MAE ↓0.98288e-53.3876e-5
POLLU validation log-MSE ↓2.4054e-52.48481e-5
POLLU time to quality (s) ↓1,182.561,071.80
Allen–Cahn validation MSE ↓0.037250.06731
PDE control test cost ↓0.6209540.629771

On POLLU, OM-GN reaches paired Adam quality earlier, whereas OM-EKI achieves lower validation error. The comparison shows the accuracy–time tradeoff between the two update mechanisms, as well as the potential of OM beyond a single optimizer.

Reviewer_R1wHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Observable-Mapping Ensemble Kalman Inversion (OM-EKI) as a derivative-free optimizer for structured-response dynamical-system problems.

Critical strengths
  • There are useful practical results. The deterministic ODE improvement over Adam is clearly demonstrated.
  • Limitations are stated and analyzed well.
Critical weaknesses
  • The paper needs better framing. The contribution should be stated precisely in terms of the particular coordinate construction, residual-aware capacity, probe reuse, and verification which is missing.
  • Matched random-coordinate EKI controls are absent from the primary high-dimensional tasks.
Questions for authors
  • What are the realized numerical ranks and coordinate dimensions in the affine Duffing control? Are null directions padded, or does this experiment use a modified selection rule?
  • Please refer to weaknesses.
2 comments

W1 / Question 2 — Framing the contribution and method components

Response

We thank the reviewer for the suggestion. We recognize that the contributions of some components, such as probe reuse, were not sufficiently organized and explained previously. We have clarified in Section 4 how the components work together. OM measures structured responses and constructs search coordinates; response energy and the current residual guide capacity selection; map reconstruction retains informative directions and adds new probes; and fresh forward evaluations verify finite-step proposals. The mechanism experiments in Section 5.2 (Figure 4; Tables 5–7) connect these choices to optimization performance: response alignment organizes search directions, residual-aware capacity improves the correction that can be represented, drift guides coordinate renewal, and verification connects local predictions with simulator behavior. For example, residual-aware capacity reduces validation MAE on the deterministic ODE by 5.86× with almost the same forward-evaluation budget (Table 6); the controlled drift experiment shows when coordinate renewal is useful (Figure 4(c)). These results make clearer how the components work together to form a response representation that can be used for optimization.

Residual-aware capacity control (Table 6; Figure 4(b))

Capacity ruleResidual capture ↑Validation MAE ↓
Energy only5.76e-74.079e-3
Energy + residual0.9906.961e-4

W2 / Question 2 — Random-coordinate EKI controls

Response

We thank the reviewer for the valuable suggestion. We have added random-coordinate EKI controls on ODE, POLLU, and SHD (Appendix D.4). Table 13 and Figure 6 show the three-task comparison, and Table 14 reports four POLLU runs. OM-EKI achieves lower validation error in each of the three displayed task comparisons and all four POLLU runs. The random method refreshes random subspaces and uses its own probing and acceptance protocol, so this comparison shows the performance of the two complete methods. A further analysis of the four-seed POLLU experiment (Table 14) finds that random EKI makes faster early progress on seed 1, whereas OM-EKI reaches paired Adam quality earlier on the other three seeds and has lower validation error on all four seeds. The performance of random coordinates shows the value of low-dimensional search; the results obtained with measured responses further support organizing search using response structure. On SHD (Figure 6), OM-EKI still shows an improving trend at the end of the recorded run. These results also motivate us to further develop methods that make fuller use of response structure.

Random-coordinate comparison (displayed runs; Table 13)

TaskMethodValidation error ↓Test error ↓Validation accuracy ↑Evaluations
ODEOM-EKI2.231e-65.403e-6—794,151
ODERandom EKI2.775e-52.501e-5—794,170
POLLUOM-EKI2.420e-56.599e-6—100,384
POLLURandom EKI2.648e-56.090e-6—100,362
SHDOM-EKI0.040734—34.67%67,721
SHDRandom EKI0.041838—32.33%52,428

ODE reports MAE; POLLU reports log-MSE; SHD reports Brier score and accuracy.

Four-run POLLU comparison (Table 14; Appendix D.4)

RunOM-EKI validation log-MSE ↓Random EKI validation log-MSE ↓OM-EKI time to Adam quality (s) ↓Random EKI time to Adam quality (s) ↓
12.4195e-52.6479e-5622.5496.4
22.5275e-52.6406e-5456.1679.4
32.4318e-52.6346e-5394.3493.6
42.3842e-52.6536e-5576.1834.9
Median2.4257e-52.6443e-5——

Question 1 — Affine Duffing numerical rank and coordinate dimension

Response

We thank the reviewer for highlighting the relationship between theoretical rank and numerical rank. No null directions are padded, and the coordinate-selection rule is unchanged. In the original single-precision affine Duffing experiment, the measured numerical rank was approximately 39–42, and 32 coordinates were retained. Thus, the 32-dimensional representation was selected from the numerically detected directions rather than constructed by padding null directions. However, this numerical rank exceeds the theoretical upper bound of 16, indicating sensitivity to numerical precision.

To examine this discrepancy, we added a double-precision experiment with 16 retained coordinates. All nine affine Duffing reconstructions identify numerical rank 16, consistent with the theoretical bound, and achieve a validation MSE of 4.826e-13. In comparison, the original single-precision experiment achieves 6.073e-13. For nonlinear Duffing, the measured numerical rank is 23–31, and restricting the representation to 16 coordinates leads to higher reconstruction error.

These results clarify the distinction between theoretical and measured numerical rank and highlight the importance of numerical precision in identifying informative response directions (Appendix D.2; Table 5).

Duffing numerical-rank check (Table 5; Appendix D.2)

SettingMeasured numerical rankCoordinatesValidation MSE ↓
Affine, original single precision39–42326.073e-13
Affine, double precision16164.826e-13
Nonlinear, double precision23–31161.368e-7

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response

We thank the reviewers for their constructive comments. This paper studies how structured forward responses organize derivative-free search in dynamical systems. Observable mapping (OM) transforms these responses into residual-aware search coordinates, and OM-EKI performs ensemble residual correction in these coordinates. The revision strengthens this contribution in six respects:

  1. We clarify how OM constructs and adapts response-informed coordinates according to response energy and the current residual. We also introduce OM-GN, which applies regularized Gauss–Newton updates in the same coordinates. On POLLU, OM-GN reaches paired Adam quality in a median of 1,071.80 s, compared with 1,182.56 s for OM-EKI, while OM-EKI achieves lower validation error. This comparison shows that OM provides a shared coordinate construction for different optimization updates (Section 4; Section 5.1.2 and Table 2; Appendix B.1).
  2. We add a conditional analysis of regularized, clipped GN steps and covariance-weighted EKI ensemble updates. The local-model decrease bounds connect progress to response sensitivity, residual reachability, regularization, and ensemble covariance. We also show how agreement-based acceptance transfers predicted decrease to actual decrease when the prediction and acceptance test use the same objective and decision statistic (Appendix C.5).
  3. We add Q-EKI comparisons using the original probe coordinates on POLLU and random-coordinate EKI controls on ODE, POLLU, and SHD. Q-128 and OM-128 achieve validation log-MSEs of 2.43379×10−52.43379 \times 10^{-5} and 2.41953×10−52.41953 \times 10^{-5}, respectively. Random coordinates also support effective optimization, while OM-EKI achieves lower validation error in the displayed runs. These results show that the probe space can contain useful directions and that measured responses can organize and select them, motivating further work on probe construction and adaptation (Appendices D.3–D.4; Tables 12–14).
  4. We add timed-restart DFBGN controls on ODE, POLLU, SHD, and Allen–Cahn. On POLLU, OM-EKI achieves a lower median validation log-MSE (2.4054×10−52.4054 \times 10^{-5} versus 2.50096×10−52.50096 \times 10^{-5}) and a shorter median time to quality (1,182.56 s versus 1,786.01 s). We report full-run evaluation counts and a batching control: increasing the batch size from 8 to 64 reduces elapsed time by 6.93× for a fixed OM-EKI workload. These results illustrate the role of parallel forward evaluations alongside their computational cost (Tables 2 and 10).
  5. We add 2D and 3D Allen–Cahn field-inversion task, extend closed-loop PDE control to three seeds. On 2D Allen–Cahn, the median validation MSEs of OM-EKI and Adam are 0.03725 and 0.03802, respectively. In PDE control, OM-EKI achieves lower test trajectory cost in all three runs and reaches paired Adam quality a median of 12.36× sooner. In the 3D task, OM-EKI achieves a median validation MSE of 3.033×10−63.033 \times 10^{-6}, close to Adam’s 3.063×10−63.063 \times 10^{-6}. These experiments extend the evaluation to spatial-field reconstruction, feedback control, and 3D coefficient recovery (Section 5.1; Appendices D.8–D.9).
  6. A deeper understanding of response geometry. The one-step and 16-step Burgers results distinguish local residual correction from longer-horizon prediction. A double-precision affine Duffing study identifies rank 16 in all nine reconstructions, consistent with the theoretical upper bound. A residual-aware capacity control reduces ODE validation MAE by 5.86× with nearly the same forward-evaluation budget. Together with the drift and verification controls, these results support the observable-geometry hypothesis and motivate more reliable numerical identification and structured probing of informative directions (Section 5.2 and Tables 5–7; Appendices D.2 and D.7; Tables 18–19).

Together, these revisions strengthen our central claim: structured simulator responses can guide the construction and adaptation of search coordinates across dynamical-system problems.

AuthorsOpenReview ↗

Final Summary

Dear Area Chair,

We thank the reviewers for their time and constructive feedback. As the discussion period concludes, we would like to summarize the central contribution of our work and the additional evidence provided during the rebuttal.

1. Central contribution

Structured forward evaluations contain optimization-relevant information beyond their scalar objective values. We propose to exploit this information by constructing and adapting search coordinates, enabling derivative-free optimization to benefit directly from the underlying response structure.

We introduce observable mapping (OM) to realize this idea and instantiate it through OM-EKI using ensemble Kalman updates, and further through OM-GN using Gauss–Newton updates within the same representation. These two implementations demonstrate the value of OM as a response-informed search representation that extends beyond a particular choice of optimization update.

2. Additional evidence in response to the reviews

During the rebuttal, we substantially strengthened the theoretical and experimental evaluation:

  • Methodological contribution: Introduced OM-GN and additional Q-EKI and random-coordinate EKI controls to clarify the role of response-informed search.
  • Theoretical foundation: Extended the analysis to finite-step descent conditions for regularized and clipped GN updates and covariance-weighted EKI updates, connecting response geometry with optimization progress.
  • Experimental comparisons: Strengthened comparisons with derivative-free and gradient-based optimizers, including restarted DFBGN, and provided further evidence on computational efficiency and the effectiveness of response-informed coordinates.
  • Broader applicability: Extended the evaluation to 2D and 3D PDE inverse problems and multi-seed closed-loop PDE control, achieving competitive reconstruction accuracy and improved control performance compared with Adam. These results further support the applicability of OM to spatially distributed dynamical systems.

Together, the theoretical and experimental results support the proposed methodology across diverse settings, including stiff chemical dynamics, non-smooth spiking networks, PDE inverse problems, and closed-loop control.

3. Overall significance

Beyond the specific algorithms presented, the main message of our work is that forward evaluations can inform how the search itself is organized. Our results suggest that exploiting structured responses in this way can be useful for derivative-free optimization, particularly in scientific and engineering problems where differentiation is expensive or difficult.

We hope that the additional theoretical analyses, controlled comparisons, and extended benchmarks will be taken into account in the final assessment.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.