- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ORACLE wraps a native optimizer. After the optimizer proposes a step, ORACLE evaluates the PINN residuals at the proposed point using fresh samples. If the step increases the residual energy to first order, it computes a damped least-squares correction in the optimizer's own preconditioner metric. The correction is capped relative to the native step. It is accepted only after a ratio test and a held-out check. Appendix B provides one-step properties but no convergence result. The experiments cover 8 PDEs with four optimizers on PirateNet, three model types, cost, six baselines on two PDEs, and ablations. The paper reports that ORACLE improves or matches the native optimizer in 94% and 92% of configurations.
Critical strengths
- The method is simple and fully specified. The residual-space CG solve avoids forming the Jacobian and scales to PirateNet with about parameters. As far as I checked, the one-step derivations in Appendix B are correct.
- Some gains on solved problems are large. With Adam and SOAP, error drops by 40-70% on Allen-Cahn and Wave (Table 11). The paper also reports its negative results: the 4-6x step cost, the loss at equal wall-clock time, and the Pure MLP degradations.
Critical weaknesses
- The evidence does not support the headline claim. The 94% figure counts any lower number as a gain and identical numbers as a match. Under this rule a method that changes nothing would score 100%. Table 11 counts gains of 0.2-1.0% as wins, while the caption gives a std below 5%. In Table 11, 14 of 32 native runs fail with rel-L2 . Only 10 of 32 cells improve by more than 5% on a solved problem. At equal wall-clock time (Fig. 5d) ORACLE is about 5-10x worse for Adam, SOAP and Kron.
- The gains are not tied to the claimed mechanism. Eq. 9 is a Levenberg-Marquardt step, and the acceptance rule is a trust-region rule. Except for Pure MLP, the constraints are the same residuals as the native loss. So ORACLE acts as a bounded Gauss-Newton step on the same loss, not as constrained learning. Pure MLP is the only case with the PDE outside the objective, and there all errors are 0.41-1.72 (Table 13). The First-order variant, one extra preconditioned step, already gives most of the gain (Table 15). The baselines in Section 4.4 are not shown to be tuned: on Allen-Cahn the tuned penalty is about 2x worse than native, although unit weights recover native (Fig. 6c).
Questions for authors
- Please report per-seed values and paired differences with intervals. Count a gain only above seed noise, and list failed runs separately.
- Please give the numbers behind Fig. 5d and compare at equal wall-clock time with native runs with a tuned learning rate, and with a Gauss-Newton or LM optimizer for PINNs.
- Which optimizer and model do Q4 and Q5 use? How were the baselines tuned? Why is the tuned penalty worse than native?
- Please report residual norms per constraint family at the endpoint.
- How were the Table 9 values chosen, and on which PDEs?
5 comments
I thank the authors for the additional results. The revision addresses my secondary questions: per-seed values and paired intervals for Q1, the baseline tuning protocol, per-family residuals, and the numbers behind Fig. 5d. I raise Soundness from 1 to 2 for this. My two main weaknesses remain. First, the evidence still does not support the headline claim. Second, the gains are not separated from a bounded Gauss–Newton step on the same loss, and Native with a tuned learning rate at equal compute is still missing. I therefore keep my rating.
Thank you for the follow-up and for raising Soundness. We answer the three remaining points.
1. "The evidence still does not support the headline claim."
We already replaced the headline claim, and the main results are now tied to per-seed evidence. The 94% and 92% figures are removed from the abstract (lines 20-25), the contributions (lines 100-102), Sections 4.1-4.2 (lines 319-323, 347-350) and the conclusion (lines 480-483). In their place, Section 4.1 reports what the data shows: ORACLE lowers the error beyond seed noise in 23 of 32 configurations, the largest gain on a problem that Native already solves is 72%, and gains above 5% occur under every optimizer family. Table 14 (line 1634) gives the per-seed values and paired 95% intervals behind each number, and Table 13 (line 1587) lists the failed Native runs separately. This makes the paper stronger: each number in Section 4.1 can now be checked against a table.
2. "The gains are not separated from a bounded Gauss-Newton step on the same loss."
We ran this step as a control: Native plus a bounded Gauss-Newton step on the same loss, at equal wall-clock time (relative L2 error, lower is better).
| PDE | ORACLE | Native | LM | Native + bounded GN |
|---|---|---|---|---|
| Burgers | 7.15e-4 | 1.33e-4 | 1.07e-4 | 1.61e-4 |
| KdV | 2.26e-2 | 6.42e-2 | 3.26e-2 | 4.50e-2 |
| Wave | 3.77e-4 | 7.11e-4 | 1.49e-2 | 1.52e-3 |
| Allen-Cahn | 6.52e-3 | 5.26e-3 | 3.65e-2 | 4.11e-1 |
ORACLE does not act as this step. The bounded Gauss-Newton step makes Native worse on three of four PDEs and fails on Allen-Cahn, while ORACLE has the lower error than this step on three of four. ORACLE is also the most consistent of the three alternatives to Native: it is the best method on two PDEs, and its largest loss to Native is 5.4x, against 21x for LM and 78x for the bounded Gauss-Newton step.
The two PDEs where ORACLE is behind Native have a clear reason. On Burgers, Native already converges to high accuracy, so there is little left to correct (Section 5, line 483); at matched steps the two are within 1% (Table 14). On Allen-Cahn, ORACLE is ahead at matched steps (38% lower error, Table 14), but it completes about two thirds of Native's steps in the same time (Table 18), so it is behind at equal wall-clock time.
3. "Native with a tuned learning rate at equal compute is still missing."
We ran a robustness analysis over the learning rate at equal wall-clock time, for both methods on the same grid (relative L2 error, lower is better).
| Learning rate | Burgers: Native | Burgers: ORACLE | Wave: Native | Wave: ORACLE | Allen-Cahn: Native | Allen-Cahn: ORACLE |
|---|---|---|---|---|---|---|
| 3e-6 | 1.9e-4 | 9.8e-5 | 5.4e-4 | 5.8e-4 | 4.85e-2 | 4.82e-2 |
| 1e-5 | 9.5e-5 | 9.05e-5 | 2.6e-4 | 2.3e-4 | 1.42e-2 | 1.31e-2 |
| 3e-5 | 9.7e-5 | 9.2e-5 | 1.7e-4 | 1.1e-4 | 3.45e-3 | 3.29e-3 |
| 1e-4 | 2.1e-4 | 9.3e-5 | 2.7e-4 | 1.6e-4 | 1.26e-3 | 1.13e-3 |
| 3e-4 | 2.8e-4 | 2.9e-4 | 7.0e-4 | 6.0e-3 | 1.97e-3 | 1.53e-3 |
| 1e-3 | 6.1e-4 | 6.5e-4 | 1.7e-3 | 2.3e-3 | 8.00e-3 | 7.78e-3 |
| 3e-3 | 6.0e-3 | 5.2e-2 | 1.31 | 2.9e-2 | 5.41e-3 | 9.46e-3 |
At each method's best learning rate, ORACLE has the lower error than Native on all three PDEs, and it has the lower error at 14 of the 21 settings. Below 3e-4 it is behind only once (Wave at 3e-6, by 7%); its clear losses are at the larger learning rates (3e-4 and above). A likely reason is that ORACLE relies on a local model of the constraints (Section 5, line 484), which is less accurate when the native step is large.
We did not tune ORACLE in this paper, and this is deliberate. ORACLE is meant as a general method: one configuration, built from standard trust-region constants and fixed design choices, is used for every PDE, optimizer and model in the paper (Table 10; Appendix C.9, lines 1491-1501). ORACLE does have parameters that can be tuned for a specific benchmark, such as the support size, the number of CG iterations, the damping and the authority bounds. We keep them fixed so that the results show what ORACLE gives without per-problem tuning.
We thank the reviewer for the careful evaluation. Your comments, together with those of Reviewer En5J, prompted us to examine ORACLE's computational overhead more closely. This led to a updated GPU-optimized implementation that substantially improves equal-wall-clock performance without changing the underlying method.
Table A. Outcome of the 32 cells (paper Table 13, line 1587).
Table B. Numbers behind Fig. 5(d): geometric-mean error ratio ORACLE/Native.
[W1] We updated the claim in the abstract (lines 18-25), Sections 4.1-4.2 (lines 319-323, 347-350) and the conclusion (lines 480-485). We also updated the outcome table: Table A counts a gain only if its paired 95% interval excludes zero, reports gains below 5% as a separate class. ORACLE was less efficient at equal wall-clock time (Fig. 5d) because of implementation overhead, which we removed without changing the method. ORACLE now has the lower error on geometric mean for all four optimizers (Table B). Section 4.3 (lines 411-413) names where it is still worse.
[W2] Replacing the optimizer geometry, which is the claimed mechanism, with Pt=I while keeping all other components the same makes the error larger than Native (Section 4.5, line 456; Table 22). ORACLE composes a bounded Gauss-Newton step with a stateful optimizer: in the optimizer's certified metric (Eq. 6), bounded relative to its step (Eq. 11), and without changing its state (line 161). This composition is what makes it a constrained-learning mechanism. The gain is not from minimizing the same loss harder: on Allen-Cahn, the methods with the lowest residuals have larger errors than ORACLE (Table 20, line 1959). The First-order variant keeps the optimizer geometry and the gates and gives about two thirds of the gain (Table 22, line 1999), so these are the main source of the improvement. For Pure MLP, the paper reports the large errors as a limitation (Section 4.2, line 351). We also report how each baseline was tuned (Appendix C.8, lines 1449-1458; Table 9).
[Q1] We report all seeds with paired differences and 95% intervals for all 32 cells in Table 14 (line 1634), and the updated counts in Table A (paper Table 13).
[Q2] The equal wall clock numbers are in Table B; Table 17 (line 1904) gives all 32 cells. Table 18 (line 1925) adds an LM optimizer at equal wall-clock time on four PDEs. Native and ORACLE share the learning rate of Table 5 (line 1382).
[Q3] Adam is the native optimizer and PirateNet is the architecture for Q4 and Q5 (Appendix C.8, line 1449). Only Penalty and ConFIG+GUA have a free hyperparameter (grids in Table 9, line 1475). The rule was fixed before tuning: lowest constraint energy on a held-out validation support, never rel-L2 (lines 1451-1454). The tuned penalty is worse than Native because the tuning rule does not see the solution error. The grid contains Native (μ=1), but on Allen-Cahn the rule selects μ=31.6: lower constraint energy, higher solution error (lines 1455-1458).
[Q4] We report residual norms per constraint family at the endpoint in Tables 20-21 (lines 1959, 1970) and Table 26 (line 2136).
[Q5] One configuration (now Table 10) was fixed for all PDEs; none was tuned per PDE. Appendix C.9 (lines 1491-1501) gives the origin of each value.