acceptodds
Under review as a conference paper at ICLR 2027

ORACLE: Optimizer-Relative Alignment for Constrained LEarning

Abstract

Constraint-handling methods typically intervene before the optimizer acts, by modifying the objective or the gradient. Yet momentum, adaptive scaling, and structured preconditioning can substantially reshape that signal before it becomes a parameter update. We formulate optimizer-relative constrained learning, where constraint compatibility is assessed on the post optimiser update. Building on this view, we introduce ORACLE, which takes the native optimizer's original proposed step, aligns it when locally adverse in the optimizer's own geometry, limits the correction, and commits it only after validation. We evaluate ORACLE across eight Partial Differential Equation benchmarks and four optimizers spanning Euclidean, diagonal-adaptive, and structured preconditioned geometries, where it improves or matches native optimiser in 94% of configurations. Cross-model analysis shows the same behavior in 92% of configurations, while matched comparisons show improvements over alternative constraint-handling methods acting at the objective, gradient, and post optimizer levels.

Primary area:
optimization
Keywords:
constrained learning, optimizer geometry, preconditioning, physics-informed neural networks, post-optimizer correction, trust region, partial differential equations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4DaYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ORACLE wraps a native optimizer. After the optimizer proposes a step, ORACLE evaluates the PINN residuals at the proposed point using fresh samples. If the step increases the residual energy to first order, it computes a damped least-squares correction in the optimizer's own preconditioner metric. The correction is capped relative to the native step. It is accepted only after a ratio test and a held-out check. Appendix B provides one-step properties but no convergence result. The experiments cover 8 PDEs with four optimizers on PirateNet, three model types, cost, six baselines on two PDEs, and ablations. The paper reports that ORACLE improves or matches the native optimizer in 94% and 92% of configurations.

Critical strengths
  • The method is simple and fully specified. The residual-space CG solve avoids forming the Jacobian and scales to PirateNet with about 7×1057\times10^5 parameters. As far as I checked, the one-step derivations in Appendix B are correct.
  • Some gains on solved problems are large. With Adam and SOAP, error drops by 40-70% on Allen-Cahn and Wave (Table 11). The paper also reports its negative results: the 4-6x step cost, the loss at equal wall-clock time, and the Pure MLP degradations.
Critical weaknesses
  • The evidence does not support the headline claim. The 94% figure counts any lower number as a gain and identical numbers as a match. Under this rule a method that changes nothing would score 100%. Table 11 counts gains of 0.2-1.0% as wins, while the caption gives a std below 5%. In Table 11, 14 of 32 native runs fail with rel-L2 ≥0.5\ge0.5. Only 10 of 32 cells improve by more than 5% on a solved problem. At equal wall-clock time (Fig. 5d) ORACLE is about 5-10x worse for Adam, SOAP and Kron.
  • The gains are not tied to the claimed mechanism. Eq. 9 is a Levenberg-Marquardt step, and the acceptance rule is a trust-region rule. Except for Pure MLP, the constraints are the same residuals as the native loss. So ORACLE acts as a bounded Gauss-Newton step on the same loss, not as constrained learning. Pure MLP is the only case with the PDE outside the objective, and there all errors are 0.41-1.72 (Table 13). The First-order variant, one extra preconditioned step, already gives most of the gain (Table 15). The baselines in Section 4.4 are not shown to be tuned: on Allen-Cahn the tuned penalty is about 2x worse than native, although unit weights recover native (Fig. 6c).
Questions for authors
  • Please report per-seed values and paired differences with intervals. Count a gain only above seed noise, and list failed runs separately.
  • Please give the numbers behind Fig. 5d and compare at equal wall-clock time with native runs with a tuned learning rate, and with a Gauss-Newton or LM optimizer for PINNs.
  • Which optimizer and model do Q4 and Q5 use? How were the baselines tuned? Why is the tuned penalty worse than native?
  • Please report residual norms per constraint family at the endpoint.
  • How were the Table 9 values chosen, and on which PDEs?
5 comments

We thank the reviewer for the careful evaluation. Your comments, together with those of Reviewer En5J, prompted us to examine ORACLE's computational overhead more closely. This led to a updated GPU-optimized implementation that substantially improves equal-wall-clock performance without changing the underlying method.

Table A. Outcome of the 32 cells (paper Table 13, line 1587).

Improve >5%Improve <5%IdenticalDegrade <5%Degrade >5%
Native solves (18)85311
Native fails (14)64400

Table B. Numbers behind Fig. 5(d): geometric-mean error ratio ORACLE/Native.

SGDAdamSOAPKron
Submitted paper0.274.89.99.4
Revised paper0.210.850.760.91

[W1] We updated the claim in the abstract (lines 18-25), Sections 4.1-4.2 (lines 319-323, 347-350) and the conclusion (lines 480-485). We also updated the outcome table: Table A counts a gain only if its paired 95% interval excludes zero, reports gains below 5% as a separate class. ORACLE was less efficient at equal wall-clock time (Fig. 5d) because of implementation overhead, which we removed without changing the method. ORACLE now has the lower error on geometric mean for all four optimizers (Table B). Section 4.3 (lines 411-413) names where it is still worse.

[W2] Replacing the optimizer geometry, which is the claimed mechanism, with Pt=IP_t = I while keeping all other components the same makes the error larger than Native (Section 4.5, line 456; Table 22). ORACLE composes a bounded Gauss-Newton step with a stateful optimizer: in the optimizer's certified metric (Eq. 6), bounded relative to its step (Eq. 11), and without changing its state (line 161). This composition is what makes it a constrained-learning mechanism. The gain is not from minimizing the same loss harder: on Allen-Cahn, the methods with the lowest residuals have larger errors than ORACLE (Table 20, line 1959). The First-order variant keeps the optimizer geometry and the gates and gives about two thirds of the gain (Table 22, line 1999), so these are the main source of the improvement. For Pure MLP, the paper reports the large errors as a limitation (Section 4.2, line 351). We also report how each baseline was tuned (Appendix C.8, lines 1449-1458; Table 9).

[Q1] We report all seeds with paired differences and 95% intervals for all 32 cells in Table 14 (line 1634), and the updated counts in Table A (paper Table 13).

[Q2] The equal wall clock numbers are in Table B; Table 17 (line 1904) gives all 32 cells. Table 18 (line 1925) adds an LM optimizer at equal wall-clock time on four PDEs. Native and ORACLE share the learning rate of Table 5 (line 1382).

[Q3] Adam is the native optimizer and PirateNet is the architecture for Q4 and Q5 (Appendix C.8, line 1449). Only Penalty and ConFIG+GUA have a free hyperparameter (grids in Table 9, line 1475). The rule was fixed before tuning: lowest constraint energy on a held-out validation support, never rel-L2 (lines 1451-1454). The tuned penalty is worse than Native because the tuning rule does not see the solution error. The grid contains Native (μ=1\mu = 1), but on Allen-Cahn the rule selects μ=31.6\mu = 31.6: lower constraint energy, higher solution error (lines 1455-1458).

[Q4] We report residual norms per constraint family at the endpoint in Tables 20-21 (lines 1959, 1970) and Table 26 (line 2136).

[Q5] One configuration (now Table 10) was fixed for all PDEs; none was tuned per PDE. Appendix C.9 (lines 1491-1501) gives the origin of each value.

I thank the authors for the additional results. The revision addresses my secondary questions: per-seed values and paired intervals for Q1, the baseline tuning protocol, per-family residuals, and the numbers behind Fig. 5d. I raise Soundness from 1 to 2 for this. My two main weaknesses remain. First, the evidence still does not support the headline claim. Second, the gains are not separated from a bounded Gauss–Newton step on the same loss, and Native with a tuned learning rate at equal compute is still missing. I therefore keep my rating.

Thank you for the follow-up and for raising Soundness. We answer the three remaining points.

1. "The evidence still does not support the headline claim."

We already replaced the headline claim, and the main results are now tied to per-seed evidence. The 94% and 92% figures are removed from the abstract (lines 20-25), the contributions (lines 100-102), Sections 4.1-4.2 (lines 319-323, 347-350) and the conclusion (lines 480-483). In their place, Section 4.1 reports what the data shows: ORACLE lowers the error beyond seed noise in 23 of 32 configurations, the largest gain on a problem that Native already solves is 72%, and gains above 5% occur under every optimizer family. Table 14 (line 1634) gives the per-seed values and paired 95% intervals behind each number, and Table 13 (line 1587) lists the failed Native runs separately. This makes the paper stronger: each number in Section 4.1 can now be checked against a table.

2. "The gains are not separated from a bounded Gauss-Newton step on the same loss."

We ran this step as a control: Native plus a bounded Gauss-Newton step on the same loss, at equal wall-clock time (relative L2 error, lower is better).

PDEORACLENativeLMNative + bounded GN
Burgers7.15e-41.33e-41.07e-41.61e-4
KdV2.26e-26.42e-23.26e-24.50e-2
Wave3.77e-47.11e-41.49e-21.52e-3
Allen-Cahn6.52e-35.26e-33.65e-24.11e-1

ORACLE does not act as this step. The bounded Gauss-Newton step makes Native worse on three of four PDEs and fails on Allen-Cahn, while ORACLE has the lower error than this step on three of four. ORACLE is also the most consistent of the three alternatives to Native: it is the best method on two PDEs, and its largest loss to Native is 5.4x, against 21x for LM and 78x for the bounded Gauss-Newton step.

The two PDEs where ORACLE is behind Native have a clear reason. On Burgers, Native already converges to high accuracy, so there is little left to correct (Section 5, line 483); at matched steps the two are within 1% (Table 14). On Allen-Cahn, ORACLE is ahead at matched steps (38% lower error, Table 14), but it completes about two thirds of Native's steps in the same time (Table 18), so it is behind at equal wall-clock time.

3. "Native with a tuned learning rate at equal compute is still missing."

We ran a robustness analysis over the learning rate at equal wall-clock time, for both methods on the same grid (relative L2 error, lower is better).

Learning rateBurgers: NativeBurgers: ORACLEWave: NativeWave: ORACLEAllen-Cahn: NativeAllen-Cahn: ORACLE
3e-61.9e-49.8e-55.4e-45.8e-44.85e-24.82e-2
1e-59.5e-59.05e-52.6e-42.3e-41.42e-21.31e-2
3e-59.7e-59.2e-51.7e-41.1e-43.45e-33.29e-3
1e-42.1e-49.3e-52.7e-41.6e-41.26e-31.13e-3
3e-42.8e-42.9e-47.0e-46.0e-31.97e-31.53e-3
1e-36.1e-46.5e-41.7e-32.3e-38.00e-37.78e-3
3e-36.0e-35.2e-21.312.9e-25.41e-39.46e-3

At each method's best learning rate, ORACLE has the lower error than Native on all three PDEs, and it has the lower error at 14 of the 21 settings. Below 3e-4 it is behind only once (Wave at 3e-6, by 7%); its clear losses are at the larger learning rates (3e-4 and above). A likely reason is that ORACLE relies on a local model of the constraints (Section 5, line 484), which is less accurate when the native step is large.

We did not tune ORACLE in this paper, and this is deliberate. ORACLE is meant as a general method: one configuration, built from standard trust-region constants and fixed design choices, is used for every PDE, optimizer and model in the paper (Table 10; Appendix C.9, lines 1491-1501). ORACLE does have parameters that can be tuned for a specific benchmark, such as the support size, the number of CG iterations, the damping and the authority bounds. We keep them fixed so that the results show what ORACLE gives without per-problem tuning.

Reviewer_En5JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies and tackles the problem of enforcing constraints on optimization problem solutions. They identify fixes applied directly to the gradient or loss function as insufficient to prevent constraint violations due to sophisticated optimizer mechanisms (momentum, etc.), and propose fixing the parameter update instead. They evaluate the step proposed by the native optimizer on a linear combination of the constraints to determine compatibility with the constraints, and propose a corrective step (a fix) to improve the compliance of the updated parameters with the given constraints, conditional on several conditions intent on maintaining acceptability of the proposed and corrective step. They evaluate their model on PINNs to demonstrate the viability of their method.

Critical strengths

This paper identifies and tackles an important problem - that of enforcing constraints on optimization problem solutions. The proposed solution seems to be a good approach to the problem, satisfying the identified weaknesses of previous methods.

The paper evaluates the computational cost of applying its method, demonstrating that it may not be prohibitive while remaining candid about the added training time and about the effect this has on fixed-budget training. They also present a very strong experiments section, which is a core strength of this paper.

Critical weaknesses

Poorly defined gates, see question 3 (line 186) below. I believe this was a typo, especially since you didn't use P_t despite defining it, so a meaningful and thoughtful correction to this definition that makes the "geometric" justification for that gate make sense (currently I see no connection between cert_t and any particular geometric concerns) will eliminate this concern.

In equation 1, you apply several gates to the update fix, but this could cause your method to fail to do anything to satisfy the constraints. In particular, equation 8 only requires the native optimizer step s_t to step towards constraint energy reduction to disable the corrective step d_t, but the step may be so minor that the constraint energy remains high, which is remains problematic.

Please also provide a thoughtful answer to question 1 below questioning your choice of evaluation on a linear combination of the constraints instead of on the constraints themselves.

Note to authors: while these weaknesses are sufficient to make me unable to support accepting this paper in its current form, I believe that they should not be overly difficult to overcome, and I encourage the authors to do so, as this is a well written paper with an interesting idea. I will seriously consider raising my score accordingly.

Questions for authors
  1. What is the effect of evaluating a linear combination of the constraints r_k instead of the constraints themselves, one at a time? Is it possible that a linear combination of constraints could be satisfied while the constraints themselves are not?

  2. Is there existing research that tackles the problem of constraint satisfaction by modifying the parameter update proposed by the native optimizer? I didn't see any discussed in the related work section, yet you included another post-update method (config+GUA) in your experiments section. Please at least include some explanation of the post-update method used in the experiments section in the paper, it is highly related.

  3. Line 186, equation 6: Doen't the LHS of that inequality always equal 0? On line 178 you defined s_t=-B_t u_t, and the LHS of equation 6 is s_t + B_t u_t; plugging in the definition of s_t we get s_t + B_t u_t = -B_t u_t + B_t u_t = 0. What is cert_t supposed to mean and why would it justify cancelling the update fix d_t?

  4. What is the memory overhead imposed by ORACLE?

2 comments

We thank the reviewer for recognizing the quality and novelty of our work. Your comments, together with those of Reviewer 4DaY, prompted us to investigate ORACLE's computational overhead more closely. This led to a GPU-optimized implementation that substantially improves equal-wall-clock performance without changing the underlying method.

Table A. Wave: violation of each constraint at the end of training (paper Table 21, line 1970; Allen-Cahn: Table 20, line 1959).

MethodPDEIC: u0u_0IC: ut,0u_{t,0}BCrel-L2
Native1.41e+001.39e-031.92e-021.76e-035.11e-03
ReLoBRaLo3.11e-012.23e-011.14e-026.59e-023.90e-01
ORACLE9.47e-017.39e-041.11e-021.14e-032.84e-03

[W1] We discuss the gate in detail under Q3. PtP_t is used: it defines the alignment (Eqs. 9-10) and the authority bound (Eq. 11). The gate guarantees that PtP_t is the geometry that produced the step; this is its geometric justification (Section 3.2, lines 195-196).

[W2] Equation 1 has three gates: certification, the trigger (equation 8) and validation. Each gate can cancel the update fix dtd_t on one step, but together they do not stop the method. The update fix passes all three gates and is applied on 45% of all steps with SGD and on 84-90% with Adam, SOAP and PSGD-Kron (Table 27, line 2161, column "Accept").

Equation 8 disables dtd_t only on a step where the native optimizer step sts_t already steps towards constraint energy reduction. On that step the native optimizer is already doing what the update fix would do, so ORACLE leaves sts_t unchanged (Section 3.2, lines 210-212). If that step is minor and the constraint energy remains high, this is not final: equation 8 is evaluated again at the next step on fresh samples, and it enables dtd_t as soon as a native step increases the constraint energy. In practice it enables dtd_t on 60% of steps with SGD and on 88-95% with Adam, SOAP and PSGD-Kron (Table 27, column "Trigger").

[W3] Please see Q1.

[Q1] The corrective step does not use the linear combination, it fits every residual of every constraint (Eq. 9, line 220). Only the trigger (Eq. 8) and the validation (Eqs. 12-13) use it. The combination is a weighted sum of squared residuals (Eq. 3, Section 3.1, line 165), so it is zero only when every constraint is satisfied. We also measured the constraints one at a time (Table A): on Wave, ORACLE lowers every constraint, while a reweighted combination (ReLoBRaLo) lowers the PDE residual and raises two others. On Allen-Cahn, ORACLE lowers the IC residual and has a higher PDE residual than Native (Table 20).

[Q2] The closest existing method is GUA (Xiao et al., 2026). We added GUA to the related work section, with how ORACLE differs from it (Section 2, lines 131-135), and we explain the post-update baseline in Table 8 (Appendix C.8, line 1459).

[Q3] Reviewer YzQT also pointed this about the notation. We used sts_t for two different steps. The corrected Eq. 6 (Section 3.2, lines 189-198) separates them:

Certt={∥stdep+B^tu^t∥2≤τrel∥stdep∥2+τabs},Pt=B^t−1\mathrm{Cert_t} = \Big\lbrace \Vert s_t^{\mathrm{dep}} + \hat B_t \hat u_t \Vert_2 \le \tau_{\mathrm{rel}} \Vert s_t^{\mathrm{dep}} \Vert_2 + \tau_{\mathrm{abs}} \Big\rbrace, \qquad P_t = \hat B_t^{-1}

stdeps_t^{\text{dep}} is the step the optimizer code really took. −B^tu^t-\hat B_t \hat u_t is the step rebuilt from the optimizer's state (Appendix B.2). Certt\text{Cert}_t means that the rebuilt step matches the step the optimizer really took. If they do not match, the update fix dtd_t would be built and bounded in the wrong geometry (lines 195-196). Cancelling dtd_t is then the safe choice: using Pt=IP_t = I instead makes the error larger than Native (Table 22, line 1999).

[Q4] ORACLE's peak GPU memory equals Native's in all 32 PDE-optimizer configurations (new Table 16, line 1865; Section 4.3, line 373). Your question led us to optimize the implementation, without changing the method. The original implementation ran constraint evaluation, metric construction and validation as separate GPU operations, with repeated computation and CPU-GPU synchronization. The GPU-optimised ORACLE avoids recomputing quantities the native optimizer already produced, fuses the optimizer and correction steps through JIT compilation, runs gating and validation entirely on the GPU, and batches diagnostic logging. The paper also has an updated equal wall-clock comparison (Fig. 5d), which shows that ORACLE is efficient as well.

Thank you again for your detailed review. As the discussion closes on 18 November, we have uploaded a revised PDF with the corrected Eq. 6 and the geometric meaning of the certification gate (Section 3.2), a clarification of when the gates apply the update fix and why the trigger does not stop the method (Section 3.2, Table 27), per-constraint residuals instead of only their combination (Tables 20-21), GUA and the post-update baseline in the related work (Section 2, Table 8), and a GPU-optimized implementation whose peak memory equals the native optimizer's in all 32 configurations (Table 16).

The other reviewers also recognized the value of the work. Reviewer YzQT found "the paper is well-motivated" and noted the "thorough ablation studies, showing that ORACLE works across different choices of optimizers, model architectures". Reviewer 4DaY found that "the method is simple and fully specified" and that "the one-step derivations in Appendix B are correct".

We hope our responses have addressed the weaknesses you raised. If so, we would greatly appreciate it if you could raise your score, as you kindly offered in your review.

Reviewer_YzQTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a method for enforcing constraints on the optimizer's update direction instead of the raw gradient, specifically designed for physics-informed Partial Differential Equation learning. The method is comprised of: (1) a certification gate that checks that the geometry of the optimizer can be captured by a matrix BtB_t, (2) a trigger gate that determines if the update direction locally increases constraint violation, (3) an alignment step that computes a direction to locally undo the constraint violation due to the optimizer's update, (4) step size αt\alpha_t for the alignment direction, (5) a validation gate that checks if the actual constraint violation is improved.

Critical strengths
  1. The paper is well-motivated. Prior works either cannot reconcile competing gradients, or directly enforce constraints on the raw gradient which does not resepct the optimizer's geometry.
  2. Thorough ablation studies, showing that ORACLE works across different choices of optimizers, model architectures.
Critical weaknesses
  1. One of the advantage of ORACLE is that the alignment direction respects the optimizer geometry, but the paper does not evaluate how this improves constraint handling.
  2. Limited theoretical guarantees and lack of convergence guarantees. The method seems to be heavily driven by heuristics. However, it is justifiable if strong empirical performance is demonstrated.
Questions for authors
  1. Can you please clarify why the certification gate is needed? When you already know that the optimizer's update has the form st=−Btuts_t = -B_t u_t, why do you still need the gate? When the optimizer's update does not follow this form, how do you obtain BtB_t and PtP_t?
  2. How often does the validation gate set the alignment direction to zero?
2 comments

We thank the reviewer for the thoughtful comments. Line numbers refer to the revised PDF.

Table A. Constraint violation of each constraint family at the end of training (paper: new Table 26, Appendix E.4, line 2136).

MethodAC: PDEAC: ICWave: PDEWave: BCWave: IC
Native1.34e-021.72e-031.41e+001.76e-031.36e-02
Pt=IP_t = I5.24e-026.96e-033.73e+003.58e-038.03e-03
ORACLE1.94e-021.22e-039.47e-011.14e-038.05e-03

Table B. How often the validation gate sets the alignment direction to zero (computed from paper Table 27, Appendix E.6, line 2161).

OptimizerCorrection proposed (% of steps)Correction used (% of steps)Set to zero by validation (% of proposals)
SGD59.6%44.8%24.8%
Adam88.5%84.4%4.6%
SOAP94.7%90.5%4.5%
PSGD-Kron90.2%87.2%3.3%

[W1] Figure 7 reports the importance of the optimizer geometry. Table A strengthens this and shows the improvement in constraint handling: it compares the alignment direction that respects the optimizer geometry (ORACLE) with the one that does not (Pt=IP_t = I). Respecting the optimizer geometry lowers the constraint violation on four of five families and keeps it equal on the fifth. This is reported in Section 4.5 (line 472).

[W2] ORACLE gives per-step guarantees, not a convergence guarantee (Section 5, line 499). This is by design, because ORACLE is a wrapper around the native optimizer. It never changes the optimizer's state (Section 3.1, line 161), it returns the native update exactly when no correction is committed (Eq. 77, line 1213), and it bounds every correction relative to the native step (Eq. 62, line 1077). Each gate is a standard tool with a proven one-step property. The correction is a Levenberg-Marquardt step that is a descent direction for the constraints (Appendix B.5, line 1024). The validation is a trust-region ratio test and a held-out audit with a finite-sample guarantee (Appendix B.6, Eq. 73). Optimizer Geometry is supported by the strong empirical results on multiple native optimizers, benchmarks and architectures (Tables 12-15).

[Q1] The certification gate checks that the step the optimizer code really took is the step that the form st=−Btuts_t = -B_t u_t predicts. Eq. 6 was unclear because we used sts_t for both steps. Reviewer En5J also pointed this about the notation. We updated it (Section 3.2, lines 189-198):

Certt={∥stdep+B^tu^t∥2≤τrel∥stdep∥2+τabs}\mathrm{Cert}_t = \Big\lbrace \Vert s_t^{\mathrm{dep}} + \hat B_t \hat u_t \Vert_2 \le \tau_{\mathrm{rel}} \Vert s_t^{\mathrm{dep}} \Vert_2 + \tau_{\mathrm{abs}} \Big\rbrace

Here stdeps_t^{\text{dep}} is the step the optimizer code took, and −B^tu^t-\hat B_t \hat u_t is the step rebuilt from the form. The gate is needed because the optimizer's update does not follow this form on every step. PSGD-Kron leaves the form when its clipping is active (Appendix B.3, lines 901-903), and SOAP must be rebuilt with the basis it really used for the step (Appendix B.2, line 862). When the optimizer's update does not follow this form, ORACLE keeps the native step and never replaces the geometry with Pt=IP_t = I (Section 3.2, lines 196-197). Replacing it with Pt=IP_t = I makes the error larger than Native (Table 22, line 1999).

[Q2] The validation gate sets the alignment direction to zero for 24.8% of the proposed corrections with SGD, and for 3-5% with Adam, SOAP and PSGD-Kron (Table B). In these cases ORACLE keeps the native step.

Thank you again for your detailed review. As the discussion closes on 18 November, we have uploaded a revised PDF with a direct measurement of how the optimizer geometry improves constraint handling (new Table 26, Section 4.5), the corrected definition of the certification gate and why it is needed (Eq. 6, Section 3.2), how often the validation gate sets the alignment direction to zero (Table 27), per-seed results with paired 95% intervals (Tables 13-14), and the scope of ORACLE's per-step guarantees stated in Section 5.

The other reviewers also recognized the value of the work. Reviewer En5J noted that the paper presents "a very strong experiments section, which is a core strength of this paper" and called it "a well written paper with an interesting idea". Reviewer 4DaY found that "the method is simple and fully specified" and that "the one-step derivations in Appendix B are correct".

We hope our responses have addressed your concerns. If so, we would greatly appreciate it if you could consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.