acceptodds
Under review as a conference paper at ICLR 2027

Training Numerical Intelligence via Auto-Diagnosis and Skill Discovery

Abstract

AI agents are becoming increasingly capable of generating scientific code, but generating code is not the same as improving the algorithms behind it. For numerical solvers, execution feedback can expose poor performance, but rarely reveals its underlying cause and how to address it. We introduce uto-iagnosis and kill iscovery (ADSD), a framework that links numerical diagnosis to reusable solver self-improvement. ADSD follows a diagnosis-first paradigm that first explains why a solver performs poorly, then uses this diagnosis to guide the discovery of appropriate numerical methods. The resulting knowledge is packaged into reusable solver skills, turning solver improvement from trial-and-error editing into a structured process of diagnosis, discovery, and implementation. Across four challenging numerical domains—power flow equation, AC optimal power flow control, stiff ordinary differential equations, and heterogeneous diffusion PDEs—ADSD consistently improves solver accuracy, robustness, and efficiency. On GOC-500 power flow, for example, ADSD reduces mean solver error by nearly , with improvements further transferring to unseen grid topologies and operating regimes.

Primary area:
generative models
Keywords:
LLM agents, numerical solvers, power-grid engineering, dynamic systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rp9HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ADSD, a diagnosis-driven framework for improving numerical solvers without updating model parameters. ADSD identifies numerical issues from execution traces, retrieves relevant literature, and converts validated methods into reusable solver skills. Experiments across power flow, AC-OPF, stiff ODEs, and diffusion PDEs demonstrate improvements in accuracy, efficiency, and robustness.

Critical strengths
  • The paper addresses a meaningful limitation of existing coding agents by emphasizing numerical diagnosis rather than trial-and-error code refinement.

  • The framework integrates numerical diagnosis, literature retrieval, theoretical verification, and executable skill construction into a coherent workflow.

  • Extensive experiments across four numerical domains and three foundation models demonstrate consistent improvements, with additional evaluations of generalization across grid topologies and operating conditions.

Critical weaknesses
  • Section 3.1 accepts hypotheses exceeding a 90% verification threshold without clearly defining what constitutes successful verification.
  • Eq. 7 in Appendix B.2 uses the number of KKT solver calls to predict non-convergence, although this may be a consequence of premature termination rather than its underlying cause.
  • Table 1 highlights a roughly 71-fold error reduction on GOC-500, although the baseline residual is already around (10^{-9}), with no downstream engineering evaluation demonstrating the value of further reductions.
  • Table 6 shows that GPT-5.5's runtime on the GOC-500 N1 topology increases from 0.38 to 0.62 seconds. It seems that the efficiency gains are not consistent across settings.
Questions for authors

see weakness

2 comments
AuthorsOpenReview ↗

Author response

We genuinely appreciate your time to provide these constructive and helpful feedbacks. We address each point below:


W1: without clearly defining what constitutes successful verification.

We apologize for the lack of clarity. As described in lines 206–207, we split the public cases into a diagnosis pool for developing hypotheses and a separate verification pool for testing them. Each hypothesis is converted into a deterministic detector with an explicit numerical condition and a predicted solver behavior. For example, Eq. 3 in Appendix B.1 hypothesizes that cases satisfying σmin⁡(J)≤0.16\sigma_{\min}(J)\leq 0.16 require more nonlinear iterations on average under N0N_0 than cases above this cutoff. Verification accuracy is the fraction of all cases in the verification pool for which the predicted behavior matches the observed behavior, and a hypothesis is retained only when this accuracy exceeds 90%.

Typically, rejected candidates achieved verification accuracies less 50%, whereas effective candidates exceeded 90%. To illustrate this screening process, we provide an example from Qwen3.8 AC-OPF, with all its hypotheses provided:

HypothesisVerification pool passing rateResult
H1: Early stall termination explains many failures.99.5%passed to next stage
H2: Narrow reactive-power bounds restrict steps and increase residuals.94.5%passed to next stage
H3: Linear costs create weak curvature and poorly scaled steps.21.0%discarded
H4: Load-dependent residuals arise from unrepaired initialization mismatch.37.2%discarded
H5: Insufficient backtracking leaves larger residuals.6.8%discarded

W2: Eq.7, low KKT solver may be due to premature stop

Good point, we agree, and we have already stated it immediately after Eq.7. Furthermore, the agent found the root cause and solved problem.

Specifically, the premature stop happened because a large initial dual steps led to very small primal step fractions, and the baseline terminated after 13 consecutive iterations without sufficient progress. The agent further identified a scaling imbalance as the underlying cause and addressed it by applying row scaling and revising the KKT progress measure.


W3: significance of the residue & downstream application.

We agree. PF solutions are inputs to downstream engineering tasks, where their accuracy affects the physical quantities used for operational assessment. To demonstrate the downstream benefits of our PF improvements, we evaluate branch-loading assessment on 5000 GOC grids near the voltage-stability boundary, which uses PF solutions to compute branch-loading ratios and measures line utilization relative to capacity and support the assessment of overloads. We note that the ill-conditioning can amplify numerical errors, so high precision in the initial PF solution is then very important.

SolverMean loading-ratio errorP95 error
N0N_04.23×10−54.23\times 10^{-5}1.04×10−41.04\times 10^{-4}
ADSD9.07×10−79.07\times 10^{-7}3.42×10−63.42\times 10^{-6}

ADSD reduces the mean loading-ratio error by approximately 47$\times$ and the P95 error by 30$\times$. We believe these results demonstrate a direct downstream benefit, as the improved PF solutions yield substantially more accurate branch-loading estimates for grid security assessment.

Apart from this task, PF solutions are also used in alternating current (AC) contingency analysis, security assessment, and operational planning. In these applications, PF solutions determine quantities such as bus voltages, branch flows, and constraint violations. Numerical errors and inaccuracies can affect the reliability of these quantities, particularly when operating conditions are close to security limits. Moreover, repeated PF evaluations are common in contingency and time-series analyses, where previous solutions serve as warm starts to reduce computation. Accurate intermediate solutions can therefore improve the numerical reliability of these workflows.

We have added more industrial examples in Appendix C to illustrate situations where low PF residuals are important.


W4: additional runtime on some cases

We agree that runtime improvements are not uniform across settings. We have revised our claim in the updated manuscript to state that ADSD improves accuracy with setting-dependent runtime trade-offs.


In the end, we thank you for your close reading and useful feedback. We hope that our responses have addressed your concerns, and we look forward to an open-minded discussion if any such concerns remain.

Sincerely,

Authors of Paper 186

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts, and some of my concerns have been addressed.

However, I remain unconvinced that the verification results establish the correctness of the underlying numerical diagnosis. I also feel that the practical significance of the reported error reductions is not yet sufficiently demonstrated.

Overall, while the rebuttal strengthens the paper, these concerns remain. I will therefore maintain my original score.

Reviewer_qcCJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how AI agents can make better number solvers. The authors say that code that runs can still be slow, or still be a weak method. They build a plan called ADSD. The model weights stay frozen.

ADSD has two steps. First, auto-diagnosis runs a frozen base solver N0 on public cases. It reads papers, builds tests, and keeps a guess only when that guess passes on more than 90% of a check pool. Second, skill discovery uses the kept guess to find a method, code it, and pack it as a skill. A fresh agent then builds a new solver from N0 plus those skills.

They test four tasks. These are power flow from PFΔ (Rivera et al., 2025), AC optimal power flow from OPFData (Lovett et al., 2024), stiff ODEs from Städter et al. (2021), and diffusion PDEs from ConDiff (Trifonov et al., 2024). The agents are GPT-5.5, Claude-Sonnet-5, and Qwen3.8-2.4T. The main example is Qwen on GOC-500. Table 1 says the mean power-flow error drops by about 71 times.

Critical strengths

The goal is clear and useful. A solver can finish with no crash and still be a poor numerical method. The three-step split is easy to follow: what the run shows, what is wrong in the numbers, and what code to add. Algorithm 1 and Figure 1 match that split.

Appendix B is the strongest part of the paper. For power flow, the authors show that N0 is a dense Newton solver. They show that the cutoff 0.16 was taken from the data (the lower quartile was about 0.16008), not copied from a book. They name the source of each skill: multistep factorization reuse from Tostado-Véliz et al. (2020a), and the seventh-order method from Tostado-Véliz et al. (2020b). For AC-OPF, they show a real stall. The stationarity residual stays near ‖∇f(x0)‖∞ = 3718.8979. The skill they keep is row scaling plus a scaled KKT score from Wächter and Biegler (2006), the Ipopt paper. A reader can check the story.

The data split is the right idea. Public cases are for building. Hidden cases are for the final score. They also use three model families, not one.

Figures 3 and 4 are useful. They ask the same model to rewrite N0, up to five times, with no new papers. That rewrite does not catch ADSD. Table 5 also helps. It splits the two power-flow skills. Multistep reuse is mostly about speed. Method 9 is mostly about the residual.

Critical weaknesses

1. The 71 times claim is too big for the evidence.

In Table 1, Qwen on GOC-500 goes from 1.0×10⁻⁹ to 1.4×10⁻¹¹. That ratio is about 71. The spread on the new error is ±2.4×10⁻¹⁰. The spread is about 17 times the mean. A few tiny cases can pull a mean down. GPT on IEEE-118 has the same pattern: 2.7×10⁻¹⁰ ± 9.9×10⁻¹⁰. Please also show the median.

Section 2 already says this “error” is the power mismatch, not the voltage error against a reference solution. Both 10⁻⁹ and 10⁻¹¹ are already far below a normal power-flow stop test. Appendix C cites Higham (2002), Moré and Wild (2012), and Molzahn and Lesieutre (2013) to say a tighter mismatch can help later grid checks. The paper never runs those later checks. So a reader cannot see if 71 times changes any engineering choice.

Other cells are much smaller. Claude on IEEE-118 goes from 1.1×10⁻⁹ to 7.4×10⁻¹⁰. That is about 1.5 times. The abstract leads with the best cell.

2. Speed does not always stay the same.

The abstract says ADSD keeps or improves speed. Table 6 says otherwise for GPT on GOC-500. On the unseen N1 grid, time goes from 0.38 s to 0.62 s. On the full N grid, time goes from 0.37 s to 0.60 s. The new solver is slower on those grids.

3. The power-flow base solver is dense. Real grid solvers are sparse.

Appendix B says each Newton step builds one dense factorization. Tinney and Hart (1967), already in the reference list, use sparse ordered elimination. MATPOWER (Zimmerman, Murillo-Sánchez, and Thomas, 2011) and pandapower (Thurner et al., 2018) also use sparse linear algebra. GOC-500 is a 500-bus grid. A dense base makes the start look weak.

Table 5 shows what the skills change. Factorizations go from 5.116 to 2.755 or 3.114. That win is “do fewer dense factorizations.” A sparse base can shrink this win. The paper has no sparse run.

4. The tables have no strong classic solver, and no close agent method.

On AC-OPF, the kept skill is a known Ipopt trick from Wächter and Biegler (2006). The same scaling idea is also in Nocedal and Wright (2006). Qwen’s pass rate jumps from 14.42% to 98.36% (Table 2). GPT only moves from 97.82% to 99.26%. The big jump fits a weak model that forgot a standard trick. A short prompt that names Ipopt scaling could do the same job. That control is missing. There is also no Ipopt, MATPOWER, or pandapower number on the same cases.

On stiff ODEs, the agent may use only Python and NumPy. Städter et al. (2021) already score strong integrators on these biology models. Hairer and Wanner (1996) is the standard book for BDF and Radau. Those methods are not in the tables. Claude’s base solver already passes 94.0%. Qwen’s already passes 98.3% (Table 3).

On the diffusion PDE, there is no algebraic multigrid run, and no ILU-GMRES run. Saad (2003) is the usual source for GMRES and preconditioners. ConDiff (Trifonov et al., 2024) is a high-contrast set, so those baselines matter. The closest agent papers are only in the text: SciML Agents (Gaonkar et al., 2025), CodePDE (Li et al., 2026), and AutoNumerics (Du et al., 2026). PDEAgent-Bench (Hang et al., 2026) and the PETSc agent study (Zhang et al., 2026) are also only in the related work. Guo et al. (2022), who learn ODE integrator coefficients, are in the same position.

5. The self-improve test does not isolate diagnosis.

The paper’s main claim is that diagnosis is the key step. Figures 3 and 4 only remove the whole ADSD plan. The rewrite arm cannot read the paper database. A gap there can come from reading papers alone. Appendix C says open search is not enough, but that part is one story, not a table.

This missing arm matters because the skill step overlaps older work. Voyager (Wang et al., 2024) already stores skills and reuses them. Reflexion (Shinn et al., 2023) and ExpeL (Zhao et al., 2024) already turn run logs into the next try. Paper2Agent (Miao et al., 2025), SkillFoundry (Shen et al., 2026c), and ResearchCodeAgent (Gandhi et al., 2025) already turn papers into code tools. AgentRx (Barke et al., 2026) already reads failure traces. Huang et al. (2025a) already test guesses against data. The new piece in ADSD is the checked numerical guess in the middle. The experiments never turn only that piece off.

FunSearch (Romera-Paredes et al., 2024) and AlphaEvolve (Novikov et al., 2025) search for better code with many tries. The intro says code search is a different job. There is still no search baseline with the same public score. ProOPF (Shen et al., 2026a) is very close to the AC-OPF section, and it is not a baseline. Cai et al. (2026) update weights with physics rewards. This paper does not update weights, but the title says “Training.” That word can mix up the two ideas.

The biggest AC-OPF jump is also outside the ablation. Section 4.5 and Figures 3 and 4 cover power flow and ODEs. They do not cover the Qwen pass-rate jump, and they do not cover the PDE.

6. The AC-OPF error skips the hard cases.

Table 2 computes mean error only on cases that N0 and ADSD both pass. Qwen fails most cases at the start. Cases that only the new solver finishes never enter the error. Those cases can look worse. The pass rate is the real result. The error line can look too kind. The paper also never states the pass rule: which limit gap, and which optimality gap.

7. The 90% rule can keep a weak common pattern and drop a rare real bug.

A guess is kept if it passes on more than 90% of pool V. V comes from the same public set as the discovery cases. The power-flow link that they keep is small: 5.78 Newton steps versus 5.01 steps. The cutoff was fit on those same 200 cases. The later check is still the same public pool. It is not the hidden set.

A bug that hits only a few cases will fail a 90% rule on all of V, even when the guess is right on those few cases. The paper also never says how many guesses were tried and thrown away. Many cutoffs make a pass more likely. Huang et al. (2025a) give a stricter test for a guess. This paper cites that work and then uses a flat 90% cutoff. The score itself is also unclear: accuracy on all of V, or precision only when the condition is true. In the AC-OPF case study, the authors report the subset. The main text says “90% of V.”

8. The “new regime” tests stay close to the training cases.

Tables 6 and 7 move from N2 to N1 to full N on the same IEEE-118 and GOC-500 grids. That puts one or two parts back. It is not a jump to a new grid family in PFΔ (Rivera et al., 2025). On the harder near-infeasible GOC-500 cases, GPT’s error drop is about 23% (Table 7: 1.1×10⁻⁹ to 8.5×10⁻¹⁰). The paper cites Canizares and Alvarado (1993) for cases near the voltage limit. The skills in Appendix B are higher-order Newton steps. They are not a continuation method for that limit.

For the PDE, train and test share the same 12 coefficient families. New draws from the same family are a weak shift. The abstract’s words “unseen grid topologies and operating regimes” are broader than these tests.

9. One agent run is not enough, and the build cost is missing.

The ± values are over test cases. They are not over new runs of the agent. A coding agent can be lucky once. The PDE repeats each solve three times. It does not repeat the full ADSD plan.

Appendix C says the power-flow search starts from 312 papers. The AC-OPF search uses 376 papers and 4,136 screening judgments. The tables report solver time only. Five rewrite loops are not the same budget as hundreds of paper reads. The paper list, the prompts, and the code are not provided. ODE and PDE paper-list sizes are not even stated.

Table 5 also shows a selection problem. Method 9 alone reaches 7.46×10⁻¹³. Both skills together reach only 2.76×10⁻¹⁰. The main GPT IEEE-118 result, 2.7×10⁻¹⁰, matches the weaker mix. The caption says the agent must use the listed skills. The plan can force a skill that hurts the error.

The “theory audit” is another LLM read of the same papers. It is not an outside check.

Questions for authors
  1. Can you report the median error, and the error at the 90th percentile, next to every mean in Tables 1, 6, and 7? For Qwen on GOC-500, how many cases are below 10⁻¹², and how many stay near 10⁻⁹?

  2. Can you score voltage error against the PFΔ reference solution, not only the power mismatch? Appendix C names later uses from Higham (2002), Moré and Wild (2012), and Molzahn and Lesieutre (2013). Can you run one of those uses, such as line-flow error or a finite-difference sensitivity, and show that 71 times changes the result?

  3. What code was banned as an “off-the-shelf solver”? Could the agent call SciPy sparse LU, KLU, or a Newton code like the one in Tinney and Hart (1967)? Please add a sparse MATPOWER or pandapower run on the same IEEE-118 and GOC-500 cases (Zimmerman, Murillo-Sánchez, and Thomas, 2011; Thurner et al., 2018).

  4. For AC-OPF, what is the exact pass rule? Please split Table 2 into three sets: cases both pass, cases only ADSD passes, and cases only N0 passes. Please also add Ipopt on the same 5,000 edge cases (Wächter and Biegler, 2006). As a small control, please prompt N0 once with “use the row scaling in Section 3.8 of Wächter and Biegler (2006)” and no other ADSD step.

  5. Please add one arm with the same paper database, the same read budget, and no diagnosis step. Please also add a code-search arm in the style of FunSearch or AlphaEvolve (Romera-Paredes et al., 2024; Novikov et al., 2025), with the public score as the target. For the ODE task, please add SciPy BDF and Radau under the same 10⁻⁴ rule used by Städter et al. (2021). For the PDE, please add one ILU-GMRES run and one algebraic multigrid run (Saad, 2003).

  6. How is “passed over 90% of V” computed? Please list every guess that was proposed, kept, or dropped. Please give the hit rate of each kept detector on the hidden set. Why can a rare but real failure mode survive this rule?

  7. Did the paper database contain the benchmark papers themselves, or papers that describe their recommended solvers? That list includes Rivera et al. (2025), Lovett et al. (2024), Städter et al. (2021), and Trifonov et al. (2024). Please give the snapshot date, the keyword list, and the ODE and PDE database sizes.

  8. Table 3 says “72 source families with 117 models.” The main text says 111 families, with 39 for training and 72 for testing. Which count is the held-out set? Were whole families held out, or single models from a family seen in training?

  9. How many full ADSD runs, with different seeds, support each bar? What was the token cost and the number of model calls for diagnosis, paper screening, and the final solve? Can you match that cost in the self-improve arm?

  10. In Table 5, method 9 alone has a much smaller residual than both skills together. Why must the final solver use both? For the ODE and the PDE, can you add a case study like Appendix B, and name the paper each skill came from?

References

  • Barke et al. (2026). AgentRx. arXiv:2602.02475.
  • Cai et al. (2026). Reinforcement learning with verifiable physics. arXiv:2607.10474.
  • Canizares and Alvarado (1993). Point of collapse and continuation methods. IEEE Trans. Power Systems.
  • Du et al. (2026). AutoNumerics. ICLR 2026 workshop on AI and PDEs.
  • Gandhi et al. (2025). ResearchCodeAgent. AAAI workshop.
  • Gaonkar et al. (2025). SciML agents. arXiv:2509.09936.
  • Guo et al. (2022). Personalized algorithm generation for ODE integrators. SIAM J. Sci. Comput.
  • Hairer and Wanner (1996). Solving Ordinary Differential Equations II. Springer.
  • Hang et al. (2026). PDEAgent-Bench. arXiv:2605.09636.
  • Higham (2002). Accuracy and Stability of Numerical Algorithms. SIAM.
  • Huang et al. (2025a). Automated hypothesis validation. ICML.
  • Li et al. (2026). CodePDE. TMLR.
  • Lovett et al. (2024). OPFData. arXiv:2406.07234.
  • Miao et al. (2025). Paper2Agent. arXiv:2509.06917.
  • Molzahn and Lesieutre (2013). Initializing dynamic power system simulations. IEEE TCAS I.
  • Moré and Wild (2012). Estimating derivatives of noisy simulations. ACM TOMS.
  • Nocedal and Wright (2006). Numerical Optimization. Springer.
  • Novikov et al. (2025). AlphaEvolve. arXiv:2506.13131.
  • Rivera et al. (2025). PFΔ. NeurIPS.
  • Romera-Paredes et al. (2024). FunSearch. Nature.
  • Saad (2003). Iterative Methods for Sparse Linear Systems. SIAM.
  • Shen et al. (2026a). ProOPF. ICML.
  • Shen et al. (2026c). SkillFoundry. COLM.
  • Shinn et al. (2023). Reflexion. NeurIPS.
  • Städter et al. (2021). Benchmarking ODE integrators for biological systems. Scientific Reports.
  • Thurner et al. (2018). pandapower. IEEE Trans. Power Systems.
  • Tinney and Hart (1967). Power flow solution by Newton’s method. IEEE Trans. Power Apparatus and Systems.
  • Tostado-Véliz, Kamel, and Jurado (2020a, 2020b). Higher-order power-flow methods. IEEE Systems Journal.
  • Trifonov et al. (2024). ConDiff. arXiv:2406.04709.
  • Wächter and Biegler (2006). On the implementation of an interior-point filter line-search algorithm. Mathematical Programming.
  • Wang et al. (2024). Voyager. TMLR.
  • Zhang et al. (2026). Agentic evaluation for AI-generated PETSc code. arXiv:2603.15976.
  • Zhao et al. (2024). ExpeL. AAAI.
  • Zimmerman, Murillo-Sánchez, and Thomas (2011). MATPOWER. IEEE Trans. Power Systems.
5 comments
AuthorsOpenReview ↗

Author response (1/4)

Thank you for providing the review. We answer each point below:


W1, Q1, Q2: Residual significance and downstream evaluation

We first answer Q2 and W1 regarding the downstream significance brought by the lower error. PF solutions are inputs to downstream engineering tasks, where their accuracy affects the physical quantities used for operational assessment. To demonstrate the downstream benefits of our PF improvements, we take an example from the 71×71\times improvement one and evaluate branch-loading assessment on 5000 GOC grids near the voltage-stability boundary, which uses PF solutions to compute branch-loading ratios and measures line utilization relative to capacity and support the assessment of overloads. We note that the ill-conditioning can amplify numerical errors, so high precision in the initial PF solution is then very important.

SolverMean loading-ratio errorP95 error
N0N_04.23×10−54.23\times 10^{-5}1.04×10−41.04\times 10^{-4}
ADSD9.07×10−79.07\times 10^{-7}3.42×10−63.42\times 10^{-6}

We believe these results demonstrate a direct downstream benefit, as the improved PF solutions yield substantially more accurate branch-loading estimates for grid security assessment.

Can you report the median error, and the error at the 90th percentile, next to every mean in Tables 1, 6, and 7?

We have added to the updated manuscript.

For Qwen on GOC-500, how many cases are below 10⁻¹², and how many stay near 10⁻⁹?

Fewer than 1% of cases have residuals in (8×10−10,10−9](8\times10^{-10},10^{-9}], while 97% lie in (10−12,10−10](10^{-12},10^{-10}].

Can you score voltage error against the PFΔ reference solution, not only the power mismatch?

Power-flow equations can admit multiple solutions, and PFDelta stores the particular solution obtained during data generation. A different solution satisfying the same equations can therefore receive a nonzero reference error without being numerically incorrect, which is exactly the reason why PFDelta explicitly adopts mismatch-based evaluation to avoid this bias toward the data-generation solver. Therefore, our choice of metric follows the benchmark's stated rationale, and we have added this discussion to the paper.


W2: Speed does not always stay the same

We agree that runtime improvements are not uniform across settings. We have revised our claim in the updated manuscript to state that ADSD improves accuracy with setting-dependent runtime trade-offs.


W3: dense & sparse solver

We clarify that the dense solver reflects the agent's initial implementation, which ADSD is tasked with diagnosing and improving. The discovered skills modify the nonlinear update strategy and reuse Jacobian factorizations across corrections, and these mechanisms are compatible with sparse linear algebra. Specifically, sparsity reduces the cost of each factorization, while the learned methods reduce how often numerical factorization is required. Although a sparse backend can change the magnitude of the runtime gains, we believe this doesn't establish that the improvements are artifacts of dense computation. Furthermore, Table 5 demonstrates that ADSD identifies a computational bottleneck and acquires numerical methods that address it.


W4 & Q3 & Q4: no strong classic solver

Our contribution is the agent's ability to diagnose numerical deficiencies and independently identify and implement appropriate methods on its own. Prompting the agent to "use Ipopt scaling" supplies the proposed remedy in advance, bypassing the central question our work addresses. Such a control tests whether the agent can implement an expert-selected method, and we respectively suggest that success would not demonstrate that the agent could diagnose the problem and identify that method on its own. Moreover, the improvements are not confined to weak baselines. For Qwen on ODEs, despite a baseline pass rate of 98.3%, ADSD reduces runtime by 33% while also lowering trajectory error. Mature numerical solvers would provide useful performance context.

For power engineering tasks, the agent was allowed to implement Newton methods, and for ODE and PDE, agent cannot use direct differential equation solver without any further respections. Again, we iterate that our work test whether ADSD improves an agent's ability to diagnose numerical deficiencies and revise its own solver. The relevant comparison starts from the same agent-generated solver and measures how much the agent improves it with and without ADSD under the same implementation constraints. Calling MATPOWER delegates the numerical work to a pre-existing package and bypasses the capability under study.

AuthorsOpenReview ↗

Author response (2/4)

W5 & Q5: self-improve test does not isolate diagnosis

We provide three key ablations to directly address your concern.

  • literature-guided improvement without diagnosis

We removed the full diganosis part, where the agent directly use the literature to improve their solvers, with the mean error and time over both IEEE-118 and GOC-500 cases reported below:

Error (IEEE-118)TimeError (GOC-500)Time
GPT 5.59.1×10−109.1\times 10^{-10}0.730.739.3×10−109.3\times 10^{-10}0.510.51
w literature-only1.2×10−91.2\times 10^{-9}0.660.666.2×10−96.2\times 10^{-9}0.480.48
w ADSD2.7×10−102.7\times 10^{-10}0.550.551.6×10−101.6\times 10^{-10}0.500.50
Qwen 3.89.9×10−109.9\times 10^{-10}0.570.571.0×10−91.0\times 10^{-9}0.180.18
w literature-only2.7×10−92.7\times 10^{-9}0.520.521.6×10−91.6\times 10^{-9}0.180.18
w ADSD6.5×10−106.5\times 10^{-10}0.540.541.4×10−111.4\times 10^{-11}0.180.18

For ODE task:

Pass rateMedian errorTime
GPT 5.570.1%9.39×10−69.39 \times 10^{-6}4.713
w literature-only60.7%1.32×10−51.32\times 10^{-5}4.412
w ADSD90.6%3.93×10−73.93 \times 10^{-7}3.451
Qwen 3.898.3%1.30×10−81.30\times 10^{-8}0.961
w literature-only95.7%1.56×10−81.56\times 10^{-8}0.792
w ADSD99.2%5.95×10−95.95\times 10^{-9}0.646

The biggest problem we found is that without diagnosis, agents tended to apply local superficial tricks from the literature but not addressing the underlying numerical issues, which leads to some runtime gains while increasing residuals. We believe this highlights the importance of diagnosis in guiding effective method selection.

  • diagnosis without literature retrieval

Next, we evaluate diagnosis without literature retrieval, i.e., the agent diagnoses and improves the solver using its inherent knowledge, without access to diagnostic skills and external literature.

Error (IEEE-118)TimeError (GOC-500)Time
GPT 5.59.1×10−109.1\times 10^{-10}0.730.739.3×10−109.3\times 10^{-10}0.510.51
w self-diagnosis-only ADSD8.8×10−108.8\times 10^{-10}0.720.729.0×10−109.0\times 10^{-10}0.500.50
w ADSD2.7×10−102.7\times 10^{-10}0.550.551.6×10−101.6\times 10^{-10}0.500.50
Qwen 3.89.9×10−109.9\times 10^{-10}0.570.571.0×10−91.0\times 10^{-9}0.180.18
w self-diagnosis-only ADSD1.1×10−91.1\times 10^{-9}0.570.578.2×10−108.2\times 10^{-10}0.180.18
w ADSD6.5×10−106.5\times 10^{-10}0.540.541.4×10−111.4\times 10^{-11}0.180.18

For ODE task:

Pass rateMedian errorTime
GPT 5.570.1%9.39×10−69.39\times 10^{-6}4.713
w self-diagnosis-only ADSD71.8%9.16×10−69.16\times 10^{-6}4.932
w ADSD90.6%3.93×10−73.93\times 10^{-7}3.451
Qwen 3.898.3%1.30×10−81.30\times 10^{-8}0.961
w self-diagnosis-only ADSD94.0%1.98×10−81.98\times 10^{-8}0.902
w ADSD99.2%5.95×10−95.95\times 10^{-9}0.646

Without literature support, agents rely on internal knowledge, which may be insufficient to distinguish competing numerical causes and identify suitable remedies. This can produce plausible diagnoses that do not translate into effective solver improvements.

  • ADSD without hypothesis verification

The final key component hasn't been ablated is the hypothesis verification (HV) part. We therefore remove verification and send all the hypothesis to later stage.

Error (IEEE-118)TimeError (GOC-500)Time
GPT 5.59.1×10−109.1\times 10^{-10}0.730.739.3×10−109.3\times 10^{-10}0.510.51
ADSD w/o HV2.7×10−102.7\times 10^{-10}0.550.559.8×10−109.8\times 10^{-10}0.510.51
ADSD2.7×10−102.7\times 10^{-10}0.550.551.6×10−101.6\times 10^{-10}0.500.50
Qwen 3.89.9×10−109.9\times 10^{-10}0.570.571.0×10−91.0\times 10^{-9}0.180.18
ADSD w/o HV7.2×10−107.2\times 10^{-10}0.560.561.4×10−111.4\times 10^{-11}0.180.18
ADSD6.5×10−106.5\times 10^{-10}0.540.541.4×10−111.4\times 10^{-11}0.180.18

For ODE task:

Pass rateMedian errorTime
GPT 5.570.1%9.39×10−69.39\times 10^{-6}4.713
ADSD w/o HV83.8%7.22×10−77.22\times 10^{-7}3.562
w ADSD90.6%3.93×10−73.93\times 10^{-7}3.451
Qwen 3.898.3%1.30×10−81.30\times 10^{-8}0.961
ADSD w/o HV99.2%5.95×10−95.95\times 10^{-9}0.646
w ADSD99.2%5.95×10−95.95\times 10^{-9}0.646

Hypothesis verification filters out unsupported hypotheses before they guide method selection. Removing HV leaves performance unchanged in some cases, but increases GPT's GOC-500 residual from 1.6×10−101.6\times10^{-10} to 9.8×10−109.8\times10^{-10} and reduces its ODE pass rate from 90.6% to 83.8%, which suggests that hypothesis verification helps make diagnosis-guided improvements more reliable.

AuthorsOpenReview ↗

Author response (3/4)

W6 & Q3: Qwen fails most cases at the start. Cases that only the new solver finishes never enter the error

We additionally evaluated all the cases solved only by ADSD (Of the 5000 cases, 721 are solved by both solvers, 4197 only by ADSD, and none only by N0N_0). Their mean absolute relative objective error is 1.13×10−81.13\times10^{-8}, with a maximum of 2.42×10−72.42\times10^{-7}, demonstrating accurate solutions on the newly successful cases. For Qwen, ADSD increases successful solves from 721 to 4918. The common-success comparison measures objective accuracy on identical cases, while the pass rate includes all 5000 cases.

The pass criteria are: power-balance mismatch and thermal-limit violations ≤10−6\le10^{-6} p.u., voltage/generator-bound violations ≤10−7\le10^{-7} p.u., angle violations ≤10−7\le10^{-7} rad, and max⁡(f−fref,0)≤max⁡(10−2,10−5max⁡(∣fref∣,1))\max(f-f_{\mathrm{ref}},0)\le \max\left(10^{-2},10^{-5}\max(|f_{\mathrm{ref}}|,1)\right).


W7 & Q6: The 90% rule can keep a weak common pattern and drop a rare real bug.

We provide further statistics. Across the evaluated hypotheses, rejected candidates achieved verification accuracies less 50%, whereas effective candidates exceeded 90%. This clear separation means that a threshold within the intervening gap would produce the same selection, so we believe the result is not sensitive to the exact cutoff. Furthermore, the score measures prediction accuracy, not how frequently the diagnosed condition occurs. A rare failure mode can therefore pass verification if its detector correctly identifies both affected and unaffected cases.

In terms of "guess" in your review, we interpret it as hypothesis. We provide an example of all hypothesis proposed in Qwen3.8 baseline in AC-OPF and their test over the validation pool:

HypothesisVerification pool passing rateResult
H1: Early stall termination explains many failures.99.5%passed to next stage
H2: Narrow reactive-power bounds restrict steps and increase residuals.94.5%passed to next stage
H3: Linear costs create weak curvature and poorly scaled steps.21.0%discarded
H4: Load-dependent residuals arise from unrepaired initialization mismatch.37.2%discarded
H5: Insufficient backtracking leaves larger residuals.6.8%discarded

For H1, the detector correctly identifies 832 non-convergent and 163 convergent cases, with one false positive and four false negatives. This yields 99.5% accuracy across all 1,000 verification cases, demonstrating accurate classification of both outcomes.


W8: The “new regime” tests stay close to the training cases. It is not a jump to a new grid family in PFΔ (Rivera et al., 2025).

As explicitly stated in Section 4.1, the IEEE and GOC benchmarks use different problem formulations and input–output schemas. A solver developed for IEEE-118 therefore cannot be directly applied to GOC-500 without adaptation. Our generalization tests instead keep the solver interface fixed while changing network topology and operating conditions, evaluating the frozen solver without further adaptation. This is precisely the transfer to "unseen grid topologies and operating regimes" claimed in our manuscript. For ODEs and PDEs, we followed established numerical benchmarks with held-out evaluation. The ODE evaluation tests unseen system families, while the PDE evaluation tests unseen coefficient and forcing realizations within the benchmark’s predefined configuration families. These experiments assess whether the generated numerical solvers remain effective on new instances, with the scope of each test explicitly defined.

Reviewer_DjFGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Auto-Diagnosis and Skill Discovery (ADSD), an agentic framework that diagnoses numerical solver failures, retrieves relevant numerical methods from scientific literature, and converts them into reusable solver skills. Experiments across power flow, optimal power flow, stiff ODEs, and elliptic PDEs demonstrate promising improvements in solver accuracy, robustness, and efficiency.

However, I have several concerns:

  1. Limited methodological novelty. The framework primarily integrates existing techniques, including execution-based diagnosis, literature retrieval, hypothesis verification, and skill generation. The individual components are not particularly novel, and the conceptual contribution beyond their integration is somewhat limited.

  2. Insufficient ablation studies. The experiments compare ADSD mainly against direct and iterative self-improvement. More informative comparisons are needed, particularly literature-guided improvement without diagnosis, diagnosis without literature retrieval, and individual components of the proposed pipeline.

  3. Questionable significance of some numerical improvements. In the power-flow experiments, baseline residuals are already around 10−910^{-9}. Further reducing these residuals may offer limited practical benefit, particularly without demonstrating improvements in downstream physical quantities or engineering decisions.

  4. Incomplete computational cost analysis. ADSD involves extensive literature screening, diagnosis, hypothesis verification, and skill construction. The reported solver runtimes do not capture these substantial development costs. A complete end-to-end cost comparison is necessary to establish the framework's efficiency.

  5. Limited reproducibility and generalization evidence. The framework depends on a preconstructed, domain-specific literature database and a complex agentic workflow. More implementation details, repeated independent agent runs, and evaluations on genuinely new numerical problem classes would strengthen the claims.

Overall, the paper addresses an interesting and practically relevant problem, and the empirical results are encouraging. However, the limited methodological novelty and insufficient experimental isolation of the key contributions weaken the case for acceptance.

Critical strengths

Practical relevance: The framework addresses an important challenge in autonomous numerical computing and demonstrates applicability across four problem domains.

Promising empirical results: The improvements in solver robustness, particularly for AC-OPF and stiff ODEs, demonstrate practical value.

Critical weaknesses

Limited methodological novelty. The framework mainly integrates existing techniques for diagnosis, literature retrieval, and skill generation, without substantial new algorithmic contributions.

Insufficient experimental validation. The ablation studies do not adequately isolate the contributions of individual components, and the substantial computational costs of the full pipeline are not evaluated.

Questions for authors

Can the authors clarify the methodological novelty of ADSD beyond integrating existing diagnosis, literature retrieval, and skill-generation techniques?

Can the authors provide ablations isolating the contributions of diagnosis and literature retrieval, together with an end-to-end computational cost comparison against the baselines?

Can the pipeline discover new algorithm?

4 comments
AuthorsOpenReview ↗

Author response (1/3)

We sincerely appreciate your time for providing these detailed and constructive feedbacks. We provide additional experiments and address each points below:


W1 & Q1. methodological novelty

Numerical solvers are notoriously difficult to improve because symptoms don't reveal their causes, i.e., the same failure can arise from different numerical conditions, each requiring a different remedy. We agree that the individual components may exist for general agentic tasks, but the biggest gap is that poor solver performance doesn't reveal which numerical method will help. ADSD addresses this by (1) converting numerical knowledge literature into executable diagnostic probes, (2) testing symptom condition hypotheses on held-out observations, and (3) using the supported hypotheses to guide method retrieval and construct solver-specific executable skills.

Our contribution is therefore to connect these components through verified numerical diagnosis, so that method selection targets the identified numerical condition. This reframes solver improvement as matching diagnosed numerical conditions to the assumptions and mechanisms of candidate methods, which we believe is the core conceptual novelty of ADSD.


W3: Questionable significance of some numerical improvements

Sorry for not making this clear, we add further background on why high accuracy matters.

Power flow (PF) solutions are often used in downstream computations, including alternating current (AC) contingency analysis, security assessment, and operational planning. In these applications, PF solutions determine quantities such as bus voltages, branch flows, and constraint violations. Numerical errors and inaccuracies can affect the reliability of these quantities, particularly when operating conditions are close to security limits. To demonstrate the downstream benefits of our PF improvements, we evaluate branch-loading assessment on 5000 GOC grids near the voltage-stability boundary, which uses PF solutions to compute branch-loading ratios and measures line utilization relative to capacity and support the assessment of overloads. We note that the ill-conditioning can amplify numerical errors, so high precision in the initial PF solution is then very important.

SolverMean loading-ratio errorP95 error
N0N_04.23×10−54.23\times 10^{-5}1.04×10−41.04\times 10^{-4}
ADSD9.07×10−79.07\times 10^{-7}3.42×10−63.42\times 10^{-6}

Moreover, repeated PF evaluations are common in contingency and time-series analyses, where previous solutions serve as warm starts to reduce computation. Accurate intermediate solutions can therefore improve the numerical reliability of these workflows.

We have added industrial examples in Appendix C to illustrate situations where low PF residuals are important.

AuthorsOpenReview ↗

Author response (2/3)

W2 & Q2. More ablations

Great point, thanks for bringing out these. We present additional experiments over PFDelta and ODE task with one closed-source (GPT) and one open-source (Qwen) baseline.

  • literature-guided improvement without diagnosis

We removed the full diganosis part, where the agent directly use the literature to improve their solvers, with the mean error and time over both IEEE-118 and GOC-500 cases reported below:

Error (IEEE-118)TimeError (GOC-500)Time
GPT 5.59.1×10−109.1\times 10^{-10}0.730.739.3×10−109.3\times 10^{-10}0.510.51
w literature-only1.2×10−91.2\times 10^{-9}0.660.666.2×10−96.2\times 10^{-9}0.480.48
w ADSD2.7×10−102.7\times 10^{-10}0.550.551.6×10−101.6\times 10^{-10}0.500.50
Qwen 3.89.9×10−109.9\times 10^{-10}0.570.571.0×10−91.0\times 10^{-9}0.180.18
w literature-only2.7×10−92.7\times 10^{-9}0.520.521.6×10−91.6\times 10^{-9}0.180.18
w ADSD6.5×10−106.5\times 10^{-10}0.540.541.4×10−111.4\times 10^{-11}0.180.18

For ODE task:

Pass rateMedian errorTime
GPT 5.570.1%9.39×10−69.39 \times 10^{-6}4.713
w literature-only60.7%1.32×10−51.32\times 10^{-5}4.412
w ADSD90.6%3.93×10−73.93 \times 10^{-7}3.451
Qwen 3.898.3%1.30×10−81.30\times 10^{-8}0.961
w literature-only95.7%1.56×10−81.56\times 10^{-8}0.792
w ADSD99.2%5.95×10−95.95\times 10^{-9}0.646

The biggest problem we found is that without diagnosis, agents tended to apply local superficial tricks from the literature but not addressing the underlying numerical issues, which leads to some runtime gains while increasing residuals. We believe this highlights the importance of diagnosis in guiding effective method selection.

  • diagnosis without literature retrieval

Next, we evaluate diagnosis without literature retrieval, i.e., the agent diagnoses and improves the solver using its inherent knowledge, without access to diagnostic skills and external literature.

Error (IEEE-118)TimeError (GOC-500)Time
GPT 5.59.1×10−109.1\times 10^{-10}0.730.739.3×10−109.3\times 10^{-10}0.510.51
w self-diagnosis-only ADSD8.8×10−108.8\times 10^{-10}0.720.729.0×10−109.0\times 10^{-10}0.500.50
w ADSD2.7×10−102.7\times 10^{-10}0.550.551.6×10−101.6\times 10^{-10}0.500.50
Qwen 3.89.9×10−109.9\times 10^{-10}0.570.571.0×10−91.0\times 10^{-9}0.180.18
w self-diagnosis-only ADSD1.1×10−91.1\times 10^{-9}0.570.578.2×10−108.2\times 10^{-10}0.180.18
w ADSD6.5×10−106.5\times 10^{-10}0.540.541.4×10−111.4\times 10^{-11}0.180.18

For ODE task:

Pass rateMedian errorTime
GPT 5.570.1%9.39×10−69.39\times 10^{-6}4.713
w self-diagnosis-only ADSD71.8%9.16×10−69.16\times 10^{-6}4.932
w ADSD90.6%3.93×10−73.93\times 10^{-7}3.451
Qwen 3.898.3%1.30×10−81.30\times 10^{-8}0.961
w self-diagnosis-only ADSD94.0%1.98×10−81.98\times 10^{-8}0.902
w ADSD99.2%5.95×10−95.95\times 10^{-9}0.646

Without literature support, agents rely on internal knowledge, which may be insufficient to distinguish competing numerical causes and identify suitable remedies. This can produce plausible diagnoses that do not translate into effective solver improvements.

  • ADSD without hypothesis verification

The final key component hasn't been ablated is the hypothesis verification (HV) part. We therefore remove verification and send all the hypothesis to later stage.

Error (IEEE-118)TimeError (GOC-500)Time
GPT 5.59.1×10−109.1\times 10^{-10}0.730.739.3×10−109.3\times 10^{-10}0.510.51
ADSD w/o HV2.7×10−102.7\times 10^{-10}0.550.559.8×10−109.8\times 10^{-10}0.510.51
ADSD2.7×10−102.7\times 10^{-10}0.550.551.6×10−101.6\times 10^{-10}0.500.50
Qwen 3.89.9×10−109.9\times 10^{-10}0.570.571.0×10−91.0\times 10^{-9}0.180.18
ADSD w/o HV7.2×10−107.2\times 10^{-10}0.560.561.4×10−111.4\times 10^{-11}0.180.18
ADSD6.5×10−106.5\times 10^{-10}0.540.541.4×10−111.4\times 10^{-11}0.180.18

For ODE task:

Pass rateMedian errorTime
GPT 5.570.1%9.39×10−69.39\times 10^{-6}4.713
ADSD w/o HV83.8%7.22×10−77.22\times 10^{-7}3.562
w ADSD90.6%3.93×10−73.93\times 10^{-7}3.451
Qwen 3.898.3%1.30×10−81.30\times 10^{-8}0.961
ADSD w/o HV99.2%5.95×10−95.95\times 10^{-9}0.646
w ADSD99.2%5.95×10−95.95\times 10^{-9}0.646

Hypothesis verification filters out unsupported hypotheses before they guide method selection. Removing HV leaves performance unchanged in some cases, but increases GPT's GOC-500 residual from 1.6×10−101.6\times10^{-10} to 9.8×10−109.8\times10^{-10} and reduces its ODE pass rate from 90.6% to 83.8%, which suggests that hypothesis verification helps make diagnosis-guided improvements more reliable.

We defer the discussion of cost to W4.

AuthorsOpenReview ↗

Author response (3/3)

W4 & Q2: Cost analysis

We agree. Recall in lines 196–197, we screen papers using preprocessed previews and reserve full-text reading for shortlisted papers to reduce token usage. Taking GOC-500 as an example, the total API costs per development run in USD are:

GPTQwen
Literature-guided improvement without diagnosisUS$52US$24
Diagnosis without literature retrievalUS$56US$26
ADSD without hypothesis verificationUS$66US$31
Full ADSDUS$70US$33

We note that removing hypothesis verification yields only small net savings because more hypotheses proceed to downstream processing. Overall, we believe the observed accuracy gains justify the additional development cost, particularly when the resulting solvers are reused across many problem instances.


W5: Reproducibility and generalization evidence

We conducted 3 independent agent runs for each combination of GPT & Qwen over IEEE-118 & ODE. Final solver performance varied by less than 7%, with mostly same-level of performance improvement comparing to baseline and with minor differences in skill implementation. After examine those trajectories, we find that those runs identified the same broad numerical issues and improvement strategies.

Regarding generalization, we clarify that our experiment in PFDelta includes explicit generalization tests from N2 to unseen grid topologies and from feasible to close-to-infeasible operating regimes. We also plan to release the product after publication.


Q3: Discover new algorithm

We consider inventing new numerical algorithms not present in the literature from discover algorithms that are new to the agent. Because the discovered algorithms are absent from its initial solver, we also discover reliable combinations of algorithms. In other words, ADSD instead addresses a practical gap: even when suitable numerical methods already exist, agents often fail to identify and implement them for a specific numerical deficiency.


In the end, we appreciate your time in providing the detailed feedback, which strengthened our empirical significance. We hope that our responses have addressed your concerns, and we look forward to an open-minded discussion if any such concerns remain.

Sincerely,

Authors of Paper 186

Thank you for the detailed rebuttal and additional experiments. The new ablations, computational cost analysis, and repeated-run results address several of my concerns. In particular, the comparisons with literature-only improvement and diagnosis-only improvement provide useful evidence for the value of the complete pipeline. I also appreciate the clarification that the objective is to identify and adapt existing numerical methods rather than invent new ones.

Nevertheless, my main concern about methodological novelty remains. The framework integrates established techniques for diagnosis, literature retrieval, hypothesis verification, and skill generation. Although the experiments demonstrate the practical benefits of this integration, the underlying methodological contribution remains relatively modest.

I also remain somewhat concerned about the practical significance of reducing already-small power-flow residuals. The proposed downstream evaluation is relevant, but the numerical results are missing from the provided response. Furthermore, the reported API costs do not fully characterize the end-to-end computational effort or establish the cost-effectiveness of the approach over repeated solver use.

Overall, the rebuttal strengthens the empirical evaluation and resolves some of my concerns. However, I do not think it sufficiently changes the assessment of methodological novelty to justify changing my recommendation at this stage.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thanks for your time: author response & updated manuscript

Dear reviewers,

Thank you all for your time in providing the feedback. We have provided detailed responses and updated the manuscript, with major changes in purple. We again appreciate your time to check our response.


Sincerely,

Authors of Paper 186

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.