acceptodds
Under review as a conference paper at ICLR 2027

PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation

Abstract

PDE-to-solver code generation aims to automatically synthesize executable numerical solvers from partial differential equation (PDE) specifications. This task requires not only understanding the mathematical structure of PDEs, but also selecting appropriate discretization schemes and solver configurations, and correctly implementing the resulting formulations in finite-element method (FEM) libraries. Existing code generation benchmarks mainly evaluate syntactic correctness, or success on predefined test cases. To our knowledge, there is currently no publicly available benchmark specifically for PDE-to-solver code generation, and general-purpose code benchmarks do not fully capture the unique challenges of numerical PDE solution, such as ensuring solver accuracy, efficiency, and compatibility with professional FEM libraries. We introduce PDEAgent-Bench, a multi-metric, multi-library benchmark for PDE-to-solver code generation. PDEAgent-Bench contains 645 instances across 6 mathematical categories and 11 PDE families, with common FEM libraries for DOLFINx, Firedrake, and deal.II. Each instance provides an agent-facing problem specification, a reference solution on a prescribed evaluation grid, and case-specific accuracy and runtime targets. PDEAgent-Bench adopts a staged evaluation framework in which generated solvers must sequentially pass executability, numerical accuracy, and computational efficiency checks. Experiments with representative LLMs and code agents show that models can often produce runnable code, but their pass rate drops substantially once accuracy and efficiency requirements are enforced. These results indicate that current agents remain limited in producing numerically reliable and efficient PDE solvers, and that PDEAgent-Bench provides a reproducible testbed grounded in the practical requirements of numerical PDE solving. Dataset, scripts, and codes are publicly released.

Primary area:
datasets and benchmarks
Keywords:
PDE-to-solver code generation, scientific computing, code agents, numerical PDEs, benchmark, multi-metric evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fuevOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces PDEAgent-Bench, a benchmark for evaluating LLM-generated PDE solvers across 645 instances, 11 PDE families, and three FEM libraries. It evaluates generated code through three stages: executability, numerical accuracy, and computational efficiency, with stage-level metrics to diagnose failures. Experiments show that runnable LLM-generated code often fails to meet accuracy or runtime requirements, with performance varying across libraries and PDE families. The results also examine iterative feedback and template guidance, highlighting the gap between executable code and reliable numerical solvers.

Critical strengths
  1. Practical and important evaluation gap: The paper addresses a key limitation of general-purpose code benchmarks by evaluating numerical accuracy and computational efficiency in addition to executability. This is important because runnable code does not necessarily constitute a reliable numerical solver.

  2. Comprehensive and reproducible benchmark: The 645 instances across 11 PDE families and three FEM libraries provide a broad testbed for systematic comparisons. The staged evaluation and publicly released resources support reproducibility and enable future research on reliable PDE solver generation.

Critical weaknesses
  1. The main weaknesses concern the paper's structure and readability. The abstract exceeds the one-paragraph format typically used in ICLR papers. Additionally, lines 184–187 discuss relative and absolute L2 errors, followed by Equation (1) without an explicit reference to it in the preceding text. Referencing Equation (1) when introducing the relative L2 error would improve clarity. Finally, the labels “F-Exec,” “F-Acc,” “F-Time,” and “Pass” at the top of Figure 4 are quite small and could be enlarged to improve readability.

  2. Since Figures 3, 4, and 5 show different results across FEM libraries, how easy would it be to extend the benchmark to support new libraries? Is adding a library relatively straightforward, or does it require substantial changes to the benchmark?

Questions for authors

See "critical weaknesses" above.

Reviewer_gN74OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces PDEAgent-Bench, consisting of 645 PDE simulation tasks. It evaluates the ability of LLMs and coding agents to generate numerical solvers using three open-source finite element libraries: DOLFINx, Firedrake, and deal.II. The generated implementations are evaluated in terms of executability, numerical accuracy, and computational efficiency.

Critical strengths

I could not identify significant strengths in the current experimental design.

Critical weaknesses
  1. Presentation and benchmark description

The manuscript is extremely long, yet many of the essential details are relegated to the appendix. The main paper provides insufficient information about the benchmark itself, making it difficult to understand and evaluate the experimental methodology.

In particular, the selection of problems, geometries, discretizations, and evaluation criteria should be explained much more clearly in the main paper.

The manuscript also appears to overstate its coverage of 3D problems. From the geometric templates listed in the appendix, I could identify only one 3D domain: the unit cube. This is a very limited representation of three-dimensional numerical simulation.

  1. Benchmark design

The motivation behind the selection of benchmark problems is unclear.

For example, what numerical capabilities are being tested by solving the heat equation on a square rather than a circle? Similarly, what specific differences are the authors expecting to observe between elliptic, parabolic, and hyperbolic problems?

These PDE classes have fundamentally different numerical properties, but the manuscript does not establish how these properties are reflected in the benchmark or how they affect the evaluation.

Furthermore, many of the problems appear to rely on manufactured solutions and relatively simple geometries and boundary conditions. It is unclear whether the benchmark includes sufficient numerical complexity to evaluate meaningful solver-generation capabilities.

The authors should explain which numerical challenges each category of problems is designed to test and why the selected examples are representative.

  1. Numerical accuracy and evaluation methodology

I have substantial concerns about the use of L2 error as an evaluation criterion.

The numerical error of a finite element solution depends on several factors, including mesh resolution, polynomial degree, discretization choices, and solution regularity. A relatively large L2 error does not necessarily indicate an incorrect implementation; it may be entirely reasonable for the chosen discretization.

Conversely, a generated implementation may contain errors while still satisfying the prescribed accuracy threshold.

How does the benchmark distinguish between an incorrect implementation and a correct implementation with insufficient spatial or temporal resolution?

Relatedly, how are the meshes generated? Are they prescribed by the benchmark, or are the agents responsible for selecting the mesh and discretization parameters? This distinction is essential because mesh resolution directly affects both numerical accuracy and computational efficiency.

More generally, the manuscript does not sufficiently justify why its accuracy and runtime thresholds provide meaningful measures of numerical solver quality.

  1. Interpretation of experimental failures

The experimental results raise several questions that are not adequately investigated.

For example, how can LLMs fail on such a large fraction of elementary problems, such as Poisson’s equation or the heat equation on simple domains?

These failures deserve a much more detailed analysis.

The paper distinguishes between execution, accuracy, and runtime failures, but these categories do not explain the underlying causes.

An unsuccessful run could result from incorrect code, an incorrect weak formulation, incorrectly imposed boundary conditions, an unsuitable discretization, an excessively coarse mesh, or an inappropriate solver configuration.

These are fundamentally different problems, and they should not be interpreted as equivalent failures of numerical reasoning.

The manuscript primarily reports aggregate success and failure rates without adequately characterizing the reasons behind the failures. As a result, it is difficult to understand what the experiments actually demonstrate about the capabilities and limitations of the evaluated models.

  1. Hidden evaluation objectives and iterative improvement

The benchmark uses hidden accuracy and runtime targets, but it is unclear how an agent is expected to improve its numerical solution without knowing the objectives it is trying to satisfy.

For example, choosing whether to refine the mesh, increase the polynomial degree, or change the solver configuration requires information about the desired accuracy and computational constraints.

Hiding the reference solution is reasonable, but hiding the evaluation objectives raises questions about what numerical decisions the agent is expected to make.

The authors should clarify what information is available to the agents during iterative improvement and explain how the proposed evaluation setup supports meaningful numerical optimization.

Overall assessment

My main concern is that the benchmark does not establish a clear connection between the selected problems, the evaluation metrics, and the numerical capabilities it claims to assess.

The paper presents a large number of experiments, but the experimental design and analysis are insufficient to determine whether the observed failures arise from code generation, familiarity with the FEM libraries, numerical discretization, or solver configuration.

Without a clearer justification of the benchmark construction and a more detailed investigation of the observed failures, I find it difficult to draw meaningful conclusions from the reported results.

Questions for authors

Please see the weaknesses section for my detailed questions and concerns regarding the benchmark design, numerical evaluation methodology, and interpretation of the experimental results.

Reviewer_or4qOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces the benchmark for LLM agent which helps to evaluate "executability, numerical accuracy, and computational efficiency checks in a controlled environment". Despite the previous benchmarks, PDEAgent-bench is a multi-metric and multi-library.

Critical strengths

The benchmark evaluates numerical accuracy and computational efficiency rather than executability alone, which is the appropriate standard for scientific computing. The central finding that a substantial amount of runnable solvers are either numerically inaccurate or exceed practical runtime budgets is consistent with this design choice and is a useful result in its own right. The three stage decomposition into execution, accuracy, and runtime failures is an improvement over a single pass/fail signal. It localizes where a model fails rather than merely reporting that it failed, and the resulting per-stage profiles (weaker models bottlenecked at execution, stronger models at accuracy and efficiency) are more informative for diagnosing model capability than an aggregate pass rate.

Critical weaknesses

1.- Possible exposure of the reference solution through boundary data. In all four walkthroughs of Appendix N, the Dirichlet boundary condition supplied to the agent appears to be the manufactured solution itself in closed form (Case B, where the BC is sin(2pix)sin(2pi*y) and the manufactured solution is the same. Case D specifies the same expression on all_boundaries). Since 536 of 645 instances are manufactured-solution cases, this raises the possibility that a submission could evaluate the boundary expression on the evaluation grid, apply the prescribed mask, and satisfy both the accuracy and runtime gates without solving the PDE. This would conflict with the claim in Section 3.2 that the agent-facing specification “does not expose the target solution field.” I may be misreading the relationship between the appendix excerpts and the serialized prompt, but as presented the manuscript does not rule this out.

2.- The accuracy gate measures agreement with a hidden calibration solver rather than solver correctness. The threshold is defined as ten times the calibration-baseline error with a fixed floor, and the runtime threshold as three times the calibration runtime. Neither of this is disclosed to the agent. The paper’s own walkthroughs shows the consequence, Case C is numerically accurate (1.7e-7 against a 5.9e-6 threshold) but fails on runtime at 7.5 s against a 4.8 s budget. Case D fails at 1.3e-6 against a floor-controlled 1e-6 threshold. The F-Acc category therefore mixes incorrect variational formulations with correct, but marginally underresolved discretizations, and F-Time is sensitive to resolution choices the agent must guess. This weakens the paper’s central claims about where models fail. Reporting the distribution of relative errors among F-Acc cases, or separating gross errors from near-threshold misses, would substantially clarify the analysis.

Questions for authors

Can you confirm that the manufactured solution is never recoverable from the agent-facing specification, and quantify how many of the 536 manufactured-solution instances supply a closed-form Dirichlet expression that coincides with the reference field? Relatedly, what would a trivial baseline that simply evaluates the boundary expression on the evaluation grid score on the DOLFINx track?

Appendix J presents the boundary condition as a redacted placeholder (g(x,y)), whereas Appendix N reports the full closed-form expression in the agent-visible excerpt. Which reflects the prompt actually serialized during evaluation, and what accounts for the discrepancy?

Among F-Acc cases, what is the distribution of relative L^2 errors? Specifically, what fraction fall within a small factor of the threshold, as opposed to failing by orders of magnitude?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.