- Presentation and benchmark description
The manuscript is extremely long, yet many of the essential details are relegated to the appendix. The main paper provides insufficient information about the benchmark itself, making it difficult to understand and evaluate the experimental methodology.
In particular, the selection of problems, geometries, discretizations, and evaluation criteria should be explained much more clearly in the main paper.
The manuscript also appears to overstate its coverage of 3D problems. From the geometric templates listed in the appendix, I could identify only one 3D domain: the unit cube. This is a very limited representation of three-dimensional numerical simulation.
- Benchmark design
The motivation behind the selection of benchmark problems is unclear.
For example, what numerical capabilities are being tested by solving the heat equation on a square rather than a circle? Similarly, what specific differences are the authors expecting to observe between elliptic, parabolic, and hyperbolic problems?
These PDE classes have fundamentally different numerical properties, but the manuscript does not establish how these properties are reflected in the benchmark or how they affect the evaluation.
Furthermore, many of the problems appear to rely on manufactured solutions and relatively simple geometries and boundary conditions. It is unclear whether the benchmark includes sufficient numerical complexity to evaluate meaningful solver-generation capabilities.
The authors should explain which numerical challenges each category of problems is designed to test and why the selected examples are representative.
- Numerical accuracy and evaluation methodology
I have substantial concerns about the use of L2 error as an evaluation criterion.
The numerical error of a finite element solution depends on several factors, including mesh resolution, polynomial degree, discretization choices, and solution regularity. A relatively large L2 error does not necessarily indicate an incorrect implementation; it may be entirely reasonable for the chosen discretization.
Conversely, a generated implementation may contain errors while still satisfying the prescribed accuracy threshold.
How does the benchmark distinguish between an incorrect implementation and a correct implementation with insufficient spatial or temporal resolution?
Relatedly, how are the meshes generated? Are they prescribed by the benchmark, or are the agents responsible for selecting the mesh and discretization parameters? This distinction is essential because mesh resolution directly affects both numerical accuracy and computational efficiency.
More generally, the manuscript does not sufficiently justify why its accuracy and runtime thresholds provide meaningful measures of numerical solver quality.
- Interpretation of experimental failures
The experimental results raise several questions that are not adequately investigated.
For example, how can LLMs fail on such a large fraction of elementary problems, such as Poisson’s equation or the heat equation on simple domains?
These failures deserve a much more detailed analysis.
The paper distinguishes between execution, accuracy, and runtime failures, but these categories do not explain the underlying causes.
An unsuccessful run could result from incorrect code, an incorrect weak formulation, incorrectly imposed boundary conditions, an unsuitable discretization, an excessively coarse mesh, or an inappropriate solver configuration.
These are fundamentally different problems, and they should not be interpreted as equivalent failures of numerical reasoning.
The manuscript primarily reports aggregate success and failure rates without adequately characterizing the reasons behind the failures. As a result, it is difficult to understand what the experiments actually demonstrate about the capabilities and limitations of the evaluated models.
- Hidden evaluation objectives and iterative improvement
The benchmark uses hidden accuracy and runtime targets, but it is unclear how an agent is expected to improve its numerical solution without knowing the objectives it is trying to satisfy.
For example, choosing whether to refine the mesh, increase the polynomial degree, or change the solver configuration requires information about the desired accuracy and computational constraints.
Hiding the reference solution is reasonable, but hiding the evaluation objectives raises questions about what numerical decisions the agent is expected to make.
The authors should clarify what information is available to the agents during iterative improvement and explain how the proposed evaluation setup supports meaningful numerical optimization.
Overall assessment
My main concern is that the benchmark does not establish a clear connection between the selected problems, the evaluation metrics, and the numerical capabilities it claims to assess.
The paper presents a large number of experiments, but the experimental design and analysis are insufficient to determine whether the observed failures arise from code generation, familiarity with the FEM libraries, numerical discretization, or solver configuration.
Without a clearer justification of the benchmark construction and a more detailed investigation of the observed failures, I find it difficult to draw meaningful conclusions from the reported results.