acceptodds
Under review as a conference paper at ICLR 2027

AutoPDEBench: Benchmarking LLM Auto-Research for Neural PDE Solver Design

Abstract

Partial differential equations (PDEs) are essential for modeling complex physical systems, and neural solvers have recently emerged as powerful data-driven tools for numerically solving them. However, existing neural solvers struggle with domain-specific challenges, such as varying parameters and high-speed flows, necessitating specialized architectures. Manually designing these specialized solver architectures is a highly iterative, time-consuming process requiring deep expertise, creating a significant bottleneck in scientific discovery. We propose leveraging autonomous AI research agents to automate the synthesis of specialized solvers. To support this, we introduce AutoPDEBench, a benchmark dedicated to LLM-driven automated research for PDE solver design. The benchmark includes 25 challenging datasets featuring both novel and actively studied physical scenarios. We evaluate a suite of general-purpose models (transformer, ROM, and graph-based) alongside a multi-agent instantiation of the iterative automated research pipeline, which serves as an agentic baseline. Empirical results show that the iterative automated research system significantly outperforms the general-purpose neural solver baselines. Our findings demonstrate the viability of using AI agents to automatically design neural solvers for complex physical systems. AutoPDEBench provides a foundational testbed to accelerate agent-driven scientific discovery in physics and engineering.

Primary area:
datasets and benchmarks
Keywords:
LLM Auto Research, Physics Informed Deep Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kjViOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces 25 datasets for evaluating agents that design neural PDE solvers. The tasks cover different discretizations, physical parameters, partial observations, long rollouts, and other modeling difficulties. A planner, coder, and debugger propose a model, training loss, and configuration, using validation feedback and a GPU memory constraint. The default search runs for five iterations. The resulting solvers are compared with five general-purpose architectures, with additional specialist comparisons and architecture-transfer experiments on three scenarios.

Critical strengths
  1. The 25 datasets cover more than small changes to one equation. Variable particle counts, mixed representations, external forcing, and long autoregressive prediction pose different engineering problems. The detailed modeling objectives in Appendix I make this a potentially valuable testbed for scientific coding agents.

  2. I appreciate the additional physical metrics, the specialist comparisons in Section 4.2, and the retraining of transferred architectures in Section 4.3. These checks are more informative than only beating a generic solver on its original dataset.

Critical weaknesses
  1. The paper does not give enough detail on baseline tuning, input conditioning, training schedules, or the winning agent designs to tell whether the gains come from scientific design or basic adaptation and additional search.

  2. The paper states that the training, validation, and test sets do not overlap, but it does not clearly explain how the trajectories are split or how many samples are in each set. Therefore, it is unclear whether very similar time windows appear across different splits. In MixedParam, the training Reynolds numbers are spaced by 100, while the validation and test numbers are spaced by 25, all within the same range. This means some parameter values are shared between training and testing.

Questions for authors

Q1. For each method in the main results table, how many GPUs and how much GPU memory were used? How long was the training, how much tuning was done, and what was the total GPU time? Also, the paper states that there is no time limit, so why were some methods excluded for taking too long? What were the criteria?

Q2. Do all baselines use the same additional inputs as the agent-designed methods, such as PDE parameters, time coordinates, and masks? If the baselines are given the same tuning and adaptation budget, can they achieve similar improvements?

Q3. How exactly are trajectories, initial conditions, and time windows split into training, validation, and test sets?

2 comments

We thank you for your insightful comments on improving our paper. We hope the following could address your concerns.

Q1 Computational Budget

We apologize for the confusion regarding the statement in Section 3.1. When we noted that we place “no restrictions on training or inference time, FLOPs, or parameter count,” we meant these are not enforced as explicit numerical constraints. However, the experimental environment is strictly controlled to ensure fair comparisons. Every general-purpose and agent-designed solver is trained on the same single GPU with the same memory budget for 100 epochs. The training loop and the 100-epoch limit are hardcoded in our agent harness and cannot be changed by the agent. The shared memory budget bounds the model size and batch size, which in turn bounds the compute of each training step. Because the number of epochs is fixed and the cost of each epoch is bounded, the total training compute of every solver is bounded as well. This protocol reflects the practical setting of neural solver design, where researchers develop models on a given GPU and compare them under a common number of epochs. Empirically, the agent-designed solvers never use more training time than the general-purpose baselines, as we report below.

Methods are excluded from Table 2 for one of two reasons. AMG is excluded on some large three-dimensional and high-resolution datasets because its training would take more than a month, which is unacceptable in practice. This is the only exclusion due to training time. All other exclusions arise because a model inherently cannot handle the task, such as differing numbers of input and output particles in LagInletOutlet or five-dimensional data in HighDimensional.

TaskGPUGNOTTransolverTransolver++LNOAMGAgent
1_HighSpeedFlowNvidia Titan XP 12GB10.0911.9111.982.7827.283.33
2_RealDataNvidia V100 16GB5.155.636.132.786.171.45
3_MixedParamNvidia V100 16GB13.6115.2822.044.8425.095.08
4_HeterogeneousNvidia V100 16GB5.686.616.252.065.665.21
5_PartialNvidia Titan XP 12GB9.7511.5811.544.1626.434.02
6_MixedDomainNvidia V100 16GB4.388.985.911.4617.728.51
7_MultiPhysicsNvidia V100 16GB45.7136.5239.8316.2178.9117.85
8_3DNvidia A6000 48GB37.7649.3256.2313.81–13.76
9_IrregularTimeNvidia V100 16GB9.7511.6211.592.7426.294.60
10_LagDynamicNvidia V100 16GB–––6.89–6.49
11_LagFSINvidia V100 16GB8.048.766.433.568.887.71
12_ExtremeNvidia L40S 48GB35.4541.8247.049.68–9.65
13_MixedResNvidia A6000 48GB11.2610.5611.841.59–5.47
14_ExtLag1Nvidia A100 40GB11.5012.2111.812.796.297.93
15_ExtLag2Nvidia V100 16GB4.274.694.621.513.304.46
16_MixedDimNvidia A6000 48GB15.4520.4322.175.18–12.62
17_MixedBCNvidia V100 16GB8.6110.1810.183.6523.122.24
18_LongHorizonNvidia V100 16GB2.284.443.0410.316.442.06
19_StochasticNvidia RTX 4090 24GB47.2339.2038.0918.9077.8320.46
20_MultiPhaseNvidia V100 16GB7.5310.678.992.3625.057.85
21_MixedPDENvidia Titan XP 12GB11.0213.0014.164.6329.624.27
22_LargeLagNvidia H200 141GB8.8614.0912.221.386.123.51
23_Sim2RealNvidia RTX 4090 24GB12.4914.7814.673.53133.974.42
24_HighDimensionalNvidia H200 141GB–––––88.61
25_ComplexGeometryNvidia L40S 48GB5.854.316.191.4757.631.87

W1 Q2 Adaptation of Known Methods

The default implementations of the baseline models only accept initial conditions and simply cannot process additional information such as boundary conditions or governing PDEs. This structural limitation is the exact motivation for AutoPDEBench. Instead of relying on human researchers to manually engineer new neural architectures that can handle these domain-specific challenges, we propose leveraging automated research agents to design them.

To test whether adapting known methods is sufficient, we add three agent baselines that start from an existing general-purpose solver and make small adaptations to it for the domain-specific challenge of each dataset. They start from Transolver for transformer-based models, LNO for reduced-order models, and AMG for graph-based models. Below, we report their results. These adaptations improve over the original models only marginally and only on some datasets, so solving these datasets requires designs beyond small adaptations of known methods. As a benchmark, We will include these adaptation agents as additional baselines.

W1 Q2 Adaptation of Known Methods (Continued)

TaskMetricTransolveragent_transolverLNOagent_lnoAMGagent_amgAgent
1_HighSpeedFlowRMSE3.232×1013.232 \times 10^{1}3.077×1013.077 \times 10^{1}4.482×1014.482 \times 10^{1}4.115×1014.115 \times 10^{1}2.819×1012.819 \times 10^{1}2.410×1012.410 \times 10^{1}5.066×100\mathbf{5.066 \times 10^{0}}
FRMSE2.372×1032.372 \times 10^{3}2.260×1032.260 \times 10^{3}3.282×1033.282 \times 10^{3}2.985×1032.985 \times 10^{3}2.181×1032.181 \times 10^{3}1.809×1031.809 \times 10^{3}4.218×102\mathbf{4.218 \times 10^{2}}
Kinetic Energy3.067×1033.067 \times 10^{3}2.691×1032.691 \times 10^{3}6.092×1036.092 \times 10^{3}5.475×1035.475 \times 10^{3}3.255×1033.255 \times 10^{3}2.434×1032.434 \times 10^{3}3.514×102\mathbf{3.514 \times 10^{2}}
3_MixedParamRMSE4.359×1004.359 \times 10^{0}4.370×1004.370 \times 10^{0}4.346×1004.346 \times 10^{0}4.376×1004.376 \times 10^{0}3.128×1003.128 \times 10^{0}3.216×1003.216 \times 10^{0}1.345×100\mathbf{1.345 \times 10^{0}}
FRMSE3.514×1023.514 \times 10^{2}3.488×1023.488 \times 10^{2}3.553×1023.553 \times 10^{2}3.489×1023.489 \times 10^{2}2.688×1022.688 \times 10^{2}2.726×1022.726 \times 10^{2}1.078×102\mathbf{1.078 \times 10^{2}}
Kinetic Energy3.593×1003.593 \times 10^{0}3.561×1003.561 \times 10^{0}3.605×1003.605 \times 10^{0}3.514×1003.514 \times 10^{0}1.741×1001.741 \times 10^{0}1.788×1001.788 \times 10^{0}2.780×10−1\mathbf{2.780 \times 10^{-1}}
10_LagDynamicRMSE–5.843×10−15.843 \times 10^{-1}6.105×10−16.105 \times 10^{-1}6.106×10−16.106 \times 10^{-1}–5.993×10−15.993 \times 10^{-1}5.530×10−1\mathbf{5.530 \times 10^{-1}}
LagKE–2.737×1032.737 \times 10^{3}1.349×103\mathbf{1.349 \times 10^{3}}3.237×1033.237 \times 10^{3}–2.229×1032.229 \times 10^{3}1.662×1031.662 \times 10^{3}
EuARMSE–1.107×10−11.107 \times 10^{-1}1.129×10−11.129 \times 10^{-1}1.123×10−11.123 \times 10^{-1}–1.117×10−11.117 \times 10^{-1}9.951×10−2\mathbf{9.951 \times 10^{-2}}
EuAFRMSE–2.560×1002.560 \times 10^{0}2.621×1002.621 \times 10^{0}2.605×1002.605 \times 10^{0}–2.574×1002.574 \times 10^{0}2.299×100\mathbf{2.299 \times 10^{0}}

Q3 Dataset Split

All datasets are split before any time windows are extracted, so no input and target pair appears in more than one of the training, validation, and test sets. For datasets with multiple simulations, each trajectory, or each sample for steady-state data, is assigned entirely to one split, and time windows are extracted only within the trajectories of that split. For datasets that consist of a single long simulation, namely MixedDomain, LagInletOutlet, and LargeLag, the simulation is divided into contiguous, non-overlapping time blocks in the order of training, validation, and test, so the test block is separated from the training block by the entire validation block. The exact split of each dataset, including the number of trajectories and samples in each set, is defined in the data loading code of our released codebase.

The MixedParam split is designed to evaluate both in-distribution and out-of-distribution generalization. The test Reynolds numbers include the training Reynolds numbers, which measure in-distribution accuracy, and the intermediate values between them, which are unseen during training and measure generalization to new parameters. A shared Reynolds number does not imply shared data, since the test trajectories are separate simulations from the training trajectories.


W1 Baseline Tuning and Training Schedules

We tuned key hyperparameters for the baselines, including the latent dimension, number of layers, batch size, and learning rate, and applied a cosine annealing training schedule.


W1 Winning Agent Designs

We present a case study on agent designed solver on MixedBC in Appendix N (Case Study of Agent-Designed Solver), where we present the full design, an overview figure, and the complete model code. Below, we briefly explain the model.

The model is a U-Net that receives the boundary condition in two complementary forms. First, a boundary embedding combines the one-hot boundary type, the four boundary values, separate Dirichlet and Neumann copies of these values, and a periodic flag, and a feature-wise linear modulation (FiLM) layer in every residual block uses this embedding to scale and shift each channel. Because Dirichlet values prescribe the solution while Neumann values prescribe its derivative across the boundary, the separate copies spare the network from inferring their meaning from the boundary type. Second, fifteen boundary maps stacked onto the input field show where each boundary lies and what value it imposes, through the boundary type copied to every grid point and a value map, an edge mask, and a distance map for each side. Each component directly targets the boundary condition challenge of MixedBC.


Thank you for your constructive feedback. We have updated the manuscript to incorporate your suggestions. We hope this clears up any confusion and concerns, and if so, we would greatly appreciate your consideration in raising the score.

Reviewer_grVwOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AutoPDEBench, a benchmark of 25 tasks for evaluating autonomous agents that design neural PDE solvers. Agents modify model architectures, training objectives, and hyperparameters through iterative experiments with validation feedback. The paper provides a multi-agent baseline and evaluates its performance against existing solvers, with additional analyses of LLM backbones, architecture transfer, and longer research runs.

Critical strengths
  1. Designing neural solvers for different physical systems often requires substantial expertise and experimentation, making this a meaningful application for autonomous research agents.

  2. The benchmark covers a broad range of modeling challenges, including varying boundary conditions, irregular observations, particle representations, and coupled physics. I like that the tasks are organized around these challenges and share a unified evaluation interface.

  3. The analysis goes beyond aggregate performance. Comparisons with specialized solvers, transferring generated architectures to other PDEs through retraining, and extending the research horizon help clarify where the agent works well and where it still struggles.

Critical weaknesses
  1. The benchmark’s incremental contribution over existing PDE datasets is not sufficiently clear. The collection is broad, but I would like to see a more systematic comparison showing which challenges are genuinely missing from existing resources and what new capabilities the benchmark actually tests. Newly generated datasets alone also do not establish the stronger claim that successful agents must discover novel solutions rather than adapt known methods.

  2. The computational comparison is insufficiently controlled. Section 3.1 constrains GPU memory but explicitly leaves training time and FLOPs unrestricted, while total search costs and baseline tuning budgets are not clearly reported. This makes it difficult to separate better solver designs from greater computational effort and limits reproducibility.

  3. The experiments provide limited evidence about the value of the research process itself. Most comparisons are against fixed solver families, without matched-budget comparisons to simpler strategies such as generating independent candidates or tuning existing architectures. The specialized comparisons on three tasks are helpful, but strong grid-based baselines are also missing on applicable tasks. Overall, the results show that automated customization is useful, but it is less clear how much of the improvement comes from the iterative research process itself.

Questions for authors

How consistent are the agent’s improvements across independent runs of the full research pipeline?

5 comments

We thank you for your insightful comments on improving our paper. We hope the following could address your concerns.

W1 PDE Datasets Contribution

AutoPDEBench diversifies domain-specific challenges rather than PDE families, because it benchmarks auto-research agents and not neural solvers in general. We define a domain-specific challenge as a property of the problem setting that general-purpose neural solvers either cannot handle or would handle better with a specialized design, and our taxonomy (Appendix M DATASET TAXONOMY) groups these challenges into generalization, complex domains and forcing, complex physics, scale, and observational irregularity. Varying the PDE family suits benchmarks of neural solvers but not of auto-research agents. A standard neural solver such as FNO already handles a standard Navier–Stokes dataset on a regular grid, and applying it to another PDE such as Burgers’ equation does not necessitate a newly designed architecture. When the same PDE is posed under different boundary conditions, however, the solver must explicitly encode the given boundary condition, and standard neural solvers such as FNO have no built-in mechanism to do so. Challenges of this kind require specialized architectures that human researchers currently design by hand [1,2], and AutoPDEBench evaluates whether auto-research agents can automate this design process.

We compile domain-specific challenges that general-purpose solvers either cannot handle or would handle better with a specialized solver. In LagFSI, the solid is composed of many particles, and existing Lagrangian simulators predict individual particle positions and cannot enforce the integrity of the solid. In HighSpeedFlow, [3] shows that the shock waves and discontinuities of high-speed flow benefit from a specialized solver. Previous benchmarks do not provide a large set of datasets dedicated to these domain-specific challenges, and the few challenges they contain are either insufficient in scale or entangled with other challenges. For example, previous benchmarks vary PDE parameters over only a few discrete values, whereas MixedParam trains on Reynolds numbers across laminar and turbulent regimes and includes unseen intermediate values at test time. Similarly, each previous dataset fixes one boundary condition per PDE, so a boundary condition generalization task assembled from them also mixes different PDEs. AutoPDEBench separates the two, with MixedBC varying only the boundary condition of a fixed PDE and MixedPDE varying only the PDE under a shared periodic boundary condition. AutoPDEBench therefore serves as an ideal benchmark for auto-research agents.


W1, W3 Adaptation of Known Methods

To test whether adapting known methods is sufficient, we add three baselines that use our agentic system but adapt an existing general-purpose solver instead of designing one from scratch. The three agents start from Transolver for transformer-based models, LNO for reduced-order models, and AMG for graph-based models. We report their results below. These adaptations improve over the original models only marginally and only on some datasets, which shows that adapting a known method amounts to a local optimization around that method and does not reach the designs these datasets require. Solving AutoPDEBench therefore requires designs beyond small adaptations of known methods. As a benchmark, AutoPDEBench will include these adaptation agents as additional baselines.

W1, W3 Adaptation of Known Methods (Continued)

TaskMetricTransolveragent_transolverLNOagent_lnoAMGagent_amgAgent
1_HighSpeedFlowRMSE3.232×1013.232 \times 10^{1}3.077×1013.077 \times 10^{1}4.482×1014.482 \times 10^{1}4.115×1014.115 \times 10^{1}2.819×1012.819 \times 10^{1}2.410×1012.410 \times 10^{1}5.066×100\mathbf{5.066 \times 10^{0}}
FRMSE2.372×1032.372 \times 10^{3}2.260×1032.260 \times 10^{3}3.282×1033.282 \times 10^{3}2.985×1032.985 \times 10^{3}2.181×1032.181 \times 10^{3}1.809×1031.809 \times 10^{3}4.218×102\mathbf{4.218 \times 10^{2}}
Kinetic Energy3.067×1033.067 \times 10^{3}2.691×1032.691 \times 10^{3}6.092×1036.092 \times 10^{3}5.475×1035.475 \times 10^{3}3.255×1033.255 \times 10^{3}2.434×1032.434 \times 10^{3}3.514×102\mathbf{3.514 \times 10^{2}}
3_MixedParamRMSE4.359×1004.359 \times 10^{0}4.370×1004.370 \times 10^{0}4.346×1004.346 \times 10^{0}4.376×1004.376 \times 10^{0}3.128×1003.128 \times 10^{0}3.216×1003.216 \times 10^{0}1.345×100\mathbf{1.345 \times 10^{0}}
FRMSE3.514×1023.514 \times 10^{2}3.488×1023.488 \times 10^{2}3.553×1023.553 \times 10^{2}3.489×1023.489 \times 10^{2}2.688×1022.688 \times 10^{2}2.726×1022.726 \times 10^{2}1.078×102\mathbf{1.078 \times 10^{2}}
Kinetic Energy3.593×1003.593 \times 10^{0}3.561×1003.561 \times 10^{0}3.605×1003.605 \times 10^{0}3.514×1003.514 \times 10^{0}1.741×1001.741 \times 10^{0}1.788×1001.788 \times 10^{0}2.780×10−1\mathbf{2.780 \times 10^{-1}}
10_LagDynamicRMSE–5.843×10−15.843 \times 10^{-1}6.105×10−16.105 \times 10^{-1}6.106×10−16.106 \times 10^{-1}–5.993×10−15.993 \times 10^{-1}5.530×10−1\mathbf{5.530 \times 10^{-1}}
LagKE–2.737×1032.737 \times 10^{3}1.349×103\mathbf{1.349 \times 10^{3}}3.237×1033.237 \times 10^{3}–2.229×1032.229 \times 10^{3}1.662×1031.662 \times 10^{3}
EuARMSE–1.107×10−11.107 \times 10^{-1}1.129×10−11.129 \times 10^{-1}1.123×10−11.123 \times 10^{-1}–1.117×10−11.117 \times 10^{-1}9.951×10−2\mathbf{9.951 \times 10^{-2}}
EuAFRMSE–2.560×1002.560 \times 10^{0}2.621×1002.621 \times 10^{0}2.605×1002.605 \times 10^{0}–2.574×1002.574 \times 10^{0}2.299×100\mathbf{2.299 \times 10^{0}}

W2 Computational Budget

We apologize for the confusion regarding the statement in Section 3.1. When we noted that we place “no restrictions on training or inference time, FLOPs, or parameter count,” we meant these are not enforced as explicit numerical constraints. However, the experimental environment is strictly controlled to ensure fair comparisons. Every general-purpose and agent-designed solver is trained on the same GPU with the same memory budget for 100 epochs. The training loop and the 100-epoch limit are hardcoded in our agent harness and cannot be changed by the agent. The shared memory budget bounds the model size and batch size, which in turn bounds the compute of each training step. Because the number of epochs is fixed and the cost of each epoch is bounded, the total training compute of every solver is bounded as well. This protocol reflects the practical setting of neural solver design, where researchers develop models on a given GPU and compare them under a common number of epochs. Empirically, the agent-designed solvers never use more training time, FLOPs, or parameters than the general-purpose baselines, as we report below.

W2 Computational Budget (Continued)

TaskGPUGNOTTransolverTransolver++LNOAMGAgent
1_HighSpeedFlowNvidia Titan XP 12GB10.0911.9111.982.7827.283.33
2_RealDataNvidia V100 16GB5.155.636.132.786.171.45
3_MixedParamNvidia V100 16GB13.6115.2822.044.8425.095.08
4_HeterogeneousNvidia V100 16GB5.686.616.252.065.665.21
5_PartialNvidia Titan XP 12GB9.7511.5811.544.1626.434.02
6_MixedDomainNvidia V100 16GB4.388.985.911.4617.728.51
7_MultiPhysicsNvidia V100 16GB45.7136.5239.8316.2178.9117.85
8_3DNvidia A6000 48GB37.7649.3256.2313.81–13.76
9_IrregularTimeNvidia V100 16GB9.7511.6211.592.7426.294.60
10_LagDynamicNvidia V100 16GB–––6.89–6.49
11_LagFSINvidia V100 16GB8.048.766.433.568.887.71
12_ExtremeNvidia L40S 48GB35.4541.8247.049.68–9.65
13_MixedResNvidia A6000 48GB11.2610.5611.841.59–5.47
14_ExtLag1Nvidia A100 40GB11.5012.2111.812.796.297.93
15_ExtLag2Nvidia V100 16GB4.274.694.621.513.304.46
16_MixedDimNvidia A6000 48GB15.4520.4322.175.18–12.62
17_MixedBCNvidia V100 16GB8.6110.1810.183.6523.122.24
18_LongHorizonNvidia V100 16GB2.284.443.0410.316.442.06
19_StochasticNvidia RTX 4090 24GB47.2339.2038.0918.9077.8320.46
20_MultiPhaseNvidia V100 16GB7.5310.678.992.3625.057.85
21_MixedPDENvidia Titan XP 12GB11.0213.0014.164.6329.624.27
22_LargeLagNvidia H200 141GB8.8614.0912.221.386.123.51
23_Sim2RealNvidia RTX 4090 24GB12.4914.7814.673.53133.974.42
24_HighDimensionalNvidia H200 141GB–––––88.61
25_ComplexGeometryNvidia L40S 48GB5.854.316.191.4757.631.87

Consistent with established auto-research benchmarks [4,5], we do not impose a single FLOP or parameter budget. This choice also matters for our baselines, which come from distinct architecture families where the relation between parameter count and compute differs sharply. A graph-based model such as AMG processes every node and edge of the mesh, so its compute grows with the mesh size even when the network is small. A reduced-order model such as LNO operates on a compact latent representation, so it can hold many more parameters at a fraction of the compute. Matching parameter counts would therefore give the graph model far more compute than the other baselines, while matching FLOPs would force it far below its intended capacity. Any single fixed budget would systematically favor one architecture family over another, whereas a shared GPU and a fixed number of epochs treat every family equally. We report model parameter counts and FLOPs in the table below, demonstrating that the agent-designed models do not require more parameters or compute than the baselines.

TaskMetricGNOTTransolverTransolver++LNOAMGAgent
1_HighSpeedFlowParam (M)5.7025.7975.68422.8502.37918.008
FLOPs (G)248.085201.440201.708104.287108.71246.013
2_RealDataParam (M)7.5535.7977.07523.8992.3795.945
FLOPs (G)248.060201.423251.171138.580108.68733.126
3_MixedParamParam (M)5.7025.7975.71944.0933.54133.588
FLOPs (G)187.246201.415219.345277.076162.29311.885
4_HeterogeneousParam (M)5.7055.6675.66719.7032.3812.206
FLOPs (G)57.64459.38359.38355.91027.51257.848
5_PartialParam (M)9.4047.2067.20941.5342.37830.440
FLOPs (G)308.848242.703317.899156.280108.67542.363
Reviewer_vmP1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a PDE benchmark targeting at domain-specific challenges for neural solvers, which consists of 25 datasets. The datasets are evaluated on several general-purpose models alongside with a multi-agent pipeline, showcasing the potential of AI agent in autonomous neural solver designing. Contributions: 1. The paper proposed AutoPDEBench, a benchmark designed based on domain specific challenges for automate neural solver designing. 2. The paper compared general-purpose neural solver models, problem specific neural models, with agentic designed neural model pipeline, demonstrating the capability of AI agent in autonomous neural solver designing. Validation: 1. The paper compared the general-purpose neural solver with agentic guided neural solver on the 25 datasets. 2. The paper studied the effect of different Sota LLMs on the design of neural solvers.

Critical strengths

The domain specific challenging 25 datasets within the AutoPDEBench for automated neural PDE solver design.

Critical weaknesses
  1. The proposed benchmark has limited diversity in terms of PDE families. Majority of the tasks are Navier-Stokes or hydrodynamics related.
  2. The computational budgets is not standardized. The evaluation protocol constrained GPU memory but not on training time, FLOPs or network size.
Questions for authors

1.Could the author provide the details on the computation budgets constraints when comparing the general-purpose neural solvers with the agent assisted neural solver. This would make the result more reliable. 2.Could the author clarify how the diversity of PDE families was considered during benchmark generation?

4 comments

We thank you for your insightful comments on improving our paper. We hope the following could address your concerns.

W1, Q1 Diversity of PDE Families

AutoPDEBench diversifies domain-specific challenges rather than PDE families, because it benchmarks auto-research agents and not neural solvers in general. We define a domain-specific challenge as a property of the problem setting that general-purpose neural solvers either cannot handle or would handle better with a specialized design, and our taxonomy (Appendix M DATASET TAXONOMY) groups these challenges into generalization, complex domains and forcing, complex physics, scale, and observational irregularity. Varying the PDE family suits benchmarks of neural solvers but not of auto-research agents. A standard neural solver such as FNO already handles a standard Navier–Stokes dataset on a regular grid, and applying it to another PDE such as Burgers’ equation does not necessitate a newly designed architecture. However, when the same PDE is posed under different boundary conditions, the solver must explicitly encode the given boundary condition, and standard neural solvers such as FNO have no built-in mechanism to do so. Challenges of this kind require specialized architectures that human researchers currently design by hand [1,2], and AutoPDEBench evaluates whether auto-research agents can automate this design process.

Our work covers 15 distinct PDEs, 8 of which lie outside Navier–Stokes and hydrodynamics, including linear advection, advection–diffusion, Burgers, Gray–Scott, Allen–Cahn, Kuramoto–Sivashinsky, stochastic quantization and gyrokinetic. The prevalence of Navier–Stokes reflects challenge isolation. Standard solvers already handle these well-studied PDEs, so the domain-specific challenge is the only difficulty the agent faces. For example, MixedBC is built on linear advection rather than the compressible flow of HighSpeedFlow, whose shock waves would add a second challenge [3] and confound which one the agent solved.


W2 Computational Budget and Evaluation Protocol

We apologize for the confusion regarding the statement in Section 3.1. When we noted that we place “no restrictions on training or inference time, FLOPs, or parameter count,” we meant these are not enforced as explicit numerical constraints. However, the experimental environment is strictly controlled to ensure fair comparisons. Every general-purpose and agent-designed solver is trained on the same GPU with the same memory budget for 100 epochs. The training loop and the 100-epoch limit are hardcoded in our agent harness and cannot be changed by the agent. The shared memory budget bounds the model size and batch size, which in turn bounds the compute of each training step. Because the number of epochs is fixed and the cost of each epoch is bounded, the total training compute of every solver is bounded as well. This protocol reflects the practical setting of neural solver design, where researchers develop models on a given GPU and compare them under a common number of epochs. Empirically, the agent-designed solvers never use more training time, FLOPs, or parameters than the general-purpose baselines, as we report below.

Consistent with established auto-research benchmarks [4,5], we do not impose a single FLOP or parameter budget. This choice also matters for our baselines, which come from distinct architecture families where the relation between parameter count and compute differs sharply. A graph-based model such as AMG processes every node and edge of the mesh, so its compute grows with the mesh size even when the network is small. A reduced-order model such as LNO operates on a compact latent representation, so it can hold many more parameters at a fraction of the compute. Matching parameter counts would therefore give the graph model far more compute than the other baselines, while matching FLOPs would force it far below its intended capacity. Any single fixed budget would systematically favor one architecture family over another, whereas a shared GPU and a fixed number of epochs treat every family equally.

Q2 Computational Budget Details

We report the GPU memory usage and training times (in hours) in the table below. Across all experiments, the agent-designed models consistently train faster than the general-purpose baselines, demonstrating the fairness of our comparison. This efficiency stems from two factors: (1) the strict constraints of a fixed GPU memory budget and a hard 100-epoch limit, and (2) the agent naturally converging on highly efficient model architectures that are optimized for the specific task.

TaskGPUGNOTTransolverTransolver++LNOAMGAgent
1_HighSpeedFlowNvidia Titan XP 12GB10.0911.9111.982.7827.283.33
2_RealDataNvidia V100 16GB5.155.636.132.786.171.45
3_MixedParamNvidia V100 16GB13.6115.2822.044.8425.095.08
4_HeterogeneousNvidia V100 16GB5.686.616.252.065.665.21
5_PartialNvidia Titan XP 12GB9.7511.5811.544.1626.434.02
6_MixedDomainNvidia V100 16GB4.388.985.911.4617.728.51
7_MultiPhysicsNvidia V100 16GB45.7136.5239.8316.2178.9117.85
8_3DNvidia A6000 48GB37.7649.3256.2313.81–13.76
9_IrregularTimeNvidia V100 16GB9.7511.6211.592.7426.294.60
10_LagInletOutletNvidia V100 16GB–––6.89–6.49
11_LagFSINvidia V100 16GB8.048.766.433.568.887.71
12_ExtremeNvidia L40S 48GB35.4541.8247.049.68–9.65
13_MixedResNvidia A6000 48GB11.2610.5611.841.59–5.47
14_ExtLag1Nvidia A100 40GB11.5012.2111.812.796.297.93
15_ExtLag2Nvidia V100 16GB4.274.694.621.513.304.46
16_MixedDimNvidia A6000 48GB15.4520.4322.175.18–12.62
17_MixedBCNvidia V100 16GB8.6110.1810.183.6523.122.24
18_LongHorizonNvidia V100 16GB2.284.443.0410.316.442.06
19_StochasticNvidia RTX 4090 24GB47.2339.2038.0918.9077.8320.46
20_MultiPhaseNvidia V100 16GB7.5310.678.992.3625.057.85
21_MixedPDENvidia Titan XP 12GB11.0213.0014.164.6329.624.27
22_LargeLagNvidia H200 141GB8.8614.0912.221.386.123.51
23_Sim2RealNvidia RTX 4090 24GB12.4914.7814.673.53133.974.42
24_HighDimensionalNvidia H200 141GB–––––88.61
25_ComplexGeometryNvidia L40S 48GB5.854.316.191.4757.631.87

Additionally, we report model parameter counts and FLOPs in the table below, demonstrating that the agent-designed models do not require more parameters or compute than the baselines. As a benchmark work, we will include them as additional evaluation criteria, encouraging future research to jointly optimize for accuracy and computational efficiency.

TaskMetricGNOTTransolverTransolver++LNOAMGAgent
1_HighSpeedFlowParam (M)5.7025.7975.68422.8502.37918.008
FLOPs (G)248.085201.440201.708104.287108.71246.013
2_RealDataParam (M)7.5535.7977.07523.8992.3795.945
FLOPs (G)248.060201.423251.171138.580108.68733.126
3_MixedParamParam (M)5.7025.7975.71944.0933.54133.588
FLOPs (G)187.246201.415219.345277.076162.29311.885
4_HeterogeneousParam (M)5.7055.6675.66719.7032.3812.206
FLOPs (G)57.64459.38359.38355.91027.51257.848
5_PartialParam (M)9.4047.2067.20941.5342.37830.440
FLOPs (G)308.848242.703317.899156.280108.67542.363

Thank you for your constructive feedback. We have updated the manuscript to incorporate your suggestions. We hope this clears up any confusion and concerns, and if so, we would greatly appreciate your consideration in raising the score.

[1] Guiding continuous operator learning through Physics-based boundary constraints, ICLR 2023

[2] BENO: Boundary-embedded Neural Operators for Elliptic PDEs, ICLR 2024

[3] A Two-Phase Deep Learning Framework for Adaptive Time-Stepping in High-Speed Flow Modeling, ICLR 2026

[4] MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation, Arxiv

[5] MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering, ICLR 2025

I thank the authors for the detailed clarification. I will maintain my positive score.

We sincerely thank the reviewer for the positive feedback. We greatly appreciate your support and your high evaluation of our work.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback and the AC and PC for coordinating the rebuttal. We are delighted that the reviewers found the 25 datasets a valuable testbed for auto-research agents and valued the challenge-based organization along with the diverse metrics and comprehensive analyses such as the specialist comparisons and the extended research horizon. We believe the remaining concerns stem from misunderstandings of our benchmark design and evaluation protocol, which we have addressed with comprehensive new results summarized below.


Benchmark contribution and diversity (Reviewer vmP1, Reviewer grVw)

AutoPDEBench benchmarks auto-research agents, so it diversifies domain-specific challenges and not PDE families. Applying FNO to another PDE on a regular grid needs no new architecture, whereas boundary condition generalization, shocks in high-speed flow, or solid integrity in fluid–structure interaction call for specialized designs that researchers currently build by hand. Appendix M organizes these challenges into a five-category taxonomy. The benchmark still spans 15 distinct PDEs, 8 outside Navier–Stokes and hydrodynamics, and well-studied PDEs are used deliberately so that each challenge is the only difficulty the agent faces.

Prior benchmarks either lack these challenges or offer them only at limited scale or entangled with other factors. For example, they vary PDE parameters over only around 10 values, and each PDE comes with a single fixed boundary condition, so a boundary condition task built from them also mixes different PDEs.


Controlled and fair computational budget (Reviewer vmP1, Reviewer grVw, Reviewer kjVi)

The statement in Section 3.1 means that time, FLOPs, and parameters are not explicitly constrained by the code. However, the environment itself is strictly controlled. Every baseline and agent-designed solver is trained on the same single GPU with the same memory budget for exactly 100 epochs, hardcoded in the harness so the agent cannot change it. GPU memory bounds the model and batch size. The epoch count is also fixed, so the total training compute of every solver is bounded. We report training times on all 25 datasets, which show that the agent-designed solvers never use more training time than the baselines. The reported parameter counts and FLOPs further show that they do not use more compute.


Gains come from scientific design, not adaptation or extra search (Reviewer grVw, Reviewer kjVi)

  • Adaptation agents. We added three baselines that use our agentic system but start from Transolver, LNO, or AMG and adapt them to each dataset. They improve over the original models only marginally, while the agent designing from scratch reaches significantly lower error than the best adaptation. Solving AutoPDEBench thus requires designs beyond small adaptations of known method.

  • Iterative research versus independent sampling. With a matched budget of five designs, iterative search outperforms the best of five independent candidates, demonstrating the value of the research process itself.

  • Strong grid-based baselines. The agent-designed solvers achieve the lowest overall error against FNO, UNO, MoE-POT, MPP and Poseidon on five applicable datasets. 



Additional questions

Beyond the main concerns above, we also answered several specific questions from the reviewers. We briefly summarize them below.

  • Winning agent designs. Appendix N presents the agent-designed MixedBC solver, a U-Net that injects the boundary type and values into every residual block and adds maps marking where each boundary lies and what value it imposes. Each component directly targets the boundary condition challenge. 



  • Baseline tuning. All baselines were tuned over latent dimension, depth, batch size, and learning rate with a cosine annealing training schedule, and every agent uses five design iterations. 



  • Consistency across runs. Across three independent runs of the full pipeline, the agent consistently outperforms the general-purpose baselines. 



  • Data split. Every dataset is split by trajectory, or by contiguous non-overlapping time blocks, before windows are extracted, so no input and target pair appears in more than one split. MixedParam tests both training and unseen intermediate Reynolds numbers on separate simulations to measure in-distribution accuracy and out-of-distribution generalization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.