- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces 25 datasets for evaluating agents that design neural PDE solvers. The tasks cover different discretizations, physical parameters, partial observations, long rollouts, and other modeling difficulties. A planner, coder, and debugger propose a model, training loss, and configuration, using validation feedback and a GPU memory constraint. The default search runs for five iterations. The resulting solvers are compared with five general-purpose architectures, with additional specialist comparisons and architecture-transfer experiments on three scenarios.
Critical strengths
-
The 25 datasets cover more than small changes to one equation. Variable particle counts, mixed representations, external forcing, and long autoregressive prediction pose different engineering problems. The detailed modeling objectives in Appendix I make this a potentially valuable testbed for scientific coding agents.
-
I appreciate the additional physical metrics, the specialist comparisons in Section 4.2, and the retraining of transferred architectures in Section 4.3. These checks are more informative than only beating a generic solver on its original dataset.
Critical weaknesses
-
The paper does not give enough detail on baseline tuning, input conditioning, training schedules, or the winning agent designs to tell whether the gains come from scientific design or basic adaptation and additional search.
-
The paper states that the training, validation, and test sets do not overlap, but it does not clearly explain how the trajectories are split or how many samples are in each set. Therefore, it is unclear whether very similar time windows appear across different splits. In MixedParam, the training Reynolds numbers are spaced by 100, while the validation and test numbers are spaced by 25, all within the same range. This means some parameter values are shared between training and testing.
Questions for authors
Q1. For each method in the main results table, how many GPUs and how much GPU memory were used? How long was the training, how much tuning was done, and what was the total GPU time? Also, the paper states that there is no time limit, so why were some methods excluded for taking too long? What were the criteria?
Q2. Do all baselines use the same additional inputs as the agent-designed methods, such as PDE parameters, time coordinates, and masks? If the baselines are given the same tuning and adaptation budget, can they achieve similar improvements?
Q3. How exactly are trajectories, initial conditions, and time windows split into training, validation, and test sets?
2 comments
W1 Q2 Adaptation of Known Methods (Continued)
| Task | Metric | Transolver | agent_transolver | LNO | agent_lno | AMG | agent_amg | Agent |
|---|---|---|---|---|---|---|---|---|
| 1_HighSpeedFlow | RMSE | |||||||
| FRMSE | ||||||||
| Kinetic Energy | ||||||||
| 3_MixedParam | RMSE | |||||||
| FRMSE | ||||||||
| Kinetic Energy | ||||||||
| 10_LagDynamic | RMSE | – | – | |||||
| LagKE | – | – | ||||||
| EuARMSE | – | – | ||||||
| EuAFRMSE | – | – |
Q3 Dataset Split
All datasets are split before any time windows are extracted, so no input and target pair appears in more than one of the training, validation, and test sets. For datasets with multiple simulations, each trajectory, or each sample for steady-state data, is assigned entirely to one split, and time windows are extracted only within the trajectories of that split. For datasets that consist of a single long simulation, namely MixedDomain, LagInletOutlet, and LargeLag, the simulation is divided into contiguous, non-overlapping time blocks in the order of training, validation, and test, so the test block is separated from the training block by the entire validation block. The exact split of each dataset, including the number of trajectories and samples in each set, is defined in the data loading code of our released codebase.
The MixedParam split is designed to evaluate both in-distribution and out-of-distribution generalization. The test Reynolds numbers include the training Reynolds numbers, which measure in-distribution accuracy, and the intermediate values between them, which are unseen during training and measure generalization to new parameters. A shared Reynolds number does not imply shared data, since the test trajectories are separate simulations from the training trajectories.
W1 Baseline Tuning and Training Schedules
We tuned key hyperparameters for the baselines, including the latent dimension, number of layers, batch size, and learning rate, and applied a cosine annealing training schedule.
W1 Winning Agent Designs
We present a case study on agent designed solver on MixedBC in Appendix N (Case Study of Agent-Designed Solver), where we present the full design, an overview figure, and the complete model code. Below, we briefly explain the model.
The model is a U-Net that receives the boundary condition in two complementary forms. First, a boundary embedding combines the one-hot boundary type, the four boundary values, separate Dirichlet and Neumann copies of these values, and a periodic flag, and a feature-wise linear modulation (FiLM) layer in every residual block uses this embedding to scale and shift each channel. Because Dirichlet values prescribe the solution while Neumann values prescribe its derivative across the boundary, the separate copies spare the network from inferring their meaning from the boundary type. Second, fifteen boundary maps stacked onto the input field show where each boundary lies and what value it imposes, through the boundary type copied to every grid point and a value map, an edge mask, and a distance map for each side. Each component directly targets the boundary condition challenge of MixedBC.
Thank you for your constructive feedback. We have updated the manuscript to incorporate your suggestions. We hope this clears up any confusion and concerns, and if so, we would greatly appreciate your consideration in raising the score.
We thank you for your insightful comments on improving our paper. We hope the following could address your concerns.
Q1 Computational Budget
We apologize for the confusion regarding the statement in Section 3.1. When we noted that we place “no restrictions on training or inference time, FLOPs, or parameter count,” we meant these are not enforced as explicit numerical constraints. However, the experimental environment is strictly controlled to ensure fair comparisons. Every general-purpose and agent-designed solver is trained on the same single GPU with the same memory budget for 100 epochs. The training loop and the 100-epoch limit are hardcoded in our agent harness and cannot be changed by the agent. The shared memory budget bounds the model size and batch size, which in turn bounds the compute of each training step. Because the number of epochs is fixed and the cost of each epoch is bounded, the total training compute of every solver is bounded as well. This protocol reflects the practical setting of neural solver design, where researchers develop models on a given GPU and compare them under a common number of epochs. Empirically, the agent-designed solvers never use more training time than the general-purpose baselines, as we report below.
Methods are excluded from Table 2 for one of two reasons. AMG is excluded on some large three-dimensional and high-resolution datasets because its training would take more than a month, which is unacceptable in practice. This is the only exclusion due to training time. All other exclusions arise because a model inherently cannot handle the task, such as differing numbers of input and output particles in LagInletOutlet or five-dimensional data in HighDimensional.
W1 Q2 Adaptation of Known Methods
The default implementations of the baseline models only accept initial conditions and simply cannot process additional information such as boundary conditions or governing PDEs. This structural limitation is the exact motivation for AutoPDEBench. Instead of relying on human researchers to manually engineer new neural architectures that can handle these domain-specific challenges, we propose leveraging automated research agents to design them.
To test whether adapting known methods is sufficient, we add three agent baselines that start from an existing general-purpose solver and make small adaptations to it for the domain-specific challenge of each dataset. They start from Transolver for transformer-based models, LNO for reduced-order models, and AMG for graph-based models. Below, we report their results. These adaptations improve over the original models only marginally and only on some datasets, so solving these datasets requires designs beyond small adaptations of known methods. As a benchmark, We will include these adaptation agents as additional baselines.