- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces an agentic environment (or harness) called GAV to improve the performance of agentic PDE solvers. GAV uses the method of manufactured solutions to cheaply generate PDE problems to help PDE solvers improve their outputs.
Critical strengths
- This paper is studying an interesting question of enhancing the environment/harness used by PDE solver agents.
- The the specific- approach of cheaply generating PDE problems in a curriculum of sorts to help that agent improve is also an elegant idea.
Critical weaknesses
- It is not clear to me that GAV improves agentic performance on tasks of interest. Figure 5 for instance does clearly show improvement of GAV/GAV-Full over the Original environment, but if I'm understanding correctly, the evaluations are done on PDE problems that GAV itself generates. It would be more compelling IMO to demonstrate that GAV combined with something like CodePDE could improve CodePDE performance on the benchmarks that CodePDE identified. If this were teh case, I'd be much more bullish on this work.
- Figure 5 seems to show that GAV is better than GAV-Full, thus calling into question the effectiveness of guided generation and perturbation.
Questions for authors
- The theoretical results provide a reasonable conceptual framing, but it's not clear that they guide any concrete design choice for the proposed environment. If they do, can you explain how?
6 comments
Response to Reviewer ar4G (2/2)
W2 — GAV-Full versus GAV
Answer: GAV-Full is an optional extension for robustness under distribution shift, not the core algorithm. GAV's core (learning the problem distribution from data, exact problems through MMS, and continuous input queries) already improves all three agents by 11–16× over Original (Fig. 5).
GAV-Full adds two tools for problems that move away from the learned distribution:
- Guided generation samples problems that are hard for the current program while staying within a KL neighborhood of the learned distribution, the worst-case objective in Eq. (5). Every guidance update produced harder problems (App. D.2).
- Perturbation checks whether an improvement still holds on problems moved off the learned distribution (App. D.3).
Fig. 5 instead measures average error on problems from the learned distribution itself, where the core GAV is already highly effective. Even there, GAV-Full improves all three agents over Original (3.4–55×) and gives PDE-SHARP the lowest error in Fig. 5 (0.069%). The revision makes this role of GAV-Full explicit in Sec. 3.3.
Q1 — How the theory guides the design
Answer: Yes. The theory directly shapes three design choices in GAV; the revision spells out the first two in Sec. 3.2.
- Learning the problem distribution from target data. Proposition 1 bounds the Verifier's relative evaluation error through the KL divergence between learned and target problem distributions. This motivates learning the distribution from target-regime data rather than using a generic prior. Since MMS cannot increase this divergence (data-processing inequality [1]), learning the solution distribution suffices. Replacing the learned distribution with a Gaussian-process prior increases test error 3.3× (Sec. 4.4, Table 1).
- Evaluating on problems unseen during development. Proposition 1 requires the evaluated program to be chosen without using the test problems. Accordingly, the Verifier never reveals reference solutions to the Agent, and final evaluation uses a fixed test set separate from the selection problems (App. C.2).
- Guided generation as bounded tilting. Theorem 1 shows that, in an idealized convex setting, if each guidance step tilts the learned distribution toward high-error problems with bounded strength, stabilized revisions monotonically reduce the worst-case loss and converge. Guided generation therefore targets the loss-tilted distribution in Eq. (6), with its strength capped relative to the flow velocity (Table 4) so that guided problems stay close to the learned regime.
References
[1] Cover and Thomas. Elements of Information Theory. Wiley, 2006.
Thanks for sharing, thoug it looks like you only considered 3 of the 5 datasets included in Table 1 of the CodePDE paper. Moreover, providing similar results for the same benchmarking tasks used by AutoNumerics and PDE-Sharp would go a long way.
Overall, I think these results are promising, but they don't change my mind about the current version of the paper. I think the work would benefit from more systematically making this point in a future submission.
Response to Reviewer ar4G (1/2)
We thank the reviewer for finding the question of enhancing the environment of PDE agents interesting and the cheap generation of a problem curriculum elegant. We address performance on tasks of interest (W1) in this comment, and GAV-Full (W2) and how the theory guides the design (Q1) in the next. Changes in the revised PDF are highlighted in blue, and the new experiments are collected in App. F.
W1 — Performance on tasks of interest
Answer: GAV improves CodePDE on CodePDE's own benchmarks, on top of an already strong baseline. With gpt-5.6-sol, CodePDE alone reaches nRMSE below 10⁻⁴ on all three tasks we test; on the official problems and against the official reference solutions, CodePDE + GAV still lowers its error by 2.8× on reaction–diffusion and 1.4× on compressible NS, and on Burgers both versions reproduce the official reference to round-off. On our NS task, GAV's gains also hold beyond GAV-generated problems: on original forward problems with their prescribed forcing, GAV improves all three agents by 15–107×. In the revision, the main GAV-Full program is also 3.4× more accurate than the best of three established solvers.
1. CodePDE's own benchmarks
We compare CodePDE and CodePDE + GAV on three of CodePDE's native tasks: 1D Burgers (ν=0.01), 1D reaction–diffusion (ν=0.5, ρ=1), and 1D compressible NS (η=ζ=0.1, γ=5/3). Both versions use the same 50 official development problems, model, and development procedure (three candidate generations and three refinements). CodePDE + GAV additionally obtains new development problems built by recombining PDE parameters and applying MMS, without training a Generator, and queries inputs at chosen coordinates. Both select their final program on the same 50 development problems and are tested on the same 100 official problems, with the official initial conditions, parameters, boundary conditions, output grids, and numerical reference solutions. We report mean nRMSE (App. F.3, Table 12).
GAV thus further improves an already highly accurate agent wherever improvement is possible. Since GAV builds its development problems here from parameter recombination and MMS alone, it extends to new PDE families without training a Generator.
2. Original NS forward problems
We evaluate the frozen final programs of Sec. 4.3 on 20 original NS forward problems, keeping their prescribed forcing and using high-accuracy Nektar++ reference solutions, so no test problem is constructed by GAV (App. F.2, Table 11).
GAV improves all three agents by 15–107× on the original problems, so its gains carry over to the target task itself.
3. Accuracy against established solvers
In the revision, three established solvers solve the main numerical NS test problems with the same geometry, viscosity, boundary conditions, and continuous input queries, and are scored at the same points (App. F.1, Table 10).
The program developed with GAV-Full is 3.4× more accurate than the best established solver, and its accuracy carries over to the interpolated problems (0.1256%).