- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors collect 50 CUDA programming tasks for physical simulation, including individual operators and complete simulation steps. They run six existing models with their coding harnesses under fixed time budgets and compare correctness and GPU time with expert implementations. The strongest two systems pass all numerical checks and audits in the main evaluation, while most of their code still runs slower than the reference. The authors also examine some generated code and vary the time budget and number of attempts.
Critical strengths
I think the numerical results are worth-trusting within the evaluation setting. My problems are with how much I learn from the experiments and how much the paper contributes.
-
I think physical simulation is a reasonable place to test GPU coding agents, as some tasks need choices about the numerical algorithm and irregular memory access, so translating a formula into a CUDA kernel does not finish the performance work. The tuned references could be useful for people who want to measure how far an agent can go on these problems.
-
I appreciate that the authors actually examine whether their correctness checks work. They try correct variants and injected errors, check the true solver residuals, and inspect submissions close to the tolerance. They also repeat experiments for the strongest systems and report scores with equal family weights. I think the appendix gives enough support for the numbers they report.
Critical weaknesses
-
I find 50 tasks rather small for a paper that mainly contributes a benchmark, and the authors group them into 37 families themselves. Several tasks are variants of the same problem, while most hidden inputs keep the public problem size. The strongest two systems already pass all correctness checks, so I mostly expect to learn something from the performance part. I do not get much new understanding from it.
I do see what this paper adds to CodePDE, as CodePDE mainly evaluates solver accuracy although it also uses GPUs and reports runtime, while this paper gives expert CUDA performance targets. Metal-Sci also studies scientific GPU optimization, although it is a 10-task Apple Metal benchmark from the ICML 2026 DL4C workshop. The MSKernelBench preprint includes scientific CUDA routines. I guess the expert CUDA references and the simulation tasks are where this paper adds most of its value, but I find the current collection and analysis rather thin.
I get stuck at how the agents decide that the optimization is finished. The prompt does ask for speed, but the agent only knows its own runtime, as the authors hide the reference time and do not give a target. I feel like there is a missing agreement on how much effort is enough, since the agent could write a correct implementation, try some optimization, and consider the work finished, while from the authors' side it is still far from the expert. Opus and GPT use only 43% and 50% of their default budgets at the median, and neither reaches the deadline; some results still improve when the authors announce a longer budget. The authors report this honestly. I just do not see how to tell whether the agent has trouble optimizing further or thinks it has done enough. In particular, telling me that one agent uses Jacobi while the reference uses multigrid explains much of the time difference, as we already know these algorithms can differ greatly, but I fail to see how this answers why the agent makes that choice.
-
I am quite bothered that I cannot read or run the full benchmark. The authors promise to release the tasks, input generators, reference implementations, and evaluators. I can read the descriptions and one complete prompt and starter-code example in the appendix, but I still have no full collection to inspect.
I think the expert implementation is much of the benchmark here, as the authors use it for both correctness checks and speed scores. Its quality therefore matters a lot to how much I value the paper. A small collection could still have enough value if the references take substantial work and are very well optimized, but the current materials do not let me examine this part. I find a promise of future release insufficient for a paper whose main contribution is this resource.
I recommend rejection. I think the authors have checked the numerical part carefully, but I find the amount of new things in this paper too small, and I still have no full benchmark to inspect. I would need more from the task collection or the explanation of agent behavior to support acceptance.
Questions for authors
-
Could the authors provide an anonymous runnable benchmark during review, including the reference implementations, input generators, and evaluators? I would like to see how much work and optimization the references actually contain, as this could change how much I value the benchmark.
-
How did the authors choose these 50 tasks, and how much of physical simulation do they expect them to cover? I would like to know what I learn specifically from the added simulation tasks and expert references, beyond the expected effects of algorithm choice and GPU memory access.
-
Could the development traces explain why the strongest agents stop with so much time remaining? I would find a small experiment on representative tasks useful: keep the original time budget, compare the original prompt with one condition that tells the agent a reference time or target time without reference code or outputs, and a separate condition that asks it to keep optimizing when it declares completion with time left. I could then better tell how much the result depends on the agent's understanding of the effort needed and when to stop. This would affect how much I think I learn from the paper.
3 comments
Thanks for releasing the full benchmark and running the prompt experiments. It is especially interesting to me that telling GPT the reference time raised its fast₀.₅ from 42% to 66%, so the gap between Opus and GPT shrinks from 24 to 8 points. From my point of view this shows that part of the measured gap comes from the agent not knowing how far it is from the target, as it can only optimize against its own runtime. I learned something from this result and would like to see it, together with this discussion, in the main paper.
I still have one concern after inspecting the evaluator. All four CCD tasks are checked by one aggregate relative L2 over the times of impact, so a missed collision only reads as toi = 1. In my CPU reproduction of ccd_vv, a small broad-phase change (a 0.95h gap instead of h) missed thousands of collisions and still passed the tolerance. I guess the same would happen on CUDA, but I would ask the authors to check this case and clarify what a correctness pass means for CCD.
Having seen the reference implementations and the new experiments, I am raising my rating from 2 to 3.
We thank the reviewer for the careful review and for inspecting the released benchmark in detail.
Following the suggestion, we have added the prompt experiments and this discussion to the main paper. Section 4.4 now includes a paragraph on stopping behavior and a table comparing the original prompt with the reference-time and keep-optimizing prompts for Opus + Claude Code and GPT + Codex CLI. We've marked the changes in blue.
We thank the reviewer for pointing out the CCD issue, which we confirmed on CUDA. Taking ccd_vv as an example, we have fixed this with a per-vertex check: the evaluator reruns the reference at thickness , and every vertex's time of impact must lie between the two results. The time of impact can only decrease as the thickness grows, so this interval contains every correct answer. Any missed or late impact falls outside it, even if it is the only one. We verified that the variant now fails to pass the updated correctness test. ccd_ve uses the same thickness bounds, while ccd_vf and ccd_ee instead widen and narrow the inside test by . A CCD correctness pass therefore now means that every time of impact matches the reference up to this small change of the contact criterion. We re-evaluated all previous agent submissions for the CCD tasks with the updated evaluators: no correctness verdict changed. We have updated the paper (Section 3.2 and the tolerance-validation appendix) and the supplementary material accordingly.
We are grateful for the reviewer's careful assessment and concrete suggestions.
Q1: Runnable benchmark
The full benchmark is uploaded as Supplementary Material. It contains all 50 task prompts, starter code, reference implementations (
benchmark/<task>/ref/), public and hidden input generators, evaluators, audit scripts, and the Docker environment.The references comprise about 35k lines of CUDA code. A full-simulator reference such as
pic_flip,apic, ormpm_semi_implicitalone has 1.2–1.5k lines and 20–26 kernels.Q2: Task selection, coverage, and what the benchmark reveals
Selection and coverage. Physics-based simulation is commonly divided into fluids and solids. Fluids further divide into flow fields (e.g., smoke and incompressible flow), free-surface liquids, and granular materials; solids divide into elastic bodies and rigid bodies. GPUPhysBench covers every one of these classes except rigid bodies:
We exclude rigid bodies because they have very few degrees of freedom per object, so they pose little GPU-optimization challenge. The tasks also cover all common simulation data structures: grids, meshes (triangle and tetrahedral), and particles, as well as hybrid particle–grid methods that combine them.
What the benchmark reveals.
stable_fluids,pic_flip, andapic. On the standalonepoisson_dirichlet, however, it uses Jacobi-PCG and never mentions multigrid preconditioner. Across GPT's repeated runs onpoisson_dirichlet, one chooses multigrid preconditioner from the start, while another starts with Jacobi-PCG and never changes it.Q3: Why agents stop early
Following the suggestion, we kept the original budget and ran both conditions on all 50 tasks for Opus and GPT:
Knowing the reference time helps; being told to keep optimizing does not. The reference time raises fast0.5 by 8 points for Opus and 24 points for GPT. For example, on
poisson_dirichlet, Opus's first Jacobi-PCG takes 551 ms against the 18.7 ms reference. Seeing the 30× gap, it switches to multigrid preconditioner and reaches 0.19×, versus 0.05× without the reference time. The keep-optimizing instruction only modestly increases budget use and leaves fastp within run-to-run variation.The stopping pattern is unchanged. With either prompt change, agents still stop once they reach a local plateau, well before the deadline. The results therefore reflect mainly whether an agent can find the next substantive optimization, not a misunderstanding of how much effort is expected.