acceptodds
Under review as a conference paper at ICLR 2027

GPUPhysBench: Benchmarking Coding Agents for Correct and Efficient GPU Physics Simulation

Abstract

Writing fast GPU code for physical simulation is difficult: implementations must preserve numerical accuracy while handling irregular data access, synchronization, and iterative solvers. We introduce GPUPhysBench, a benchmark of 50 tasks testing whether coding agents can meet these demands. Tasks cover fluids, deformable solids, and granular materials, from individual simulation operators to complete simulators. Agents write, compile, test, and optimize GPU code with access to a NVIDIA GPU under fixed time budgets. We report pass rates and runtime performance relative to expert-optimized reference implementations. In a single-attempt evaluation of six frontier model-harness pairs, the two strongest pass all 50 tasks, but even the fastest reaches at least the reference speed on only 22% of them, and no submission is more than 5% faster than the reference. The largest gaps arise in collision detection, constraint solving, and iterative solvers. GPUPhysBench brings physical simulation workloads to coding-agent evaluation, testing both the ability to implement numerical methods correctly and the ability to make them run efficiently.

Primary area:
datasets and benchmarks
Keywords:
coding agents, GPU programming, physical simulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hWpvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
3
Read review
Paper summary

The authors collect 50 CUDA programming tasks for physical simulation, including individual operators and complete simulation steps. They run six existing models with their coding harnesses under fixed time budgets and compare correctness and GPU time with expert implementations. The strongest two systems pass all numerical checks and audits in the main evaluation, while most of their code still runs slower than the reference. The authors also examine some generated code and vary the time budget and number of attempts.

Critical strengths

I think the numerical results are worth-trusting within the evaluation setting. My problems are with how much I learn from the experiments and how much the paper contributes.

  1. I think physical simulation is a reasonable place to test GPU coding agents, as some tasks need choices about the numerical algorithm and irregular memory access, so translating a formula into a CUDA kernel does not finish the performance work. The tuned references could be useful for people who want to measure how far an agent can go on these problems.

  2. I appreciate that the authors actually examine whether their correctness checks work. They try correct variants and injected errors, check the true solver residuals, and inspect submissions close to the tolerance. They also repeat experiments for the strongest systems and report scores with equal family weights. I think the appendix gives enough support for the numbers they report.

Critical weaknesses
  1. I find 50 tasks rather small for a paper that mainly contributes a benchmark, and the authors group them into 37 families themselves. Several tasks are variants of the same problem, while most hidden inputs keep the public problem size. The strongest two systems already pass all correctness checks, so I mostly expect to learn something from the performance part. I do not get much new understanding from it.

    I do see what this paper adds to CodePDE, as CodePDE mainly evaluates solver accuracy although it also uses GPUs and reports runtime, while this paper gives expert CUDA performance targets. Metal-Sci also studies scientific GPU optimization, although it is a 10-task Apple Metal benchmark from the ICML 2026 DL4C workshop. The MSKernelBench preprint includes scientific CUDA routines. I guess the expert CUDA references and the simulation tasks are where this paper adds most of its value, but I find the current collection and analysis rather thin.

    I get stuck at how the agents decide that the optimization is finished. The prompt does ask for speed, but the agent only knows its own runtime, as the authors hide the reference time and do not give a target. I feel like there is a missing agreement on how much effort is enough, since the agent could write a correct implementation, try some optimization, and consider the work finished, while from the authors' side it is still far from the expert. Opus and GPT use only 43% and 50% of their default budgets at the median, and neither reaches the deadline; some results still improve when the authors announce a longer budget. The authors report this honestly. I just do not see how to tell whether the agent has trouble optimizing further or thinks it has done enough. In particular, telling me that one agent uses Jacobi while the reference uses multigrid explains much of the time difference, as we already know these algorithms can differ greatly, but I fail to see how this answers why the agent makes that choice.

  2. I am quite bothered that I cannot read or run the full benchmark. The authors promise to release the tasks, input generators, reference implementations, and evaluators. I can read the descriptions and one complete prompt and starter-code example in the appendix, but I still have no full collection to inspect.

    I think the expert implementation is much of the benchmark here, as the authors use it for both correctness checks and speed scores. Its quality therefore matters a lot to how much I value the paper. A small collection could still have enough value if the references take substantial work and are very well optimized, but the current materials do not let me examine this part. I find a promise of future release insufficient for a paper whose main contribution is this resource.

I recommend rejection. I think the authors have checked the numerical part carefully, but I find the amount of new things in this paper too small, and I still have no full benchmark to inspect. I would need more from the task collection or the explanation of agent behavior to support acceptance.

Questions for authors
  1. Could the authors provide an anonymous runnable benchmark during review, including the reference implementations, input generators, and evaluators? I would like to see how much work and optimization the references actually contain, as this could change how much I value the benchmark.

  2. How did the authors choose these 50 tasks, and how much of physical simulation do they expect them to cover? I would like to know what I learn specifically from the added simulation tasks and expert references, beyond the expected effects of algorithm choice and GPU memory access.

  3. Could the development traces explain why the strongest agents stop with so much time remaining? I would find a small experiment on representative tasks useful: keep the original time budget, compare the original prompt with one condition that tells the agent a reference time or target time without reference code or outputs, and a separate condition that asks it to keep optimizing when it declares completion with time left. I could then better tell how much the result depends on the agent's understanding of the effort needed and when to stop. This would affect how much I think I learn from the paper.

3 comments

We are grateful for the reviewer's careful assessment and concrete suggestions.

Q1: Runnable benchmark

The full benchmark is uploaded as Supplementary Material. It contains all 50 task prompts, starter code, reference implementations (benchmark/<task>/ref/), public and hidden input generators, evaluators, audit scripts, and the Docker environment.

The references comprise about 35k lines of CUDA code. A full-simulator reference such as pic_flip, apic, or mpm_semi_implicit alone has 1.2–1.5k lines and 20–26 kernels.

Q2: Task selection, coverage, and what the benchmark reveals

Selection and coverage. Physics-based simulation is commonly divided into fluids and solids. Fluids further divide into flow fields (e.g., smoke and incompressible flow), free-surface liquids, and granular materials; solids divide into elastic bodies and rigid bodies. GPUPhysBench covers every one of these classes except rigid bodies:

  • Flow fields: advection, LBM, Poisson and viscosity solves, Stable Fluids.
  • Free-surface liquids: PIC/FLIP, APIC, PBF, surface tension, level set, phase field.
  • Granular materials: DEM and MPM.
  • Elastic bodies: mass-spring, FEM, projective dynamics, XPBD cloth, and continuous collision detection.

We exclude rigid bodies because they have very few degrees of freedom per object, so they pose little GPU-optimization challenge. The tasks also cover all common simulation data structures: grids, meshes (triangle and tetrahedral), and particles, as well as hybrid particle–grid methods that combine them.

What the benchmark reveals.

  • Algorithm choice is unreliable, even when the agent knows the better algorithm. Opus implements a multigrid preconditioner for the pressure solve in stable_fluids, pic_flip, and apic. On the standalone poisson_dirichlet, however, it uses Jacobi-PCG and never mentions multigrid preconditioner. Across GPT's repeated runs on poisson_dirichlet, one chooses multigrid preconditioner from the start, while another starts with Jacobi-PCG and never changes it.
  • Agents stop once they reach a local plateau. A common pattern in the traces is that the agent's last optimization attempt is a low-level tuning change (launch bounds, block size, cache hints, solver parameters) that turns out slower. The agent takes this as a sign that it has reached a plateau, reverts the change, and stops, even when there is still budget remaining.

Q3: Why agents stop early

Following the suggestion, we kept the original budget and ran both conditions on all 50 tasks for Opus and GPT:

  • Reference time: we add "For comparison, a reference implementation of this task takes X ms of GPU time on the example input that run_<task>.py builds." to the prompt.
  • Keep optimizing: we add "Keep optimizing when there is time left." to the prompt.
ModelSettingPass Ratefast0.5\text{fast}_{0.5}fast0.9\text{fast}_{0.9}fast1.05\text{fast}_{1.05}Median budget used
OpusOriginal100%66%22%0%43%
OpusReference time98%74%30%4%45%
OpusKeep optimizing100%64%24%0%55%
GPTOriginal100%42%16%0%50%
GPTReference time100%66%22%2%61%
GPTKeep optimizing98%44%18%0%54%

Knowing the reference time helps; being told to keep optimizing does not. The reference time raises fast0.5\text{fast}_{0.5} by 8 points for Opus and 24 points for GPT. For example, on poisson_dirichlet, Opus's first Jacobi-PCG takes 551 ms against the 18.7 ms reference. Seeing the 30× gap, it switches to multigrid preconditioner and reaches 0.19×, versus 0.05× without the reference time. The keep-optimizing instruction only modestly increases budget use and leaves fastp\text{fast}_p within run-to-run variation.

The stopping pattern is unchanged. With either prompt change, agents still stop once they reach a local plateau, well before the deadline. The results therefore reflect mainly whether an agent can find the next substantive optimization, not a misunderstanding of how much effort is expected.

Thanks for releasing the full benchmark and running the prompt experiments. It is especially interesting to me that telling GPT the reference time raised its fast₀.₅ from 42% to 66%, so the gap between Opus and GPT shrinks from 24 to 8 points. From my point of view this shows that part of the measured gap comes from the agent not knowing how far it is from the target, as it can only optimize against its own runtime. I learned something from this result and would like to see it, together with this discussion, in the main paper.

I still have one concern after inspecting the evaluator. All four CCD tasks are checked by one aggregate relative L2 over the times of impact, so a missed collision only reads as toi = 1. In my CPU reproduction of ccd_vv, a small broad-phase change (a 0.95h gap instead of h) missed thousands of collisions and still passed the tolerance. I guess the same would happen on CUDA, but I would ask the authors to check this case and clarify what a correctness pass means for CCD.

Having seen the reference implementations and the new experiments, I am raising my rating from 2 to 3.

We thank the reviewer for the careful review and for inspecting the released benchmark in detail.

Following the suggestion, we have added the prompt experiments and this discussion to the main paper. Section 4.4 now includes a paragraph on stopping behavior and a table comparing the original prompt with the reference-time and keep-optimizing prompts for Opus + Claude Code and GPT + Codex CLI. We've marked the changes in blue.

We thank the reviewer for pointing out the CCD issue, which we confirmed on CUDA. Taking ccd_vv as an example, we have fixed this with a per-vertex check: the evaluator reruns the reference at thickness h(1±10−3)h(1\pm10^{-3}), and every vertex's time of impact must lie between the two results. The time of impact can only decrease as the thickness grows, so this interval contains every correct answer. Any missed or late impact falls outside it, even if it is the only one. We verified that the 0.95h0.95h variant now fails to pass the updated correctness test. ccd_ve uses the same thickness bounds, while ccd_vf and ccd_ee instead widen and narrow the inside test by 10−410^{-4}. A CCD correctness pass therefore now means that every time of impact matches the reference up to this small change of the contact criterion. We re-evaluated all previous agent submissions for the CCD tasks with the updated evaluators: no correctness verdict changed. We have updated the paper (Section 3.2 and the tolerance-validation appendix) and the supplementary material accordingly.

Reviewer_YKrnOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a new benchmark to test whether AI coding agents can write GPU-based physics simulation code that is both correct and fast. It comprises 50 CUDA tasks, including fluids, cloth, snow, granular materials, and collision detection, built on established methods such as PIC/FLIP, MPM, lattice Boltzmann, FEM, XPBD, and DEM. Each agent in the experiment runs in its real coding environment, disabling the ability to search for information on the internet and to use the harness as it may. The testing model includes Claude-Opus-5, GPT-5.6-Sol, Gemini-3.5-Flash, DeepSeek-V4.1-Flash, Qwen-3.8-Max, GLM-5.3. It receives a detailed task description, starter code, an RTX 4090 GPU, and a 30–60 minute time limit. During that window, it writes, compiles, tests, and optimizes its own CUDA code without internet access or help from other agents. Submissions are checked for numerical correctness on hidden test inputs, audited for rule violations, and timed against expert-optimized reference implementations. The authors also validate their correctness thresholds by confirming that planted bugs fail and correct variants pass.

Critical strengths

The paper is well written, and it covers a domain that other benchmarks miss. It evaluates realistic agentic workflows, and the anti-cheating design is thorough.

Critical weaknesses

First of all, the benchmark itself has limited scientific value given how quickly advanced models like GPT and Claude are updated and improved. That limits the contribution to the field

The construction of benchmarks is largely the curation of existing work, as shown below. It would help to articulate more clearly what is methodologically new in adapting KernelBench-style evaluation to the simulation workload.

i) The tasks are textbook methods

ii) The reference comes from public SIGGRAPH code, translated by Claude Fable and then expert-tuned

The findings are largely expected. The main findings are consistent with established intuition from GPU programming. The paper would be more informative if it tested specific hypotheses about why agents fall short, rather than presenting largely expected, almost common-sense observations.

Questions for authors

a) Table 6 shows that at the default budget, Opus and GPT finish after a median of 43% and 50% of the available time, respectively, and never reach the deadline. Yet in Table 2, increasing the stated budget to 1.5× raises Opus’s fast₀.₉ from 22% to 34%, and Section 4.4 attributes this to how agents use a longer stated deadline rather than to a lack of time. This suggests that fast_p, at the default budget, partly reflects each system’s self-termination behavior, which varies widely (e.g., Gemini stops at a median of 25% of the budget, while Qwen hits the deadline on 19 tasks). Could the authors clarify:

 i) how much wall-clock time Opus and GPT actually used at the 0.5× and 1.5× budgets?  ii) how the best-so-far speedup evolves over elapsed time within a run, e.g., by evaluating saved checkpoints at fixed times, which would separate optimization skill from stopping behavior? iii) whether cross-system rankings remain the same when systems are compared at matched elapsed time, or when agents are instructed to keep optimizing until the deadline?

b) Did the agents actually profile the code for speed? Did agents have access to profiling tools such as Nsight Compute or Nsight Systems? If so, how often did they use them, and did using them correlate with better speed? c) Did any agent attempt a stronger algorithm, such as multigrid, and then abandon it? What do the development traces show about why?

2 comments

We appreciate the reviewer's thoughtful questions, which prompted several new analyses below.

W1: Value given rapid model updates

While correctness is nearly saturated, performance is far from it: the strongest system reaches 0.9× the expert reference on only 22% of tasks (fast0.9\text{fast}_{0.9}). The benchmark therefore leaves ample headroom to track progress as models improve.

W2: Methodological novelty

The main methodological contribution is evaluating full, autonomous coding agents in a controlled environment. Prior GPU-code benchmarks either query model APIs directly (e.g., KernelBench) or wrap them in a fixed agentic workflow with scripted generate–verify–profile loops (e.g., TritonGym). In GPUPhysBench, production agents such as Claude Code and Codex CLI plan, edit, compile, test, profile, and decide when to stop on their own. Giving agents this autonomy while keeping results comparable and trustworthy required two components that we designed for this setting:

  1. Agent-facing code framework. GPU timing is tricky: allocation, data transfer, and asynchronous launches can easily blur what is measured. Our fixed interface separates them. The agent implements allocate(), compute(), and release(), while the fixed exec() times only compute() with CUDA events and device synchronization. This measures exactly the simulation's GPU time and leaves agents free in algorithms and data layouts.
  2. Anti-cheating mechanisms. A sandbox hides the references and evaluators. Static checks guard the interface and confirm that allocate() only allocates. LLM audits verify that all algorithmic work runs inside compute(), with none hidden in constructors, host threads, or state cached across calls, and flag network access and parallel agents. These safeguards are a novel and necessary part of the design, since they keep both the results and the timing of the algorithm's execution rigorous over long, autonomous agent sessions.

W3: Expected findings → tested hypotheses

Following the suggestion, we tested a specific hypothesis about why agents fall short.

Hypothesis: Agents know how to implement many algorithms but do not reliably choose the best one for the task. Opus implements a multigrid preconditioner for the pressure Poisson solve in stable_fluids and apic. Yet on poisson_dirichlet, a standalone Poisson solve where multigrid matters most, it uses Jacobi-PCG and spends the rest of the run fusing kernels. Multigrid preconditioner is never mentioned. The choice is also unstable across runs. In GPT's repeated runs on poisson_dirichlet, one chooses multigrid preconditioner from the start, while another starts with Jacobi-PCG and never changes it. The bottleneck is thus choosing the algorithm, not knowing how to implement it.

Q(a): Budget use and stopping behavior

(i) Wall-clock at each budget (median over tasks):

BudgetOpus time (budget used)GPT time (budget used)
0.5×10.5 min (64%)12.1 min (70%)
1.0×15.7 min (43%)18.0 min (50%)
1.5×20.8 min (39%)22.2 min (45%)

(ii) Best-so-far over time. As an example, we provide the optimization trajectory of Opus on apic (60-minute budget, reference 25.0 ms):

TimeGPU timeSpeedupChange
7.6 min411 ms0.06×First working version: global-atomic P2G, Jacobi-PCG
12.2 min227 ms0.11×P2G with tile sorting and shared-memory accumulation
14.8 min91 ms0.27×CG restricted to fluid cells
17.1 min73 ms0.34×Level-set scatter fused into the tiled kernel
28.1 min51 ms0.49×Multigrid-preconditioned CG
32.6 min49 ms0.51×Multigrid tuning (last improvement)
33.6 min55 ms0.46×More smoothing sweeps: slower, reverted
40.4 min49 ms0.51×Submitted after cleanup and robustness tests

(iii) Matched time and "keep optimizing". The ranking holds at matched elapsed time. Reconstructing best-so-far speedups on the tasks, Opus's best-so-far fast0.5\text{fast}_{0.5} is above GPT's at every point (5 / 10 / 20 / 30 min). We also added the sentence "Keep optimizing when there is time left." to the prompt:

ModelSettingPass Ratefast0.5\text{fast}_{0.5}fast0.9\text{fast}_{0.9}fast1.05\text{fast}_{1.05}Median budget used
OpusOriginal100%66%22%0%43%
OpusKeep optimizing100%64%24%0%55%
GPTOriginal100%42%16%0%50%
GPTKeep optimizing98%44%18%0%54%

fastp\text{fast}_p stays within run-to-run variation, and the ranking is unchanged. Agents tend to stop once they reach a plateau, typically right after a final tuning attempt turns out slower and is reverted, as in (ii). Even when told explicitly to keep going, they use only slightly more of the budget. The instruction therefore does not prevent this stopping behavior.

Q(b): Profiling

To run the many experiments in the paper, we rented multiple RTX 4090 machines on the RunPod cloud platform. RunPod does not expose the GPU performance-counter permissions that Nsight Compute requires, so agents in these runs profiled by timing individual stages with CUDA events.

To study the effect of the Nsight Compute profiler, we reran Opus on all 50 tasks on a local RTX 4090 machine where Nsight Compute is fully available. Opus used it successfully on 24 of the 50 tasks, about 3.5 times per task on average. It queried per-kernel GPU time, memory-throughput, cache-traffic, and occupancy metrics.

SettingPass Ratefast0.5\text{fast}_{0.5}fast0.9\text{fast}_{0.9}fast1.05\text{fast}_{1.05}
Original (CUDA-event timing)100%66%22%0%
With Nsight Compute100%64%26%4%

Access to Nsight Compute does not significantly change the results; fastp\text{fast}_p values stay within run-to-run variation.

Q(c): Abandoned multigrid attempts

No, we have not observed such behaviors.

Reviewer_DpFvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GPUPhysBench, a benchmark of 50 CUDA tasks in physical simulation. The tasks cover Eulerian fluids, PIC/FLIP/APIC, PBF, LBM, mass-spring, FEM, MPM, XPBD, CCD and DEM, and range from single operators to complete simulation steps. Each task prescribes the numerical method in a natural-language prompt and gives the agent fixed starter code, which includes the timing scaffold. An expert-optimized CUDA reference, hidden from the agent, serves both as the correctness baseline and as the performance target. Agents run in their production harnesses (Claude Code, Codex CLI, etc.) inside a sandbox on a single RTX 4090, with a 30- or 60-minute wall-clock budget. Submissions are rebuilt and checked on public and hidden inputs, passed through deterministic and LLM-based audits, and scored by pass rate and fastp\mathrm{fast}_p with si=tiref/tigens_i = t^{ref}_i / t^{gen}_i. Across six model–harness pairs, Opus and GPT pass all 50 tasks. Even the best system reaches si>0.9s_i>0.9 on only 22% of tasks, and no system beats the reference by more than 5%.

Critical strengths
  1. GPUPhysBench targets a domain that existing ML-operator kernel benchmarks largely do not cover: physical simulation with irregular memory access and iterative solvers. It may help point to a further direction for developing LLM coding agents.

  2. The references are compared against established libraries: they are 9.0× faster than AMGX and 2.6–7.0× faster than ports of Warp and Taichi examples (App. F). This lends some credibility to the reported performance gaps.

Critical weaknesses
  1. Hiding the reference runtime mixes optimization ability with stopping behavior. The prompt only asks agents to make the code “as fast as you can” and never shows them treft^{ref}. Agents can therefore only optimize against their own measured speed, without knowing the target. The paper’s own data suggest this matters: Opus and GPT use a median of only 43% and 50% of the budget and never hit the deadline (Table 6). Yet with 1.5× the budget, Opus’s fast0.9\mathrm{fast}_{0.9} rises from 22% to 34% (Table 2), above its 22–28% range across three default-budget runs (Table 4). A low fast0.9\mathrm{fast}_{0.9} could thus mean either “cannot reach expert speed” or “stopped early because it did not know the gap”. Giving agents an explicit target (e.g., the reference speed, or a required fraction of it) would leak nothing about the reference implementation itself. It may also produce stronger results, and it would measure the first of these two cases more cleanly.

  2. The reference sets the performance scale for the whole benchmark, but how it was built is not transparent. Beyond noting that Claude Fable translated the upstream code and human experts then rewrote and optimized it (§3.1), the paper says little about how the references were built. It does not report the experts’ effort per task, how much optimization was done, or a hardware-anchored measure of reference quality such as the roofline fraction. Without this, it is hard to tell whether treft^{ref} represents standard expert performance or a heavily tuned upper bound. The answer may also differ across tasks, which would make speedups and category-level conclusions hard to compare.

Questions for authors

Each model is evaluated only in its own harness (§4.1), yet the systems are referred to by model family, and observations such as “Opus is faster than GPT on 38 of the 50 tasks” (§4.2) read as statements about the models. The harnesses differ substantially in how they behave: budget use ranges from a median of 25% to 92%, and input tokens from 1.65M to 7.34M (Table 6). Gemini CLI exposes no reasoning-effort setting, and GLM runs in OpenCode rather than its native environment. Without cross-pairing, the gaps in Table 1 cannot be attributed to either the model or the harness. Could you run at least a small cross-validation, for example Opus and GPT swapped between Claude Code and Codex CLI, or all models in one shared harness such as OpenCode? If the ranking changes substantially, the conclusions should be framed as statements about model–harness systems rather than about models.

1 comment

We thank the reviewer for the constructive comments.

W1: Reference runtime

We added one sentence to each prompt: "For comparison, a reference implementation of this task takes X ms of GPU time on the example input that run_<task>.py builds." Everything else is unchanged.

ModelHarnessSettingPass Ratefast0.5\text{fast}_{0.5}fast0.9\text{fast}_{0.9}fast1.05\text{fast}_{1.05}Median budget used
OpusClaude CodeOriginal100%66%22%0%43%
OpusClaude CodeReference time98%74%30%4%45%
GPTCodex CLIOriginal100%42%16%0%50%
GPTCodex CLIReference time100%66%22%2%61%

How agents decide to stop. In the original runs, agents end development once they reach a plateau, where their own attempts stop yielding gains. The traces repeatedly show the same sequence: a final low-level tweak (e.g., a different block size, launch bound, or cache hint) makes the code slower, the agent restores its previous best version, and it moves straight to verification and submission, even when there is still budget unused.

What the reference time changes. The reference time raises fast0.5\text{fast}_{0.5} by 8 points for Opus + Claude Code and 24 points for GPT + Codex CLI. For example, on poisson_dirichlet, Opus + Claude Code's first Jacobi-PCG takes 551 ms against the 18.7 ms reference. Seeing the 30× gap, it switches to Multigrid-PCG and reaches 0.19×, versus 0.05× without the reference time.

What it does not change. The way agents stop stays the same. Even while knowing they are slower than the target, they still conclude once their tuning plateaus.

W2: Reference quality

Roofline. On LBM tasks, every distribution is read and written once, so the compulsory traffic is exact. Against the RTX 4090's peak of 1008 GB/s, the references reach:

TaskCompulsory trafficReference timeAchieved bandwidth% of peak
stream_d3q192.51 GB2.69 ms933 GB/s93%
stream_d3q273.57 GB3.84 ms929 GB/s92%
trt_collision_d3q192.55 GB2.77 ms920 GB/s91%
trt_collision_d3q273.62 GB3.94 ms919 GB/s91%
lbm_d3q192.51 GB2.76 ms909 GB/s90%
lbm_d3q273.57 GB4.01 ms890 GB/s88%

They run at the bandwidth ceiling. For irregular and solver-bound tasks, where a roofline is less meaningful, Appendix F anchors the references against AMGX (9.0× faster) and Warp/Taichi ports (2.6–7.0×).

Inspectability. The full benchmark is uploaded as Supplementary Material, including every reference, input generator, evaluator, and audit script.

Question: Model vs. harness

We ran both Opus and GPT in OpenCode with identical settings.

ModelHarnessPass Ratefast0.5\text{fast}_{0.5}fast0.9\text{fast}_{0.9}fast1.05\text{fast}_{1.05}Median budget used
OpusClaude Code100%66%22%0%43%
OpusOpenCode100%70%24%6%44%
GPTCodex CLI100%42%16%0%50%
GPTOpenCode100%44%16%0%42%

The ranking is preserved with OpenCode as the shared harness. Following the reviewer's suggestion, we will frame statements as evaluating model–harness systems and add this cross-harness table.

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their responses to your reviews. Please take some time to carefully examine their rebuttals and engage in the discussion by sharing your thoughts, remaining concerns, or follow-up questions.

As the discussion deadline is approaching, I would appreciate your timely participation to allow sufficient time for further exchanges with the authors.

Thank you for your time and valuable contributions.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.