acceptodds
Under review as a conference paper at ICLR 2027

PACE: Measuring LLMs in Performance-Critical Code Generation under Agentic Workflows

Abstract

Code agents can edit files, invoke compilers, run tests, and revise their implementations. Previous works have evaluated speedup over reference implementations. This can surely help people understand how good a piece of code is, but it has a gap between the speedup number and hardware potential as it depends heavily on the quality of the reference for the workload. To further study resource utilization and understand the code quality generated by agents, a shift from workload-centered view to resource-centered view could help build an intuitive view. Therefore, we introduce PACE (erformance-critical gentic ode valuation), a CPU evaluation framework, with a metric called ffective alibrated ttainment (ECA) built on measuring computing resource units. It measures how closely one bounded agent run approaches a calibrated performance reference for each task. We report a five-model baseline on x86-64 and AArch64 containing 3,000 runs. We also report a 135-run x86-64 pilot with nine conditions and DeepSeek-V4.1-Flash. From these assessments, we find that high pass rates can hide substantial performance gaps and that model rankings change across task tracks and hardware platforms. And a larger interaction budget alone may not improve the delivered code. The PACE repository can be accessed at https://anonymous.4open.science/r/PACE-411F

Primary area:
datasets and benchmarks
Keywords:
benchmark, LLM, code generation, agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RP7kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PACE measures LLMs in performance-critical CPU code generation under tool-using agentic workflows. The paper introduces a CPU code benchmark with 100 workflows between implementation from task specification (LeetGen) and optimization of correct code (LeetPort), on x86-64 and AArch64. The paper proposes the ECA metric, which combines correctness with a measured resource reference for each task, constructed from work estimates and measured resource rates. The study reports 3000 runs across five models and a 135-run pilot. The paper shows high correctness rates can coexist with low attainment relative to these constructed references.

Critical strengths
  1. The paper proposes a useful code generation evaluation benchmark. There are already some SOL analysis and evaluation metric design for GPU programs. This paper focuses on performance-critical CPU code generation instead, covering a diverse set of workloads: linear algebra, nn operators, parallel algorithms, cryptographic operations, etc.
  2. The proposed benchmark also separates wrong codes from slow code. Different failures are taken into account: build error, wrong answer, timeouts, invalid candidates. Correct but slow candidates are also scored. This is a finer-grained evaluation than just selecting the best candidates. Thus, the 3000 runs in its benchmark evaluation might serve as a fair baseline.
Critical weaknesses
  1. I think the novelty is limited compared to existing hardware-aware code generation benchmarks. Many tasks discussed in the paper, including neural network operators, attention, sparse linear algebra, and stencil computations, strongly overlap with workloads already well-studied in GPU kernel optimizations. For example, SOL-ExecBench has introduced the principle of evaluating generated code against hardware-aware performance targets. While extending this setting to CPUs is useful, I find the novelty a little limited. This paper does not clearly explain what CPU-specific challenges require a different evaluation methodology, what new insights this setting provides beyond existing GPU-centered benchmarks, what new capabilities LLM might need to improve on such benchmarks. For the selected CPU workloads, the calibrated reference in the paper is also insufficiently validated. Some probes represent relatively complex operations, making the resulting reference dependent on their implementation quality. Any possible failure modes during the probe development should also be explicitly mentioned in the paper to prove the validity of the reference design.
  2. Since the workflows in the paper are quite broad, there are some missing motivations and characterizations of those selected workloads. The term "workflow" is not clearly defined in the abstract or the beginning of the paper. The evaluated tasks seem to be mostly computational kernels rather than complex application workflows. The main paper provides limited explanation of the workflow selection criteria, whether different categories can provide different insights or demonstrate different capabilities of the evaluated models, and what makes the benchmark challenging or representative of practical CPU optimization, the distinction between CPU and GPU workloads. This makes it difficult to assess the significance and generality of the benchmark itself.
Questions for authors
  1. What is the main contribution beyond adapting SOL-centric evaluation to CPUs? Which CPU-specific challenges or findings justify the new benchmark? Some examples will be helpful.
  2. How do the calibrated references compare with strong optimized CPU implementations? How to make sure such references are precise enough for a reliable evaluation?
  3. Can the authors report per-task witness/reference ratios, the magnitude of reference exceedances, and the sensitivity of ECA rankings to different calibration choices?
  4. How representative are the selected tasks of practical CPU programming workloads? Could the authors provide more detailed examples and explain the workflow selection process? Specifically, what are the insights this CPU-centered benchmark can additionally provide compared to GPU-centered benchmarks on the same set of workloads? (Rather than the difference of programming languages, library use, etc.)
7 comments
AuthorsOpenReview ↗

Response to Official Review of Submission31826 by Reviewer RP7k

To Reviewer RP7k

Thank you for your detailed summary of the manuscript. Your feedback and suggestions are very helpful, and we would like to address your questions and comments below.

W1 & Q1: PACE evaluates whether LLMs can realize CPU performance across workloads with different dominant costs. ECA extends hardware-aware evaluation by calibrating operation classes such as vector transcendental computation, dependent factorization, and exact integer transforms. These costs are poorly resolved by a basic compute/bandwidth reference, TSOL=max⁡(F/P,B/β)T_{\mathrm{SOL}}=\max(F/P,B/\beta), even for computations also studied on GPUs: SiLU/Swish in KernelBench, batched Cholesky in cuSOLVER, and Arrhenius rates in GPU chemistry applications such as Pele.

The following measurements use the median of three uniform-input seeds at the public shape. Times are in milliseconds. The witness is the fastest correct implementation in the tested library and handwritten pool; attainment is a=Tref/Twitnessa=T_{\mathrm{ref}}/T_{\mathrm{witness}}. Library comparators are OpenBLAS SGEMM, libmvec Arrhenius/SiLU wrappers, LAPACK SPOTRF, and OpenSSL SHA256 compression.

TaskBasic SOLECA referenceLibraryWitnessAttainment
GEMM7.12787.37588.68478.684784.9%
Arrhenius0.02200.10940.29670.187658.3%
Mixed Cholesky0.30690.75811.64761.313457.7%
SHA256 compression0.28814.472611.21487.379560.6%
SiLU0.16470.32100.77410.406778.9%
Leaky ReLU0.16170.1617—0.173593.2%
GHASH0.03200.1927—0.213790.2%

GEMM is a positive control: regular reuse and independent multiply–add operations make basic SOL informative, with witness/SOL =1.22=1.22. Other workloads expose costs that require finer resource models:

  • Vector mathematics. The CPU SiLU witness implements the central-range exponential through six dependent SIMD FMA steps and reciprocal refinement. NVIDIA GPUs provide an exponential path through the SFU's MUFU.EX2. Calibrating CPU nonlinear throughput reduces the witness/reference distance from 2.472.47 to 1.271.27 for SiLU and from 8.518.51 to 1.711.71 for Arrhenius.

  • Dependencies and SIMD layout. Mixed Cholesky requires square roots, divisions, and dependent triangular updates. The CPU implementation batches independent matrices into SIMD lanes. Its calibrated factorization resource reduces the distance from 4.284.28 to 1.731.73, capturing costs missed by independent FMA throughput.

  • Exact integer computation. SHA256 has no ordinary floating-point work, but each block requires message scheduling and 64 rounds of integer arithmetic and Boolean operations. A calibrated compression resource reduces the distance from 25.6225.62 to 1.651.65.

These examples identify concrete agent capabilities: choosing vector mathematics within the precision contract, finding independent work around dependencies, organizing SIMD layouts, and selecting suitable integer instructions. Probe validation and measured failure cases are reported under Q2/Q3 below.

W2 & Q4: We agree that the term "workflow" is kind of ill-defined, and the current tasks are mostly kernels or snippets simplified rather than a complete program. We will replace workflow with workload for a computational task with an interface, layout, input generator, correctness contract, and timed evaluator. Agent workflow will describe the edit–compile–test–optimize process.

Tasks are constructed from structured specifications of established operations and application patterns. Selection covers different data types, access patterns, dependencies, and resource demands. Admission checks require correct implementations to pass, deliberate wrong-code controls to fail, hidden sizes/seeds to vary, and optimized witnesses to demonstrate stable speedups over simple compiler or parallel-loop baselines.

Examples include LBM and stencils, direct N-body interactions, Arrhenius rates, Thomas solving, Cholesky, SHA256/GHASH, and bitmap statistics. These exercise cache/halo reuse, irregular accesses, numerical dependencies, and exact bit operations.

We compare fixed versions of KernelBench (levels 1–3), SOL-ExecBench, and CUDABench (99 task names, 500 input instances) using the Berkeley 13-dwarf vocabulary. Each check means there is at least one task considering this topic.

Computational patternKernelBenchSOL-ExecBenchCUDABenchPACE
Task definitions/names25023599100
Dense linear algebra✓✓✓✓
Sparse linear algebra—✓✓✓
Spectral methods—✓✓—
N-body methods——✓✓
Structured grids✓✓✓✓
Unstructured grids——✓—
MapReduce✓✓✓✓
Combinational logic———✓
Graph traversal————
Dynamic programming———✓
Backtrack and branch-and-bound————
Graphical models————
Finite state machines———✓
Documented pattern categories3578

PACE combines scientific computation with cryptography, exact packed-data processing, and discrete state handling, broadening the mix beyond neural-network computation.

On shared computations such as GEMM and SiLU, the measurements above distinguish CPU costs that ordinary arithmetic throughput does and does not capture. Recurrence workloads additionally test whether agents can expose independent work for SIMD: two tridiagonal runs with 35 tool calls each produced 0.473 ms and 48.622 ms implementations. The faster code vectorized across independent systems with contiguous accesses; the slower code remained scalar. The CPU evaluation measures these choices alongside cache reuse and instruction selection under a fixed physical-core budget.

Q2 & Q3: Our validation shows that the references are close to strong optimized implementations, model rankings are largely preserved under the tested local calibration changes, and detected probe weaknesses can be repaired through recalibration.

The seven strong implementations in the first table attain 57.7–93.2% of their references, corresponding to witness/reference runtime ratios of 1.073–1.732. OpenBLAS is the fastest tested GEMM implementation; the Arrhenius, Cholesky, SHA256, and SiLU witnesses are 1.251.25–1.90×1.90\times faster than their mature-library counterparts. For GEMM, Arrhenius, Cholesky, and SHA256, both library and witness passed correctness and timing validation on all 24 configurations (four shapes, two distributions, three seeds). SiLU's probe and author witness share the central exponential polynomial; libmvec supplies an additional implementation comparison.

Per-task ratios and exceedances. The accompanying 100-task ledger records reference times, remeasured audit witnesses, fastest admitted main-study candidates, both ratio directions, and source hashes. The median fastest-candidate/reference ratios are 1.351 for LeetGen and 1.412 for LeetPort. The benchmark-wide audit uses frozen main-study references and retains uncapped attainment aa; a correct, timing-admitted attempt contributes min⁡(1,a)\min(1,a) to ECA.

Across 1,500 attempts, 41 exceed the reference on 12 tasks: 2.73% of all attempts, or 3.08% of 1,331 correct, timing-admitted attempts. Their raw attainments fall into (1,1.05](1,1.05], (1.05,1.25](1.05,1.25], (1.25,2](1.25,2], and >2>2 with counts 12, 13, 13, and 3. All three above 2 are INT8 grouped GEMM (7.00657.0065–7.88357.8835); the largest outside this task is GHASH at 1.8517. The old INT8 probe missed VNNI: for one candidate, independent instruction-matched calibration and remeasurement of the same source changed raw attainment from 7.1796 to 0.6164. This is a concrete failure mode involving probe implementation quality.

Probe quality is improved through an agent-assisted iteration: optimize each probe under its operation contract, validate correctness and timing stability, and freeze its implementation and resource rates before evaluation. When stronger valid code reveals a probe weakness, the cycle repeats and produces a new calibration version used consistently across models, with prior versions retained. The INT8 repair above demonstrates this reference-level analogue of recursive self-improvement (RSI). ECA can thus incorporate improved implementation strategies through explicit, versioned calibration updates.

Calibration sensitivity. We held all 1,500 candidate programs and runtimes fixed. Of 11 exp, Cholesky, GHASH, and SHA256 probe variants, eight passed complete timing admission; four deliberately incorrect controls were rejected. The table reports maximum aggregate ECA changes in percentage points (pp) and preservation of the full five-model order.

Calibration choiceMax. absolute ECA change: LeetGen (pp)Max. absolute ECA change: LeetPort (pp)Full order retained: LeetGenFull order retained: LeetPort
Admitted new probes—1.11—Same
All resource rates ±10%\pm10\%5.284.98SameSame
Required work ±10%\pm10\%4.974.82SameSame
Phase/FP32/partial-overlap grouping4.541.26SameSame
Matched arithmetic counts0<0.001<0.001SameSame
Joint rates/work ±10%\pm10\%5.287.5199.7%100%
Joint rates/work ±20%\pm20\% (stress)9.4915.6190.2%98.0%

Grouping changes merge phases, unify scalar/matrix FP32 rates, or vary memory/compute overlap in primitive-only phases. Each joint row uses 1,000 draws with independent uniform rate/work multipliers per resource, shared across tasks and models. At ±10%\pm10\%, both tracks retain their winner in every draw; LeetGen exchanges its second and third models three times. At ±20%\pm20\%, winner retention is 95.6% and 100%, respectively.

Task-level effects can be larger: the order-31 Cholesky probe increases its reference by 2.67×2.67\times relative to the frozen calibration and one model's task ECA by 55.35 pp, producing 1.11 pp at aggregate weight 1/501/50. An all-resource serial-sum stress test changes the LeetGen winner and its score by up to 25.85 pp. For close models, a paired task-bootstrap 95% interval for the LeetGen first–second difference is [−3.65,6.03][-3.65,6.03] pp. These results quantify both local ranking stability and the evaluation's resolution limits.

Reviewer_R36hOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a new benchmark/workflow framework and a new metric for performance evaluation. The workflows test kernel generation from scratch and optimization from existing code, and the new metric evaluates the delta from an "ideal". Performance results across multiple models are rpesented demonstrating how such an approach is better than simple incremental performance improvement tuning over baselines, as well as interesting insights around different workloads, CPU architectures, and parameter tuning.

Critical strengths

I liked the thesis of the paper that we want to measure "attainment towards ideal" as opposed to just speedups. The framework and experiments were also very well thought out.

I think the model could serve as a good eval bench to train improved reasoning for future models.

Critical weaknesses

I would have preferred more discussion on how to think about the new metric. Where does it fail for example? Given some of the simplifying assumptions built in the definition, I can see potential for cases where we could go above 1.0 on this metric - is that even possible?

I liked the insights around reasoning tokens more important than tool calls, but I would have liked a bit more follow up, especially around the choice of tools and their match with the tasks. Similarly, I was a bit surprised by the lack of movement in the results with additional information from the hardware optimization guides.

Overall, the paper was interesting but felt a bit incremental, and I would have liked to see more depth in the analysis and follow-through on how the results led to better outcomes.

Questions for authors

See above.

5 comments
AuthorsOpenReview ↗

Response to Official Review of Submission31826 by Reviewer R36h

To Reviewer R36h

Thank you for your detailed summary of the manuscript. We appreciate your positive feedback and the constructive suggestions you provided, and we would like to address your questions and comments below.

W1 & Q1: Yes—the uncapped reference-to-runtime ratio can exceed 1, and we have observed such cases. ECA measures attainment relative to an independently calibrated resource reference. For a correct program with an admitted runtime, the raw attainment is a=Tref/Tcandidatea=T_{\mathrm{ref}}/T_{\mathrm{candidate}}, while its contribution to ECA is min⁡(1,a)\min(1,a).

For an individual correct program, a contribution of 0.5 means that its runtime is twice the calibrated reference; a contribution of 1 means that it meets or exceeds the current target. The benchmark average combines correctness and performance, with incorrect programs contributing zero. To interpret a low attainment, compare the candidate with a strong witness on the same task. A witness close to the reference demonstrates that the target is practically reachable, and its speedup over the candidate establishes achievable improvement. When even strong witnesses remain far from the reference, its work, dependency, and resource-overlap assumptions need inspection alongside the implementation.

The reference can be too slow when its probes miss a stronger implementation or instruction class, or when a legal algorithm reduces the assumed work through reuse, algebraic transformations, or tolerance-compatible computation. INT8 grouped GEMM provides a concrete example: the frozen reference of 2.635 ms and a correct candidate runtime of 0.367 ms gave raw attainment 7.1796. The old calibration did not match the VNNI implementation class. In a follow-up calibration experiment conducted, an instruction-matched independent probe gave a reference of 0.215 ms; remeasurement of the same candidate source gave 0.349 ms, yielding raw attainment 0.6164.

Exceedances from immature probes can be repaired through iterative calibration. The INT8 example illustrates how reference failures can become calibration feedback. Agents optimize resource probes; we validate correctness and timing stability, then freeze the code and rates before evaluation. A verified exceedance can trigger another round of probe development, followed by validation and a new frozen calibration shared by all compared models. Earlier scores remain associated with their original version. This is a reference-level analogue of recursive self-improvement (RSI): stronger implementations expose weaknesses that guide improvement of the evaluation reference while the score definition stays fixed.

The opposite limitation is an overly optimistic reference. Our new mixed-size Cholesky experiment illustrates this risk: the reference is 0.758 ms, while the fastest tested implementation takes 1.313 ms (57.7% attainment) and LAPACK takes 1.648 ms. The factorization probe operates on already packed, same-size matrices; the full implementation must gather mixed-size inputs, factorize, and restore row-major outputs. Taking the maximum of isolated resource times can underrepresent these sequential layout costs, depressing attainment even for strong code. This motivates size- and layout-matched probes with explicit phase costs for packing, factorization, and output restoration.

W2 & Q2: We followed up by examining tool use and final code in all 135 trajectories from the three-task ablation. Disabling reasoning reduced ECA from 0.4439 to 0.0515 despite similar tool-call total. Tools nevertheless remained useful: the single-turn, reasoning-enabled condition without tools achieved 0.2477.

The tridiagonal solver provides a concrete illustration of task–optimization matching. Two runs each made 35 tool calls. The reasoning-enabled run vectorized across independent systems with contiguous accesses and used OpenMP, achieving 0.473 ms. The reasoning-disabled run retained a scalar, system-by-system implementation and took 48.622 ms.

The hardware-guide results also become clearer at the trajectory and task levels. The file-based guide was accessed in 13/15 runs, whereas the tool-based guide was accessed in only 4/15. Moreover, baseline INT8 implementations already contained VNNI intrinsics in all five runs, indicating overlap between the supplied guidance and optimizations the model already generated. Task-level changes also offset one another: with the file-based guide, INT8 ECA increased from 0.4510 to 0.5263, while n-body and tridiagonal ECA decreased from 0.4050 to 0.3683 and from 0.4756 to 0.4529. These observations motivate a more precise interpretation: effective optimization depends on selecting and realizing transformations suited to the task; additional interactions or access to general hardware advice alone do not ensure improvement.

PACE's main contribution is a CPU optimization benchmark that pairs diverse workloads with independently calibrated references for their dominant operation classes. ECA extends basic FLOP and bandwidth targets to costs such as vector transcendental functions, dependent factorization, and exact integer transforms. These workloads test whether models can expose SIMD parallelism, manage dependencies, and select suitable instruction sequences. The INT8 probe repair above demonstrates how the resulting diagnostics can improve the calibration itself.

We compare the CPU-adapted basic SOL, TSOL=max⁡(F/P,B/β)T_{\mathrm{SOL}}=\max(F/P,B/\beta), with ECA against the same strong implementations. The witness below is the fastest tested correct implementation; attainment is the reference time divided by its runtime.

TaskSOL (ms)ECA (ms)Witness (ms)SOL att.ECA att.
GEMM7.1278037.3757618.68473582.1%84.9%
Arrhenius0.0220470.1094160.18756611.8%58.3%
Mixed Cholesky0.3068660.7581351.31336623.4%57.7%
SHA256 compression0.2880634.4726267.3794683.9%60.6%

GEMM illustrates a case where basic SOL already captures the main cost. For Arrhenius, mixed Cholesky and SHA256, ECA additionally represents exponential evaluation, small-matrix factorization and integer compression, respectively. The resulting targets are closer to the tested witnesses and expose resource costs omitted or insufficiently represented by the basic model. This provides concrete information for evaluating CPU-generated code and designing task-specific agent feedback, while retaining SOL as a useful baseline where its resource model suffices.

Reviewer_R36hOpenReview ↗

Acknowledging author response

Thanks for the additional clarifications.

Reviewer_KUbwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

PACE evaluates LLM agents on performance-critical CPU code generation. It contains 100 workflows in two tracks: implementing code from a stub and optimizing correct but untuned code. The main metric, Effective Calibrated Attainment (ECA), compares candidate runtime with a task-specific reference built from required work and measured resource rates. The paper reports 3,000 runs with five models on x86-64 and AArch64, plus a smaller pilot on agent settings. Results show that high correctness does not always lead to high performance.

Critical strengths
  1. The benchmark has useful engineering depth. Hidden shapes, changed inputs, wrong-code controls, and repeated timing checks make the evaluation stronger than testing only visible examples.
  2. The main experiment is substantial. Three runs per workflow, two CPU platforms, and separate correctness and performance results provide useful evidence about the code agents deliver.
Critical weaknesses
  1. I am not convinced that ECA supports the paper’s main interpretation. The reference is neither a proven lower bound nor a demonstrated achievable runtime, so low ECA alone does not establish how much useful optimization remains.
  2. The claimed advantage over software references needs stronger evidence. Some probes measure complex operations such as Cholesky or SHA-256 blocks, so ECA still depends on software implementation choices and probe granularity.
  3. The two tracks do not isolate generation versus optimization. They use different task sets with different family distributions, so their score differences cannot show the effect of providing correct starting code.
  4. Performance generalization is tested too narrowly. Each workflow uses a fixed timed shape and performance seed, which gives limited evidence that the generated optimization remains useful across sizes and input distributions.
Questions for authors
  1. How close do strong optimized implementations get to each calibrated reference? Please report task-level witness attainment and comparisons with mature libraries where available.
  2. How stable are ECA scores and model rankings under reasonable changes to probe implementations, resource grouping, and required-work assumptions?
  3. Can you compare the same tasks with a stub and with correct untuned code? This would separate the effect of starting information from task difficulty.
  4. Do the performance gains hold across several hidden sizes and input distributions, rather than one timed setting per workflow?
6 comments
AuthorsOpenReview ↗

Response to Official Review of Submission31826 by Reviewer KUbw

To Reviewer KUbw

Thank you for your detailed summary of the manuscript. And we fully understand your concern about ECA and the experiment settings.

W1: ECA aims to provide a more accurate performance estimation for CPU workloads by considering multiple factors beyond just memory and compute resources. We agree that ECA is not a proven lower bound nor a demonstrated achievable runtime, since it is estimating the performance of a program by breaking it into multiple components and comparing these components. For simple programs like GEMM, ECA works well as it is generally a compute-bound program. But for some programs, it is hard to get a deterministic bound, like sparse matrix multiplication, sparse triangular solves, and graph traversal additionally dependent on input structure, intermediate work, and dependency-induced parallelism. And this is a common problem to performance modeling methods.

W2: We agree that software implementation choices and probe granularity can affect the accuracy of ECA. There is some programs that are quite difficult to break down. So larger granularity probes are more of a compromise solution to get a better estimation of the performance. Some additional experiments replying your concerns are presented in response to Q2.

Besides, ECA supports agent-assisted reference improvement, kind of like recursive self-improvement (RSI). Agents first optimize probes for fixed operation contracts; correctness and timing-stability checks precede freezing the probe implementations and resource rates for evaluation. When a validated implementation reveals a weak probe, we improve and revalidate it, then freeze a new calibration version for a uniform comparison across models, retaining the old version. This cycle was exercised for GHASH: improved carryless-multiplication reduction changed the reference from 0.398 to 0.193 ms before the formal feedback runs.

W3: The stub and untuned start contribute little to the final performance distinction. According to some results from other projects, this might be the result of the evolving of frontier models. Some additional experiments concerning this are presented in response to Q3.

W4: Your concern about the generalization for different input shapes and distributions is quite important. Sure there are some programs that are sensitive to input, both shapes and distributions. We have added some additional experiments for this concern in response to Q4.

Q1: We report the results of SOL and ECA on mature-library implementations comparison in Table 1. The results show that while SOL provides a useful baseline for simpler workloads, ECA offers a more accurate representation of performance for complex workloads. This is because ECA takes into account the factors besides memory and compute resources, allowing for a more nuanced understanding of program behavior.

Table 1. Mature-library implementations and calibrated references comparison. All times are in milliseconds; witness attainment is reported as a percentage. The CPU-adapted basic SOL is TSOL=max⁡(F/P,B/β)T_{\mathrm{SOL}}=\max(F/P,B/\beta).

TaskMature librarySOL component: F/PF/PSOL component: B/βB/\betaSOLECALibrary timeBest tested witness timeSOL witness attainmentECA witness attainment
GEMMOpenBLAS SGEMM7.1278030.0512377.1278037.3757618.6847358.68473582.1%84.9%
Arrheniusglibc libmvec wrapper0.0220470.0173510.0220470.1094160.2967130.18756611.8%58.3%
Mixed CholeskyLAPACK SPOTRF via OpenBLAS0.3068660.0687760.3068660.7581351.6475501.31336623.4%57.7%
SHA256 compressionOpenSSL SHA256_Transform0.0000000.2880630.2880634.47262611.2148317.3794683.9%60.6%
GEMVOpenBLAS SGEMV0.1175850.3849980.3849980.3849980.5315300.53153072.4%72.4%
SYRKOpenBLAS SSYRK3.7663990.0256303.7663993.7663994.8082894.80828978.3%78.3%
L2 normOpenBLAS SNRM20.0293960.0672000.0672000.0672000.4579250.07064295.1%95.1%
SiLUglibc libmvec wrapper0.0587930.1647030.1647030.3209770.7715510.40445340.7%79.4%

Q2: The observed model rankings are stable under the tested local calibration changes, although individual task scores can be more sensitive. We evaluated this by changing the calibration while keeping all 1,500 candidate programs, correctness outcomes, and measured runtimes fixed.

We tested alternative probe implementations, resource groupings, and required-work assumptions. Table 2 summarizes the largest absolute ECA changes, in percentage points (pp), and whether the complete five-model ordering was preserved.

Table 2. Sensitivity to calibration choices with fixed candidate programs and runtimes. Score changes are in percentage points; percentages in the joint rows report full-order retention across 1,000 draws.

Calibration changeMax. absolute ECA change: LeetGen (pp)Max. absolute ECA change: LeetPort (pp)Full order retained: LeetGenFull order retained: LeetPort
Admitted probe replacements—1.11—Unchanged
Resource rates ±10%\pm10\%5.284.98UnchangedUnchanged
Required work ±10%\pm10\%4.974.82UnchangedUnchanged
Phase/FP32/partial-overlap grouping4.541.26UnchangedUnchanged
Joint rate/work perturbations ±10%\pm10\%5.287.5199.7%100%
Joint perturbations ±20%\pm20\% (stress test)9.4915.6190.2%98.0%

Probe and modeling choices. We measured 11 probe variants covering exponential evaluation, Cholesky, GHASH, and SHA256, including different implementations, SIMD widths, and granularities. These resources affect LeetPort; admitted replacements preserved its model ordering. Grouping tests merged phases, changed scalar/matrix FP32 rate assignments, and varied memory–compute overlap in primitive-only phases. Work tests varied declared total, memory, or arithmetic work by ±10%\pm10\%.

Joint perturbations. We also varied resource rates and work simultaneously in 1,000 draws per range, using independent uniform multipliers per resource shared across tasks and models. Under ±10%\pm10\% perturbations, both tracks retained their first-ranked model in every draw. The only ordering changes were three exchanges between LeetGen's second and third models.

Task-level sensitivity. A smaller-order Cholesky probe increased one model's task ECA by 55.35 pp. Because each task contributes equally to a 50-task average, this changes aggregate ECA by only 55.35/50≈1.1155.35/50\approx1.11 pp. This explains why substantial sensitivity on one task can coexist with stable aggregate rankings. We therefore report task-level effects alongside aggregate scores and use a common frozen calibration version for every model comparison.

Q3: We constructed a common subset of eight LeetGen and eight LeetPort tasks and evaluated every task from both an empty stub and correct untuned C code. GPT-6.1-Sol, Claude-Opus-5.5 and DeepSeek Flash each received three independent attempts per task and start, yielding 288 final programs. Within each task and model, prompts, tests, runtime feedback, resource allocation, tool access and the 30-round/60-call budget were held fixed. Only the initial target source differed.

ModelCorrect / 48: StubCorrect / 48: UntunedECA: StubECA: UntunedUntuned–stub (pp): DifferenceUntuned–stub (pp): 95% CI
gpt-6.1-sol48480.68490.6706−1.43-1.43[−7.92,+3.80][-7.92,+3.80]
claude-opus-5.540430.53400.5418+0.78+0.78[−8.02,+9.09][-8.02,+9.09]
deepseek-4.1-flash42410.68280.6209−6.19-6.19[−16.38,+4.84][-16.38,+4.84]

On this matched task subset, correct untuned code showed no consistent advantage in final ECA across the three models, with all paired 95% confidence intervals including zero.

Q4: To understand whether performance gains still hold across hidden sizes and input distributions, we evaluated all 72 final programs taken from Q3 on four x86 workloads: GEMM and Leaky ReLU from LeetGen, and Arrhenius and GHASH from LeetPort. Each task has 18 programs: three models, two starts and three attempts. Every program was tested on three hidden shapes, three input distributions and two seeds.

The table below shows programs' behavior with different input, both in shape and distribution. All originally correct programs remained correct on every hidden input.

TaskEvaluatedCorrect on all 18 inputsFailed on any inputMinimum speedup
GEMM1818011.39×11.39\times
Leaky ReLU181802.85×2.85\times
Arrhenius181448.67×8.67\times
GHASH1817138.69×38.69\times

Same-code comparison across shapes. The same optimized code retains substantial speedups across the tested shapes and input distributions. S1–S3 report geometric-mean speedups at increasing workload sizes, using the same correct programs, distributions, and seeds. The last column reports the median of the programs' max/min ratios across shapes, computed after geometrically averaging each program's speedups over distributions and seeds to show speedup gains' consistency. Gains are particularly consistent for GEMM and GHASH; Arrhenius and Leaky ReLU show greater size dependence while remaining substantially faster than the untuned baseline.

TaskCorrect programs / evaluatedS1S2S3max/min
GEMM18/1852.69×52.69\times58.67×58.67\times59.48×59.48\times1.13
Leaky ReLU18/1831.30×31.30\times29.18×29.18\times13.20×13.20\times2.39
Arrhenius14/1822.87×22.87\times29.24×29.24\times33.68×33.68\times1.49
GHASH17/18218.58×218.58\times232.35×232.35\times206.01×206.01\times1.12

Table 3 lists the hidden shapes and input distributions.

Table 3. Hidden-input settings for the generalization experiment. Each shape uses all three distributions and two seeds. For GHASH, blocks are counted per message and keys are random.

Task / shape parametersS1S2S3Input distributions
GEMM (M,N,K)(M,N,K)(257,383,509)(257,383,509)(769,1021,511)(769,1021,511)(1153,1025,1031)(1153,1025,1031)Uniform; bimodal; 90%-zero
Leaky ReLU length1,048,5831,048,5834,194,3114,194,31112,582,92912,582,929Uniform; 95%-positive; 95%-negative
Arrhenius (cells, reactions)(4099,31)(4099,31)(15367,67)(15367,67)(24593,97)(24593,97)Uniform; endpoint-concentrated; central-range
GHASH (messages, blocks)(8191,5)(8191,5)(32771,17)(32771,17)(65521,29)(65521,29)Uniform; 90%-zero; high Hamming weight

For 16 of 18 GEMM programs and 14 of 17 GHASH programs, the largest shape-wise speedup is at most 25% higher than the smallest. Arrhenius's geometric-mean speedup increases from 22.87×22.87\times to 33.68×33.68\times with workload size. Leaky ReLU's decreases from 31.30×31.30\times to 13.20×13.20\times, likely because larger inputs make execution more memory-bandwidth-bound, reducing the relative benefit of SIMD optimization. These results show that the optimizations retain substantial speedups across the tested sizes and distributions, with greater size dependence for Arrhenius and Leaky ReLU.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.