- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PACE measures LLMs in performance-critical CPU code generation under tool-using agentic workflows. The paper introduces a CPU code benchmark with 100 workflows between implementation from task specification (LeetGen) and optimization of correct code (LeetPort), on x86-64 and AArch64. The paper proposes the ECA metric, which combines correctness with a measured resource reference for each task, constructed from work estimates and measured resource rates. The study reports 3000 runs across five models and a 135-run pilot. The paper shows high correctness rates can coexist with low attainment relative to these constructed references.
Critical strengths
- The paper proposes a useful code generation evaluation benchmark. There are already some SOL analysis and evaluation metric design for GPU programs. This paper focuses on performance-critical CPU code generation instead, covering a diverse set of workloads: linear algebra, nn operators, parallel algorithms, cryptographic operations, etc.
- The proposed benchmark also separates wrong codes from slow code. Different failures are taken into account: build error, wrong answer, timeouts, invalid candidates. Correct but slow candidates are also scored. This is a finer-grained evaluation than just selecting the best candidates. Thus, the 3000 runs in its benchmark evaluation might serve as a fair baseline.
Critical weaknesses
- I think the novelty is limited compared to existing hardware-aware code generation benchmarks. Many tasks discussed in the paper, including neural network operators, attention, sparse linear algebra, and stencil computations, strongly overlap with workloads already well-studied in GPU kernel optimizations. For example, SOL-ExecBench has introduced the principle of evaluating generated code against hardware-aware performance targets. While extending this setting to CPUs is useful, I find the novelty a little limited. This paper does not clearly explain what CPU-specific challenges require a different evaluation methodology, what new insights this setting provides beyond existing GPU-centered benchmarks, what new capabilities LLM might need to improve on such benchmarks. For the selected CPU workloads, the calibrated reference in the paper is also insufficiently validated. Some probes represent relatively complex operations, making the resulting reference dependent on their implementation quality. Any possible failure modes during the probe development should also be explicitly mentioned in the paper to prove the validity of the reference design.
- Since the workflows in the paper are quite broad, there are some missing motivations and characterizations of those selected workloads. The term "workflow" is not clearly defined in the abstract or the beginning of the paper. The evaluated tasks seem to be mostly computational kernels rather than complex application workflows. The main paper provides limited explanation of the workflow selection criteria, whether different categories can provide different insights or demonstrate different capabilities of the evaluated models, and what makes the benchmark challenging or representative of practical CPU optimization, the distinction between CPU and GPU workloads. This makes it difficult to assess the significance and generality of the benchmark itself.
Questions for authors
- What is the main contribution beyond adapting SOL-centric evaluation to CPUs? Which CPU-specific challenges or findings justify the new benchmark? Some examples will be helpful.
- How do the calibrated references compare with strong optimized CPU implementations? How to make sure such references are precise enough for a reliable evaluation?
- Can the authors report per-task witness/reference ratios, the magnitude of reference exceedances, and the sensitivity of ECA rankings to different calibration choices?
- How representative are the selected tasks of practical CPU programming workloads? Could the authors provide more detailed examples and explain the workflow selection process? Specifically, what are the insights this CPU-centered benchmark can additionally provide compared to GPU-centered benchmarks on the same set of workloads? (Rather than the difference of programming languages, library use, etc.)
7 comments
W2 & Q4: We agree that the term "workflow" is kind of ill-defined, and the current tasks are mostly kernels or snippets simplified rather than a complete program. We will replace workflow with workload for a computational task with an interface, layout, input generator, correctness contract, and timed evaluator. Agent workflow will describe the edit–compile–test–optimize process.
Tasks are constructed from structured specifications of established operations and application patterns. Selection covers different data types, access patterns, dependencies, and resource demands. Admission checks require correct implementations to pass, deliberate wrong-code controls to fail, hidden sizes/seeds to vary, and optimized witnesses to demonstrate stable speedups over simple compiler or parallel-loop baselines.
Examples include LBM and stencils, direct N-body interactions, Arrhenius rates, Thomas solving, Cholesky, SHA256/GHASH, and bitmap statistics. These exercise cache/halo reuse, irregular accesses, numerical dependencies, and exact bit operations.
We compare fixed versions of KernelBench (levels 1–3), SOL-ExecBench, and CUDABench (99 task names, 500 input instances) using the Berkeley 13-dwarf vocabulary. Each check means there is at least one task considering this topic.
| Computational pattern | KernelBench | SOL-ExecBench | CUDABench | PACE |
|---|---|---|---|---|
| Task definitions/names | 250 | 235 | 99 | 100 |
| Dense linear algebra | ✓ | ✓ | ✓ | ✓ |
| Sparse linear algebra | — | ✓ | ✓ | ✓ |
| Spectral methods | — | ✓ | ✓ | — |
| N-body methods | — | — | ✓ | ✓ |
| Structured grids | ✓ | ✓ | ✓ | ✓ |
| Unstructured grids | — | — | ✓ | — |
| MapReduce | ✓ | ✓ | ✓ | ✓ |
| Combinational logic | — | — | — | ✓ |
| Graph traversal | — | — | — | — |
| Dynamic programming | — | — | — | ✓ |
| Backtrack and branch-and-bound | — | — | — | — |
| Graphical models | — | — | — | — |
| Finite state machines | — | — | — | ✓ |
| Documented pattern categories | 3 | 5 | 7 | 8 |
PACE combines scientific computation with cryptography, exact packed-data processing, and discrete state handling, broadening the mix beyond neural-network computation.
On shared computations such as GEMM and SiLU, the measurements above distinguish CPU costs that ordinary arithmetic throughput does and does not capture. Recurrence workloads additionally test whether agents can expose independent work for SIMD: two tridiagonal runs with 35 tool calls each produced 0.473 ms and 48.622 ms implementations. The faster code vectorized across independent systems with contiguous accesses; the slower code remained scalar. The CPU evaluation measures these choices alongside cache reuse and instruction selection under a fixed physical-core budget.
Q2 & Q3: Our validation shows that the references are close to strong optimized implementations, model rankings are largely preserved under the tested local calibration changes, and detected probe weaknesses can be repaired through recalibration.
The seven strong implementations in the first table attain 57.7–93.2% of their references, corresponding to witness/reference runtime ratios of 1.073–1.732. OpenBLAS is the fastest tested GEMM implementation; the Arrhenius, Cholesky, SHA256, and SiLU witnesses are – faster than their mature-library counterparts. For GEMM, Arrhenius, Cholesky, and SHA256, both library and witness passed correctness and timing validation on all 24 configurations (four shapes, two distributions, three seeds). SiLU's probe and author witness share the central exponential polynomial; libmvec supplies an additional implementation comparison.
Per-task ratios and exceedances. The accompanying 100-task ledger records reference times, remeasured audit witnesses, fastest admitted main-study candidates, both ratio directions, and source hashes. The median fastest-candidate/reference ratios are 1.351 for LeetGen and 1.412 for LeetPort. The benchmark-wide audit uses frozen main-study references and retains uncapped attainment ; a correct, timing-admitted attempt contributes to ECA.
Across 1,500 attempts, 41 exceed the reference on 12 tasks: 2.73% of all attempts, or 3.08% of 1,331 correct, timing-admitted attempts. Their raw attainments fall into , , , and with counts 12, 13, 13, and 3. All three above 2 are INT8 grouped GEMM (–); the largest outside this task is GHASH at 1.8517. The old INT8 probe missed VNNI: for one candidate, independent instruction-matched calibration and remeasurement of the same source changed raw attainment from 7.1796 to 0.6164. This is a concrete failure mode involving probe implementation quality.
Probe quality is improved through an agent-assisted iteration: optimize each probe under its operation contract, validate correctness and timing stability, and freeze its implementation and resource rates before evaluation. When stronger valid code reveals a probe weakness, the cycle repeats and produces a new calibration version used consistently across models, with prior versions retained. The INT8 repair above demonstrates this reference-level analogue of recursive self-improvement (RSI). ECA can thus incorporate improved implementation strategies through explicit, versioned calibration updates.
Calibration sensitivity. We held all 1,500 candidate programs and runtimes fixed. Of 11 exp, Cholesky, GHASH, and SHA256 probe variants, eight passed complete timing admission; four deliberately incorrect controls were rejected. The table reports maximum aggregate ECA changes in percentage points (pp) and preservation of the full five-model order.
| Calibration choice | Max. absolute ECA change: LeetGen (pp) | Max. absolute ECA change: LeetPort (pp) | Full order retained: LeetGen | Full order retained: LeetPort |
|---|---|---|---|---|
| Admitted new probes | — | 1.11 | — | Same |
| All resource rates | 5.28 | 4.98 | Same | Same |
| Required work | 4.97 | 4.82 | Same | Same |
| Phase/FP32/partial-overlap grouping | 4.54 | 1.26 | Same | Same |
| Matched arithmetic counts | 0 | Same | Same | |
| Joint rates/work | 5.28 | 7.51 | 99.7% | 100% |
| Joint rates/work (stress) | 9.49 | 15.61 | 90.2% | 98.0% |
Grouping changes merge phases, unify scalar/matrix FP32 rates, or vary memory/compute overlap in primitive-only phases. Each joint row uses 1,000 draws with independent uniform rate/work multipliers per resource, shared across tasks and models. At , both tracks retain their winner in every draw; LeetGen exchanges its second and third models three times. At , winner retention is 95.6% and 100%, respectively.
Task-level effects can be larger: the order-31 Cholesky probe increases its reference by relative to the frozen calibration and one model's task ECA by 55.35 pp, producing 1.11 pp at aggregate weight . An all-resource serial-sum stress test changes the LeetGen winner and its score by up to 25.85 pp. For close models, a paired task-bootstrap 95% interval for the LeetGen first–second difference is pp. These results quantify both local ranking stability and the evaluation's resolution limits.
Response to Official Review of Submission31826 by Reviewer RP7k
To Reviewer RP7k
Thank you for your detailed summary of the manuscript. Your feedback and suggestions are very helpful, and we would like to address your questions and comments below.
W1 & Q1: PACE evaluates whether LLMs can realize CPU performance across workloads with different dominant costs. ECA extends hardware-aware evaluation by calibrating operation classes such as vector transcendental computation, dependent factorization, and exact integer transforms. These costs are poorly resolved by a basic compute/bandwidth reference, TSOL=max(F/P,B/β), even for computations also studied on GPUs: SiLU/Swish in KernelBench, batched Cholesky in cuSOLVER, and Arrhenius rates in GPU chemistry applications such as Pele.
The following measurements use the median of three uniform-input seeds at the public shape. Times are in milliseconds. The witness is the fastest correct implementation in the tested library and handwritten pool; attainment is a=Tref/Twitness. Library comparators are OpenBLAS SGEMM, libmvec Arrhenius/SiLU wrappers, LAPACK SPOTRF, and OpenSSL SHA256 compression.
GEMM is a positive control: regular reuse and independent multiply–add operations make basic SOL informative, with witness/SOL =1.22. Other workloads expose costs that require finer resource models:
Vector mathematics. The CPU SiLU witness implements the central-range exponential through six dependent SIMD FMA steps and reciprocal refinement. NVIDIA GPUs provide an exponential path through the SFU's
MUFU.EX2. Calibrating CPU nonlinear throughput reduces the witness/reference distance from 2.47 to 1.27 for SiLU and from 8.51 to 1.71 for Arrhenius.Dependencies and SIMD layout. Mixed Cholesky requires square roots, divisions, and dependent triangular updates. The CPU implementation batches independent matrices into SIMD lanes. Its calibrated factorization resource reduces the distance from 4.28 to 1.73, capturing costs missed by independent FMA throughput.
Exact integer computation. SHA256 has no ordinary floating-point work, but each block requires message scheduling and 64 rounds of integer arithmetic and Boolean operations. A calibrated compression resource reduces the distance from 25.62 to 1.65.
These examples identify concrete agent capabilities: choosing vector mathematics within the precision contract, finding independent work around dependencies, organizing SIMD layouts, and selecting suitable integer instructions. Probe validation and measured failure cases are reported under Q2/Q3 below.