acceptodds
Under review as a conference paper at ICLR 2027

KernelShift: Benchmarking Kernel Optimization Across Evolving Hardware

Abstract

Efficient AI computation depends on hardware that *evolves over time*, shifting the strategies needed to implement performant code. For CPU kernels, a key evolving factor is the set of available *architectural extensions*, which expose specialized hardware instructions that change the techniques used to optimize kernels. We study how effectively coding agents use these new, potentially unfamiliar extensions to write fast code. We introduce **KernelShift**, a benchmark for evaluating AI coding agents on kernel optimization across evolving CPU architectures. We design an agent-assisted, human-verified pipeline for transforming code repositories into benchmark instances, together with infrastructure that lets agents iteratively optimize code using feedback from the target hardware. Using this pipeline, we construct 85 kernels from real-world, actively developed repositories (ncnn, llama.cpp, Arm SIMD Loops, and KleidiAI), including inference workloads derived from real model executions. KernelShift supports incorporating new kernels and hardware configurations as libraries and processors evolve. We evaluate agents on proprietary and open-weight models across progressively newer architectural extensions (Arm Neon, SVE, SVE2, and SME2), examining how performance changes as hardware evolves. On established ISAs, agents exploit residual optimization headroom, with the strongest system achieving a **2.74× geometric-mean speedup** over the ncnn production reference on SVE. In end-to-end inference of two LLMs, agent-written kernels achieve a **1.3× prefill speedup** over llama.cpp's fast path. However, current agents do not yet reliably adapt to new extensions that require unfamiliar programming models, such as SME2's matrix tiles.

Primary area:
datasets and benchmarks
Keywords:
coding agents, kernel optimization, hardware adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_g5VHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

KernelShift benchmarks AI coding agents on CPU kernel optimization across Arm architectural extensions, including Neon, SVE, SVE2, and SME2. An agent-assisted, human-verified pipeline builds 85 kernels from ncnn, llama.cpp, Arm SIMD Loops, and KleidiAI, with workloads largely traced from real model executions. Agents optimize scalar C++ implementations using compilation, correctness, timing, and disassembly feedback within a tool-call budget. Across six models and four harnesses, strong agents beat production references on established ISAs, reaching up to 2.74× on ncnn, and deliver 1.3× end-to-end LLM prefill speedups. However, agents struggle with SME2 even if it was proposed in 2021, which predated all the evaluated AI coding agents.

Critical strengths

The paper proposes a realistic and hardware-aware benchmark with demonstrated practical value. The tasks come from production libraries (ncnn, llama.cpp, SIMD Loops, KleidiAI), and ncnn and llama.cpp workloads are traced from real model executions. A harness-agnostic server enables separating model effects from harness effects across 14 harness–model pairs.

Critical weaknesses
  1. The headline speedups are hard to interpret; the readers can not tell whether a result means the agent is near the hardware limit or the reference is far from it.
  2. The central “evolving hardware” claim seems weak. (1) The only documentation tested (SME programming notes, in the Table 5 ablation) omits the streaming annotation the harness requires. This means LLMs' performance on evolving hardware can be unfairly constrained by the context but not by the model's capability. (2) The intrinsic-usage audit is weak. It only checks whether certain intrinsics appear in the final source but not in the binary that actually executes. The intrinsic-usage audit covers just the two Claude models, which degrade least, and is not tied to per-kernel performance.
Questions for authors
  1. For a few representative kernels, what fraction of hardware peak do the agent and baseline implementations reach? This can provide a better estimation of the headroom.
  2. Beyond the ISA name in the task prompt, what do agents know about the SME2 target, such as vector length, required streaming attributes, or compile flags? How will the agents behaviors change if providing more correct and detailed documentation and/or examples?
2 comments
AuthorsOpenReview ↗

Response to Reviewer g5VH

We thank the reviewer for the careful reading and the concrete suggestions on documentation and auditing. Please refer to our detailed responses below:

W1. The headline speedups are hard to interpret; the readers can not tell whether a result means the agent is near the hardware limit or the reference is far from it.

Q1. For a few representative kernels, what fraction of hardware peak do the agent and baseline implementations reach? This can provide a better estimation of the headroom.

W1 & Q1:

We measured the fraction of hardware peak for kernels from Claude Opus 4.8 with the nanobot harness on Graviton3 single core:

KernelAgentExpert-O3
KleidiAI FP32 GEMM 512×51256%68%5%
ncnn FP32 3×3 conv65%26%3%
KleidiAI FP32 NHWC 3×3 conv64%71%19%

Appendix D gives this fraction for every Table 2 cell. The ncnn expert is far from peak and leaves space for agents to improve it, while the KleidiAI SVE experts are near peak and stay ahead. So the large speedups are aligned with the headroom in the reference.

W2 (1). The only documentation tested (SME programming notes, in the Table 5 ablation) omits the streaming annotation the harness requires. This means LLMs' performance on evolving hardware can be unfairly constrained by the context but not by the model's capability.

Q2. Beyond the ISA name in the task prompt, what do agents know about the SME2 target, such as vector length, required streaming attributes, or compile flags? How will the agents behaviors change if providing more correct and detailed documentation and/or examples?

W2 (1) & Q2:

We agree.

  • In the main experiments (Tables 2 and 4), agents receive the same information under every ISA: only the ISA name (sve / sve2 / sme2) is given, so the comparison across ISAs is fair.
  • The original document for SME2 is a general SME guide from Arm. To make it more specific, we put common SME2 pitfalls with examples in the task prompt of nanobot with GPT-5.6-luna (SME2 pitfalls in the prompt), and tested on the 20 KleidiAI SME2 kernels. The geometric mean rises from 0.083 to 0.167: 11 kernels improve (7 of them by 7–22×), 4 are unchanged, and 5 get worse. For those 5, the agent’s SME2 code is slower than its earlier implementation, yet the guidance pushes it toward SME2. The pitfalls in the prompt help, but they can't make up for a lack of experience with a new ISA, and they can narrow what the agent explores. We updated the ablation studies in the paper to the new setting.

W2 (2). The intrinsic-usage audit is weak. It only checks whether certain intrinsics appear in the final source but not in the binary that actually executes. The intrinsic-usage audit covers just the two Claude models, which degrade least, and is not tied to per-kernel performance.

W2 (2): We agree, and we have improved the audit as follows:

  • Binary, not source: we now verify use in the binary that runs: every kernel we count as using an extension also has that extension’s instructions in its binary. This agrees with the original source check except for cases where intrinsics are named only in comments, which the binary check does not count (two Claude Sonnet 4.6 kernels on SME2). The ISA ladder (Table 3) already works this way.
  • More models: We include DeepSeek-V4-Pro and GLM-5.3 in the appendix. On the 6 SME2 kernels of the new Table 4 (three seeds), DeepSeek-V4-Pro uses ZA on 1 of 18, and GLM-5.3 uses it on 5 of 18 (Claude Opus 4.8: 15, Claude Sonnet 4.6: 5).
  • Tied to performance: We additionally evaluate the correlation between the number of target-ISA instructions in the binary and the speedup. As shown in the following table, using the target ISA is only weakly related to per-kernel speedup on SVE and SVE2 (Spearman ρ 0.17 and 0.30). On SME2 the relation is stronger (ρ 0.62): final kernels that use ZA reach 0.56× the expert on average, against 0.12× for those that do not, so much of the SME2 gap comes from not reaching ZA.
ISASVESVE2SME2
Spearman ρ0.170.300.62

Thank you for providing additional evidence and clarifications. The hardware utilization measurements help clarify the reported speedups. The additional binary-level audit and documentation ablation alleviate several of my concerns. However, I remain concerned that the evidence does not clearly distinguish LLMs' intrinsic limitations in adapting to evolving hardware from the effects of task context and guidance. Therefore, I will keep my score.

Reviewer_DmAwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

KernelShift is a benchmark for evaluating coding agents that optimize CPU kernels as hardware evolves. It constructs 85 tasks from ncnn, llama.cpp, Arm SIMD Loops, and KleidiAI, provides a scalar starting implementation and hardware-in-the-loop feedback, and evaluates the best correct implementation found within a fixed tool-call budget. The benchmark focuses on Arm’s progression from Neon to SVE, SVE2, and SME2. The paper evaluates several models and agent harnesses, reports large gains over some production references on SVE, and argues that agents struggle when a new extension changes the programming model rather than incrementally extending an existing vector ISA. It also includes a case study in which shape-specialized GEMM kernels improve llama.cpp inference on Graviton4.

Critical strengths

The benchmark addresses an important gap. Existing kernel benchmarks usually treat hardware as fixed, whereas real optimization systems must adapt as compiler targets, instruction sets, and programming interfaces evolve.

The task design is substantially more realistic than a collection of synthetic intrinsics exercises. Kernels come from active projects, many workload shapes are derived from model executions, and agents receive real compilation, correctness, timing, and disassembly feedback from the target machine.

The paper is unusually transparent about benchmark construction. It documents the scalar starter kernels, expert references, hidden workloads, correctness tolerances, instruction-set restrictions, and the exact subsets used by each experiment.

The paper does not simply report agent wins. The analysis of expert headroom is useful: it correctly shows that high speedups against a production reference can reflect an under-optimized reference rather than exceptional agent performance. The failure cases on SME2 are also more informative than another benchmark leaderboard.

The end-to-end llama.cpp study is a useful sanity check. It makes clear that the observed gains come from deployment-time specialization rather than a general claim that agents outperform all expert libraries.

Critical weaknesses

The main conclusion about adaptation to evolving hardware is not cleanly identified. The SVE-to-SME2 comparison changes several variables at once: Graviton3 is replaced by Apple M4, the compiler and operating system change, the expert reference changes, and the comparison goes from three SVE kernels to seven SME2 kernels. The drop in Table 4 may be related to SME2, but the experiment does not isolate the programming-model shift from differences in hardware, task difficulty, or reference quality.

The fixed-machine SME2 ladder is also too narrow to support the broader conclusion. It covers only seven KleidiAI kernels, and the agent is explicitly instructed to stay on the target ISA rather than fall back to an older implementation. In this setting, exposing streaming mode and ZA can make agents attempt incorrect SME code and lose performance. That is an interesting failure mode, but it does not establish that agents intrinsically adapt worse to all new programming models.

The benchmark’s strongest headline results are heavily affected by reference quality. The ncnn expert reference is a Neon implementation evaluated on an SVE target, so beating it by 2.74x does not show that an agent surpasses a strong SVE production kernel. The paper acknowledges this issue in Section 4.3, but the abstract and introduction still foreground these numbers. Results should be reported more prominently relative to scalar code, compiler baselines, and generation-matched expert kernels.

Benchmark construction involves substantial judgment calls: kernel selection, representative subsets, workload synthesis for SIMD Loops and KleidiAI, and removal of expert references that do not meet the authors’ criteria. These choices may be reasonable, but they can introduce selection bias. In particular, the benchmark is built from public repositories and some source families have only a few kernels on the relevant target. The current contamination discussion does not rule out memorized algorithms or source-specific optimization strategies.

Most model and harness comparisons appear to be based on one optimization run per task. The paper reports three-seed variation only for nanobot with GPT-5.6-luna on one subset, where per-kernel Speedup@40 varies by a median of 9.7%. That is large enough to affect several rankings in Table 2. The paper needs repeated runs and confidence intervals for the main model-harness comparisons, especially on the small KleidiAI subsets.

A common tool-call budget is not necessarily a common optimization budget. The harnesses use different default settings, context-management strategies, temperatures, and reasoning configurations. Tool calls also hide large differences in token use and wall-clock time. The paper should either present these results strictly as end-to-end system comparisons or add matched token/time-budget experiments to support claims about model capability.

The end-to-end study is encouraging but limited. It uses a separate model-harness pair, replaces only 18 fixed-shape GEMMs, and compares against general llama.cpp kernels. This demonstrates the value of specialization, not that KernelShift performance reliably predicts general deployment performance. Generation cost, dispatch overhead, and performance on unseen shapes are not reported.

Questions for authors

Can the authors isolate the SME2 claim on a fixed machine and a matched task set, while varying only the instructions exposed to the agent? The current SVE-to-SME2 comparison changes both the hardware and the set of kernels.

For every main result, can the paper report performance relative to three baselines separately: scalar code, compiler-optimized scalar code, and a generation-matched expert implementation? This would make the large gains over ncnn and llama.cpp easier to interpret.

How many candidate kernels were excluded because the source expert failed to build, failed correctness, or was more than 2x slower than the compiler baseline? Please provide the excluded task list and the reason for each exclusion.

Can the authors report repeated-run statistics for all main model-harness pairs, including confidence intervals for geometric mean speedup and the fraction of kernels that beat the reference?

How sensitive are the results to the workload-construction procedure? In particular, what happens on held-out shapes, new model traces, and alternative manually designed workloads for SIMD Loops and KleidiAI?

How do the authors address contamination beyond non-verbatim code reproduction? A stronger test would include kernels from post-training-cutoff commits, private workloads, or newly introduced extensions that are unlikely to appear in model training data.

Can the end-to-end study report the cost of generating and maintaining the 18 specialized kernels, the dispatch mechanism, and performance on shapes not used during kernel generation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer DmAw [1/3]

We thank the reviewer for the detailed and constructive review. Please refer to our detailed responses below:

W1. The main conclusion about adaptation to evolving hardware is not cleanly identified. The SVE-to-SME2 comparison changes several variables at once ... The drop in Table 4 may be related to SME2, but the experiment does not isolate the programming-model shift from differences in hardware, task difficulty, or reference quality.

Q1. Can the authors isolate the SME2 claim on a fixed machine and a matched task set, while varying only the instructions exposed to the agent? ...

W1 & Q1:

The original Table 4 changes several variables at once. We agree that strictly controlling the hardware platform and compiler is important but technically hard, since the expert baselines are often not available on one side. We managed to add two tests:

(1) Fixed machine and task set, and only the instructions change (Q1). We reran the Apple M4 ladder on all 20 KleidiAI SME2 kernels (previously 7). All arms share the machine, compiler, kernels, workloads, and expert reference. So only the allowed instructions change:

Arm (Apple M4, 20 kernels)Speedup@40
L4: Neon only0.127
L5: + streaming SVE (no ZA)0.057
L6: + ZA (full SME2)0.083

Neon only scores highest. In addition, with 3 seeds on 10 of these kernels, L6 is below L4 in every seed (L6 / L4 geometric mean: 0.70, 0.54, 0.54). So on a fixed machine and task set, exposing SME2 lowers agent performance.

(2) Cross-generation on matched kernels. We added KleidiAI SVE tasks so that both sides of Table 4 use the same six operators, each with an expert on both generations, and only the machine and its generation-matched expert differ. With GPT-5.6-luna, the model of the ladder in (1), Speedup@40 drops from 0.178 on SVE to 0.073 on SME2 (0.41×), and on the 3 exactly matched pairs the drop appears in every seed (0.30, 0.19, 0.24). DeepSeek-V4-Pro and GLM-5.3 also drop by about 2×, while the two Claude models do not drop (updated Table 4, nanobot, 6 KleidiAI kernels, mean of 3 seeds):

ModelSVE (Graviton3)SME2 (Apple M4)SME2 / SVE
Claude Opus 4.80.4360.5621.29
Claude Sonnet 4.60.2580.2661.03
DeepSeek-V4-Pro0.2130.0910.42
GLM-5.30.4640.2130.46
GPT-5.6-luna0.1780.0730.41

On matched kernels, 3 of 5 models still drop by about 2×, except for the two Claude models. This shows that many models do not adapt to new ISAs well.

W2. The fixed-machine SME2 ladder is also too narrow to support the broader conclusion. It covers only seven KleidiAI kernels, and the agent is explicitly instructed to stay on the target ISA ... it does not establish that agents intrinsically adapt worse to all new programming models.

W2: We agree and narrowed the claim. We enlarged the ladder to 20 kernels (W1). Agents do fall back on SME2. The prompt asks them to stay on the target ISA, but this is not enforced, and none of the 20 best agent full-SME2 kernels use ZA (7 Neon, 7 plain C, 6 streaming SVE). We therefore narrow the claim: on newly introduced ISAs, current agents sometimes fail to use them by themselves. Putting SME2 pitfalls in the prompt helps but does not close the gap (g5VH W2 (1)). We revised the text accordingly.

W3. The benchmark's strongest headline results are heavily affected by reference quality. The ncnn expert reference is a Neon implementation evaluated on an SVE target ... Results should be reported more prominently relative to scalar code, compiler baselines, and generation-matched expert kernels.

Q2. For every main result, can the paper report performance relative to three baselines separately ...?

W3 & Q2: We agree, and we revised the abstract and introduction to report speedups over the compiler baseline first and to state that ncnn's reference is a Neon kernel. The starting code of every task is the scalar kernel built with -O3 and auto-vectorization, i.e., the compiler baseline. Every system beats that baseline in every column on average (2.4–19.8×), and the experts beat it by 2.1–28.2× (Appendix D).

For expert references, we use the implementation most commonly used on the target hardware. Some are not ISA-specialized because kernels for certain ISAs are poorly maintained. Testing agents on these kernels is one of this work’s main motivations, since agents could help humans improve them.

The ncnn expert reaches 6.8% of peak on average (Appendix D), and the agent reaches 17.6%, while KleidiAI’s SVE experts reach 42.9%, better than the agent (19.4%) (See response to Reviewer g5VH W1). So the large agent gains reflect a weak reference, while when against near-peak experts, agents still fall short.

AuthorsOpenReview ↗

Response to Reviewer DmAw [2/3]

W4. Benchmark construction involves substantial judgment calls ... they can introduce selection bias ... The current contamination discussion does not rule out memorized algorithms or source-specific optimization strategies.

Q3. How many candidate kernels were excluded ...? Please provide the excluded task list and the reason for each exclusion.

W4 & Q3 (selection):

The filter is an automatic rule followed by human supervision: Exclusions follow fixed rules (App. A): no expert for the target ISA, the expert fails correctness, or it is more than 2× slower than the compiler baseline, with human supervision as a final check. On the SVE tier, 21 SIMD Loops kernels were excluded:

ReasonDetected byCount
No SVE expert implementationAuto Pipeline17
Expert fails the correctness checkAuto Pipeline1
Expert hangsAuto Pipeline1
R > 2Auto Pipeline1
Agent hack to large amount of speedupHuman Supervision1

So exclusions remove kernels without a usable reference and do not prefer any agent.

Few kernels per source family: The KleidiAI sets are now larger: 20 SME2 kernels (was 7) and 6 SVE (was 3), built with the same pipeline. The limit is the number of kernels with an expert reference on each target.

W4 & Q6 (contamination):

Q6. How do the authors address contamination beyond non-verbatim code reproduction? ...

In KleidiAI, the average result on the 6 SME2 kernels whose experts were added after the model’s knowledge cutoff (GPT-5.6-luna, 2026-02-16) is 0.088, the same as on the 14 older ones (0.082). So there is no sign that the results rely on memorized code. Besides, all workloads used for the correctness check and performance measurement are hidden from the agents.

W5. Most model and harness comparisons appear to be based on one optimization run per task ... The paper needs repeated runs and confidence intervals for the main model-harness comparisons ...

Q4. Can the authors report repeated-run statistics for all main model-harness pairs ...?

W5 & Q4: We agree that one run cannot rank close systems.

  • 3 seeds on two systems: the standard deviation of a column geomean is 3–5%. Tiers are far apart, but adjacent top systems are within this noise, so we no longer argue that one is the strongest system.
  • Bootstrap 95% CIs (resampling over kernels) for every Table 2 cell are now in Appendix C.4 and agree with this.
  • The adaptation experiments now use larger, matched task sets: the SME2 arms of the ISA ladder (Table 3) cover 20 KleidiAI kernels instead of 7, and the cross-generation comparison (Table 4) uses the same six KleidiAI operators on SVE and SME2, averaged over 3 seeds.

The added experiments show the main findings hold beyond noise. Only the order within the top tier does not.

W6. A common tool-call budget is not necessarily a common optimization budget ... The paper should either present these results strictly as end-to-end system comparisons or add matched token/time-budget experiments ...

W6: We agree. We now present Table 2 as end-to-end system comparisons. Two comparisons are still controlled: models under one harness (e.g., nanobot) and harnesses with one model (e.g., Opus 4.8, Sonnet 4.6). All jobs have the same limits (40 tool calls, 150 min, 25M input / 200k output tokens). The adaptation results (Tables 3 and 4) use nanobot only, so harness differences do not affect them.

W7. The end-to-end study is encouraging but limited ... Generation cost, dispatch overhead, and performance on unseen shapes are not reported.

Q7. Can the end-to-end study report the cost of generating and maintaining the 18 specialized kernels, the dispatch mechanism, and performance on shapes not used during kernel generation?

W7 & Q7:

  • Harness: The kernels and workloads in the experiments are also collected using the KernelShift benchmarking pipeline. So, it is a demonstration that KernelShift, equipped with strong models, can actually achieve real-world model inference speedup.
  • Cost: about one agent-hour per kernel, about 590 USD for all 18, plus about 10–15 USD of AWS servers.
  • Dispatch: a hook in ggml’s mul_mat routes a matmul to the agent’s kernel only when its (quantization type, N, K) exactly matches a benchmark kernel, and the dispatch overhead is included in the measured tokens/s.
  • Unseen shapes: All other matmuls run unmodified llama.cpp code, so their speed and accuracy are unchanged. In deployment, a model’s weight shapes (N, K) are fixed, and only M (the number of tokens) varies. Each benchmark kernel already covers a range of M. (How the agent’s kernels perform on shapes outside the benchmark is answered in Q5.)
AuthorsOpenReview ↗

Response to Reviewer DmAw [3/3]

Q5. How sensitive are the results to the workload-construction procedure? In particular, what happens on held-out shapes, new model traces, and alternative manually designed workloads ...?

Q5: We ran nanobot with Opus 4.8 on 2–3 held-out shapes for each of the 76 SVE kernels of Table 2 before the KleidiAI set was expanded. 223 of 224 are correct (>99.5%), and no family gets slower by more than 4%:

FamilyCorrect on held-out shapesGeomean speedup, held-outGeomean speedup, original shapes
ncnn22/232.87×2.33×
llama.cpp26/261.47×1.53×
SIMD Loops24/241.44×1.49×
KleidiAI3/30.89×0.82×

The only failure is one ncnn conv2d_w8a8ch kernel at H = W = 2. We admit that our automatic workload generation may omit some kernel cases, while this percentage is very low (1/224).

Thank you for the author's response and the additional experiments, which have indeed addressed some of my questions.

However, the new results indicate that some models perform poorly on SME2, but this is not yet sufficient to fully support the paper's overall conclusion on hardware adaptability. Secondly, the primary comparisons lack comprehensive replication experiments, raising doubts about the stability of the results.

Therefore, I acknowledge the improvements in the paper, but these additions are not sufficient to alter my overall judgment, and I maintain the original score

Reviewer_tcpCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

designed an agent that can generalize evolving hardware generation

Critical strengths

clear motivation, clear controlled experiments, empirical studies under diff llms, establish benchmarking

Critical weaknesses

limited novelty. little new techniques, just extension of naive agentic kernel, although the motivation is clear

Questions for authors

is benchmarking going to be open sourced?

1 comment
AuthorsOpenReview ↗

Response to Reviewer tcpC

We thank the reviewer for the feedback. Please refer to our responses below:

W1. limited novelty. little new techniques, just extension of naive agentic kernel, although the motivation is clear

W1:

KernelShift’s contribution is the benchmarking pipeline and evaluation environments:

  • Evaluation across ISA generations (Neon → SVE → SVE2 → SME2) on various hardware, with an expert reference where one exists. Existing kernel benchmarks fix the hardware.
  • A semi-automatic pipeline for kernel benchmarking. We propose a workflow to find and benchmark meaningful kernels in any library, with real-world workloads and expert references collected.

Q1. is benchmarking going to be open sourced?

Q1:

Yes. We will release the full benchmark (including task definitions, workloads, expert references). We uploaded an anonymized copy, which is available in the supplementary material.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their meaningful comments. We are encouraged that the reviewers recognized several strengths of the paper, including:

  • Important gap: existing kernel benchmarks treat hardware as fixed, while KernelShift evaluates on evolving hardware and intrinsics. (Reviewers DmAw, tcpC)
  • Realistic tasks and feedback: kernels from actively developed projects, workloads from real model executions, and runtime feedback from target machines (Reviewers DmAw, g5VH)
  • Controlled comparisons across models and harnesses: a harness-agnostic server that separates model effects from harness effects across 14 harness–model pairs (Reviewers g5VH, tcpC)

The main shared concern (Reviewers DmAw, g5VH) is whether the SME2 drop comes from the agents or from other changes (hardware, tasks, documentation). We ran new experiments to separate these:

  • Fixed machine, fixed tasks, only the available instructions change: the Apple M4 ladder now covers 20 KleidiAI SME2 kernels (was 7). Neon only 0.127, + streaming SVE 0.057, + full SME2 0.083.
  • SME2 pitfalls in the prompt: unlike the earlier documentation ablation, where the agent could look up Arm's docs at any step, we put common SME2 pitfalls (streaming / ZA attributes, with examples) in the task prompt. The geomean improves but is still low (0.083 → 0.167), with 11/20 improvements and 5 decreases. So this guidance helps, but it can't solve it.
  • Matched cross-generation sets: the SVE and SME2 sides of Table 4 now use the same six operators, and the drop remains for 3 of the 5 models.

Meanwhile, some minor modifications were added:

  • Contamination: kernels added to KleidiAI after the model cutoff score the same as older ones (0.088 vs 0.082).
  • Statistics: bootstrap 95% CIs for every Table 2 cell, reported in the appendix.
  • Baselines and headroom: every system is faster than the compiler baseline (-O3, auto-vectorized) in every column (2.4–19.8×), and we report the fraction of hardware peak reached by agent, expert, and compiler code.

We highlight key changes in the revised paper in blue. Please see the individual responses below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.