- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
KernelShift benchmarks AI coding agents on CPU kernel optimization across Arm architectural extensions, including Neon, SVE, SVE2, and SME2. An agent-assisted, human-verified pipeline builds 85 kernels from ncnn, llama.cpp, Arm SIMD Loops, and KleidiAI, with workloads largely traced from real model executions. Agents optimize scalar C++ implementations using compilation, correctness, timing, and disassembly feedback within a tool-call budget. Across six models and four harnesses, strong agents beat production references on established ISAs, reaching up to 2.74× on ncnn, and deliver 1.3× end-to-end LLM prefill speedups. However, agents struggle with SME2 even if it was proposed in 2021, which predated all the evaluated AI coding agents.
Critical strengths
The paper proposes a realistic and hardware-aware benchmark with demonstrated practical value. The tasks come from production libraries (ncnn, llama.cpp, SIMD Loops, KleidiAI), and ncnn and llama.cpp workloads are traced from real model executions. A harness-agnostic server enables separating model effects from harness effects across 14 harness–model pairs.
Critical weaknesses
- The headline speedups are hard to interpret; the readers can not tell whether a result means the agent is near the hardware limit or the reference is far from it.
- The central “evolving hardware” claim seems weak. (1) The only documentation tested (SME programming notes, in the Table 5 ablation) omits the streaming annotation the harness requires. This means LLMs' performance on evolving hardware can be unfairly constrained by the context but not by the model's capability. (2) The intrinsic-usage audit is weak. It only checks whether certain intrinsics appear in the final source but not in the binary that actually executes. The intrinsic-usage audit covers just the two Claude models, which degrade least, and is not tied to per-kernel performance.
Questions for authors
- For a few representative kernels, what fraction of hardware peak do the agent and baseline implementations reach? This can provide a better estimation of the headroom.
- Beyond the ISA name in the task prompt, what do agents know about the SME2 target, such as vector length, required streaming attributes, or compile flags? How will the agents behaviors change if providing more correct and detailed documentation and/or examples?
2 comments
Thank you for providing additional evidence and clarifications. The hardware utilization measurements help clarify the reported speedups. The additional binary-level audit and documentation ablation alleviate several of my concerns. However, I remain concerned that the evidence does not clearly distinguish LLMs' intrinsic limitations in adapting to evolving hardware from the effects of task context and guidance. Therefore, I will keep my score.
Response to Reviewer g5VH
We thank the reviewer for the careful reading and the concrete suggestions on documentation and auditing. Please refer to our detailed responses below:
W1 & Q1:
We measured the fraction of hardware peak for kernels from Claude Opus 4.8 with the nanobot harness on Graviton3 single core:
Appendix D gives this fraction for every Table 2 cell. The ncnn expert is far from peak and leaves space for agents to improve it, while the KleidiAI SVE experts are near peak and stay ahead. So the large speedups are aligned with the headroom in the reference.
W2 (1) & Q2:
We agree.
W2 (2): We agree, and we have improved the audit as follows: