acceptodds
Withdrawn from ICLR 2027

A SELF-GROWING EVALUATION METHODOLOGY FOR NPU OPERATORS WITH THEORY-DERIVED PERFOR- MANCE UPPER BOUNDS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deep neural network inference now spans edge devices and data centers, and its matrix- and vector-intensive computation calls for dedicated NPUs. Running a network on such an accelerator ultimately comes down to its low-level operators, whose quality directly determines model performance. Although operators can now be written in many ways, there remains no unified standard for whether an operator is correct and efficient. We therefore propose an NPU-oriented operator benchmark whose tasks are designed around the NPU's matrix/vector units and multi-level memory hierarchy, organized into four difficulty levels from vector-only operators to fused matrix-vector pipelines. However, fixed cases are insufficient for truly evaluating operators, as they capture only local behavior and can drive optimization into a local optimum. To address this, we propose SETG, which dynamically generates cases around two kinds of risk, namely operator numerical errors and NPU execution paths and overflow, while continuously refining them near anomalies based on real-device feedback. For evaluation, to determine how far each operator still is from its hardware limit, we define HAP, which analytically derives the performance ceiling from NPU scheduling characteristics, thereby measuring both achieved optimization and remaining headroom. Our experiments show that the benchmark both discovers real problems and aids effective optimization. On the one hand, in bare base-model evaluation, only four of eight base models obtain nonzero scores on all eight L1 operators, and the best average score is only 79.8, with many submissions failing on compilation or correctness, showing that the benchmark exposes real operator problems. On the other hand, in our internal evaluation, about 70% of operators score higher on hidden cases than on public cases, with average score up by 2.5 points and average speedup up by 0.51×, indicating that the optimization yields a genuine quality gain that generalizes to unseen cases, rather than overfitting to public cases.

Primary area:
datasets and benchmarks
Keywords:
NPU, operator benchmark, hardware-anchored performance, kernel generation, test-case generation, performance upper bound, LLM code generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kD8cOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an NPU-centric benchmark for evaluating low-level operator kernels generated by LLMs and AI agents. To prevent agents from gaming static benchmarks, it proposes Self-Evolving Test-Case Generation (SETG), which dynamically generates and refines cases around numerical risks and hardware execution paths. In addition, the paper introduces Hardware-Attainable Performance (HAP), a scoring metric that evaluates generated kernels against a theoretical hardware limit rather than comparing them to potentially unoptimized software baselines.

Critical strengths
  • Relying on fixed test sets inevitably leads to agents overfitting memory layouts or hardcoding execution paths. The SETG mechanism's split between public (20) and hidden (80) cases, combined with boundary-focused test generation driven by real-device profiler traces, forces true kernel generalization.

  • Evaluating LLM-generated code against standard vendor libraries often just measures how poor the reference compiler is. Calculating an absolute physical limit based on memory bandwidth and compute throughput provides a fixed, objective measurement that exposes the actual unutilized hardware potential.

Critical weaknesses
  • The Unit-Peak Overlap Bound calculates the theoretical limit by assuming 100% perfect concurrency across the cube matrix engine, vector engine, and DMA memory transfers. While mathematically an absolute ceiling, this creates an unachievable headroom. It completely ignores hardware realities like protocol overhead, inter-engine read-after-write dependencies, and local on-chip memory double-buffering synchronization. When a highly optimized kernel hits the actual hardware limit but scores poorly against this idealized bound, it sends developers chasing optimization headroom that physically does not exist.

  • Moving data from high-bandwidth memory into local memory is not just a simple division of payload by bandwidth. It involves routing across an interconnect, introducing arbitration latency, structural hazards, and bank conflicts. Because the HAP model diagnoses bottlenecks based purely on payload size versus peak throughput, it risks misdiagnosing crossbar contention or vector/matrix alignment penalties as simple "memory read" bottlenecks.

  • The paper pitches a generalized methodology, but the empirical implementation is entirely tethered to the Huawei Ascend ecosystem (CANN software stack, msprof profiler, 910b2/950pr metadata). Without demonstrating this pipeline on an alternative architecture, it is difficult to separate the genuine scientific contribution from an incremental vendor tooling release.

Questions for authors
  • How do you address the "unachievable headroom" concern? Specifically, how might the T_{HW} calculation be adjusted to account for more realistic overhead?

  • Since HAP diagnoses bottlenecks via idealized throughput calculations, how does the benchmark distinguish between a true memory bandwidth bottleneck and an unmodeled structural hazard/limitation?

  • For cancellation regions, the benchmark accepts a candidate if its absolute error count is \le 2\times the CPU reference. Did you perform any sensitivity analysis on this threshold? It seems forgiving enough that numerically unstable kernels might slip through in low-precision pipelines.

  • Do you have plans to run SETG and HAP on a non-Ascend accelerator to validate cross-platform portability?

Reviewer_1UY7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents an NPU operator benchmark for evaluating AI-generated kernels in terms of correctness and performance. The paper introduces two main components. SETG (Self-Evolving Test-Case Generation) automatically generates and refines test cases based on numerical errors, hardware execution paths, and feedback from previous evaluations. HAP (Hardware-Attainable Performance) estimates a theoretical hardware performance limit using compute and memory characteristics and scores kernels based on how close they are to this limit.

Critical strengths

The paper provides a relatively substantial benchmark with 53 operators and 1,060 cases, covering vector operations, matrix operations, fused pipelines, multiple precisions, and different NPU execution/memory paths.

Critical weaknesses

The “self-evolving” claim is not convincingly demonstrated. SETG is presented as a feedback loop that detects anomalies, generates new cases around them, evaluates again, and continuously grows the benchmark. But Section 6.2 evaluates public vs. hidden scores for 38 operators after iterative optimization. This does not directly demonstrate that SETG itself evolves over multiple generations and improves defect/path coverage. The central claimed mechanism and the experiment do not seem well aligned.

There is a strange logic in the public-vs-hidden result. The paper says 26/38 operators have higher hidden-case scores than public-case scores and interprets this as evidence that optimization “generalizes to unseen inputs.” But this could simply mean the hidden cases are easier than the public cases. A benchmark normally needs evidence that public and hidden sets have comparable difficulty/distributions before this conclusion follows. Higher hidden performance alone does not demonstrate generalization.

There is even an apparent contradiction with the “upper bound.” They report a submission that achieves 11.5× speedup against the HAP limit.If a runtime genuinely beats a claimed physical performance ceiling, either the measurement/submission is invalid or the bound/model is not valid for that case. They call this an “audit signal,” but that doesn't by itself validate the HAP model.

Questions for authors

check the Weaknesses

Reviewer_exXuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a benchmark for Ascend NPU operator implementations, with 53 operators and 1,060 public test cases spanning four levels of hardware usage. It proposes Self-Evolving Test-Case Generation (SETG), which targets numerical risks and NPU execution paths and refines cases using device feedback. It also introduces Hardware-Attainable Performance (HAP), which scores kernel runtimes using a software baseline and an analytical hardware runtime limit. The evaluation includes eight models and agents on eight entry-level operators, public/hidden comparisons for 38 optimized operators, and baseline coverage and HAP statistics across three hardware metadata sets.

Critical strengths
  1. The benchmark addresses a practical need in NPU kernel generation. Its coverage of vector-only operators through fused matrix-vector pipelines reflects the architecture’s heterogeneous execution units and memory hierarchy. Tables 2–3 expose operator-specific compilation and correctness failures even within the entry-level tier. This breakdown gives developers useful information about where generated implementations fail.

  2. The numerical test design is concrete. Long reductions, small normalization inputs, extreme logits, and deterministic cancellation target recognizable sources of numerical error (§4.1). The measurement protocol also includes warmup, cache eviction, process isolation, and kernel-level profiler timing. These choices provide a sensible foundation for evaluating correctness and execution time.

Critical weaknesses
  1. The experiment in §6.2 conflates optimization gains with differences between test sets. Table 4 compares public and hidden scores after optimization, with 26 of 38 operators scoring higher on hidden cases. Differences in shapes, numerical difficulty, and baseline runtimes can produce this pattern. Measuring an optimization gain requires evaluating the initial and final kernels on the same held-out cases. Assessing SETG’s feedback mechanism additionally requires a control with feedback disabled under the same generation budget. Both comparisons are missing from the reported evaluation. A worked example tracing an observed anomaly through case refinement to an additional discovered defect would also help establish what the feedback loop contributes.

  2. HAP needs stronger independent calibration before its reported gap can be interpreted as optimization headroom. Section 6.3 checks that the estimated hardware runtime is below the baseline, while 525 of 1,060 anchors on 910b2 and 352 of 1,060 on 950pr use proxies derived from that hardware estimate. These cases provide limited independent validation. The score–speedup correlations also share candidate runtime in their definitions. Equation (2) further makes the score sensitive to baseline quality: fixing the hardware runtime at 1 and candidate runtime at 2, baselines of 2 and 10 yield per-case performance scores of 0.5 and 0.9, respectively. Proxy calibration therefore matters for aggregate comparisons. Measurements from representative optimized kernels or unit microbenchmarks, together with an analysis restricted to measured anchors, would give a firmer basis for interpreting HAP.

Questions for authors
  1. Were the public and hidden results in Table 4 obtained from the same final kernel implementations? Please report initial and final kernel performance on a common frozen held-out set and specify the optimization budget. What additional defects does feedback-driven generation discover relative to specification-driven generation with feedback disabled, using the same case budget?

  2. What exact formula and cap define the proxy baselines? How do the aggregate scores and submission rankings change when evaluation is restricted to cases with measured baselines?

  3. For representative memory-bound, vector-bound, and cube-bound cases, can you provide the hardware constants, predicted bottleneck, and independently measured optimized runtime? This would help determine how much of the reported gap reflects achievable optimization and how much reflects the analytical model’s looseness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.