acceptodds
Under review as a conference paper at ICLR 2027

AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization

Abstract

Optimizing AscendC (Ascend C) operators for Ascend NPUs is difficult for two reasons. First, unlike CUDA, the ecosystem offers few public kernels to learn from. Second, performance depends on a coupled two-part implementation: a host-side tiling program that controls data movement and a kernel program that schedules and pipelines computation. We present AscendOptimizer, an episodic agent that builds missing optimization knowledge from execution itself. For kernel optimization, AscendOptimizer rewinds strong implementations by removing optimizations in a controlled way, then keeps the changes whose removal measurably hurts performance as reusable experience for later rewriting. For host-side optimization, it runs profiling-in-the-loop evolutionary search to find valid, fast tiling and data-movement configurations directly from hardware feedback. On a transductive benchmark of 101 real AscendC operators, AscendOptimizer achieves a 1.21× geometric-mean speedup over the open-source references, and 53.47% of operators run faster than those references. With hardware-evaluation counts matched across methods, its aggregate geometric-mean speedup exceeds Best-of-N sampling, OpenEvolve, CudaForge, and EvoKernel.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, kernel optimization, code optimization, retrieval-augmented generation, episodic memory, AI accelerators, evolutionary search

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cjnnOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new framework to conduct post-optimization of AscendC operators. The optimization process are divided into two parts, kernel program, and tiling program. For kernel program, the authors build a shared experience bank by rewinding strong existing implementations, which can be retrieved to offer inspirations to the optimizing agent. For tiling program, evolution search with hardware in the loop helps to accelerate the operators on the specific hardware. Formulating AscendC optimization as a coupled problem of host-side tiling and device-side kernel refinement, using rewind strategy to build knowledge base and introducing hardware feedback to search progress, the framework achieves best results on a transductive benchmark, which could be a good contribution to AscendC Optimization.

Critical strengths
  • The authors uses Optimization Rewind to get reusable and verified experience in the field with limited external references, which might also be useful for other fields.

  • The authors uses comparison experiments between Doc-KB and Rewind, and shows the effectiveness of Rewind strategy, which offers a great insight of how to inject knowledge to better help models.

Critical weaknesses
  • Test-set information leaks into the experience bank. Rewind mines motifs from all 101 benchmark operators, and the same bank is then used to optimize them. Self-source retrieval is only "audited," not excluded, and no leave-one-operator-out ablation is provided. The 1.21× speedup thus mixes genuine knowledge transfer with intra-benchmark information recycling.

  • Comparison is fair only on hardware evaluations. Baselines match the 230-evaluation budget, but the extra LLM cost of the Rewind pipeline is unreported, and no baseline gets comparable cross-operator information access. A key controlled baseline is missing: plain RAG over raw, undistilled kernels, which would isolate the value of Rewind distillation itself.

  • Generalization evidence is thin. The only inductive test uses 8 single-category held-out operators; gains shrink to 1.06 vs. 1.02 against the authors' own ablation only, with no external baseline compared under the same budget.

Questions for authors
  • Can Rewind beat Doc-KB on eight held-out operators?
  • Can AscendOptimizer beat other frameworks like OpenEvolve on eight held-out operators?
  • How much budget do Stage I and Stage II use respectively in 230 turns?
  • How much speedup do the 6.5% self-sourced retrievals contribute?
Reviewer_vVd4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AscendOptimizer, which optimizes existing, correct AscendC operators. Its Rewind procedure removes an optimization motif from a strong device kernel and retains a reverse-edit experience when the degraded version remains correct but becomes slower. Retrieved experiences guide kernel rewrites, after which a separate hardware search tunes host-side tiling. On 101 operators, the full method reports a 1.21× geometric-mean speedup over the reference implementations, versus 1.10× for EvoKernel, with 230 hardware evaluations per operator: 20 per operator build the shared bank and 210 support online optimization. On eight held-out loss operators, a frozen bank obtains 1.06× versus 1.02× without experience.

Critical strengths
  1. The setting is practically relevant: it addresses optimization of native AscendC host/device implementations, including nontrivial operators, rather than treating generation from a high-level specification as the same task.
  2. Rewind screens proposed experiences with compilation, correctness tests, and measured latency. A Stage I ablation supports its usefulness (1.14× with Rewind versus 1.09× without experience), and the paper compares it with a documentation-derived experience bank.
  3. The authors disclose the transductive nature of the main experiment, include a frozen-bank held-out test selected by pre-optimization compilability/correctness/profiling, and compare several search methods under the same number of hardware evaluations.
Critical weaknesses
  1. The main 101-operator result establishes fixed-suite autotuning, but gives limited evidence of transfer to new operators. The only held-out test uses eight related loss operators; its paired geometric-mean advantage is about 1.04, without per-operator uncertainty. The reported 6.5% self-source retrieval frequency counts mutations, not the contribution of self-source records to final speedups.
  2. Table 3 shows that Rewind improves Stage I (1.14× versus 1.09×), but omits a matched no-experience Stage I + Stage II run. Consequently, the contribution of retrieved experience to the final 1.21× result is not isolated. EvoKernel is a useful full-system baseline, yet it cannot isolate Rewind as an experience-construction method. The documentation-bank comparison covers only 25 operators, including all 13 Level-3 operators, and a same-pipeline forward-experience bank is absent.
  3. The objective couples kernel code and host tiling, whereas the algorithm accepts kernel edits under fixed tiling and then tunes tiling with the selected kernel fixed. A kernel that is initially slower but improves after retiling could be discarded. The paper does not quantify this possible failure mode.
  4. The measurement and validation description leaves out warmup, timed repetition count, latency aggregation, measurement boundaries, and the profiling-noise threshold. No variance or confidence intervals accompany the main or held-out results. It is also unclear how accepted Rewind edits are checked to remove only one motif. Matching hardware-evaluation counts does not account for LLM calls or total wall-clock cost.
Questions for authors
  1. Can you provide a paired, modest-size comparison of the complete pipeline with and without experience under the same stage budgets? A run excluding experiences sourced from the target operator would further clarify the transductive result.
  2. For the existing eight held-out operators, can you provide per-operator timings and measurement variation? Why was this family selected, and is there any existing evidence on another family?
  3. What are the actual warmup, timing, aggregation, noise-threshold, and correctness-test settings, and how is the single-motif condition verified? Could you include one raw timing example?
  4. Could an external AscendC suite such as NPUKernelBench support a small frozen-bank, unseen-operator pilot? If not, which concrete issues prevent a correct native starting implementation and matched timing harness? A full new benchmark is better treated as future work.
Reviewer_GtHJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AscendOptimizer, a two-stage agent for optimizing AscendC NPU kernels. First, it builds an offline experience bank using a procedure called "Rewind": it takes strong existing kernels, strips out individual optimization patterns, keeps the variants that still pass correctness checks but run slower, and logs the inverted diff as an optimization rule. In Stage I, the agent uses this bank to guide kernel rewrites; in Stage II, it tunes host-side tiling and resource parameters using hardware profiling feedback. Across 101 operators under a total budget of 230 hardware evaluations per operator (20 for building the bank + 210 online trials), it achieves a 1.21x geomean speedup over official reference kernels, beating the best baseline at 1.10x.

Critical strengths
  • The Rewind idea, removing a specific optimization motif while holding tiling fixed to measure its local impact, is a clever way to generate concrete before/after code diffs when good reference kernels exist but documented optimization trajectories don't (Definition 3.1; Figure 2).

  • The evaluation fairly accounts for the 20 offline hardware evaluations used to build the bank, caps all methods at the same 230-evaluation budget (counting compilation/runtime failures), and measures speedups against real, working official kernels. The ablations also clearly show that Stage I with the experience bank (1.14x) beats Stage I without it (1.09x) and Stage II tiling alone (1.02x).

Critical weaknesses
  • While the paper pitches deliberate de-optimization (Rewind) as its core novelty, the experiments never cleanly isolate whether Rewind is better than simpler ways of populating the experience bank from the same code. Table 4 compares Rewind against a bank built from text documentation, and the EvoKernel comparison swaps out the entire agent and search loop. Prior work like AccelOpt [R2] already collects slow-to-fast kernel pairs naturally generated during forward search (including failed optimization attempts that slowed the code down). Because no experiment holds the source kernels, retriever, and online agent fixed while comparing Rewind against forward-search trajectories or direct pattern extraction from those same reference kernels, we don't know if deliberate de-optimization is actually necessary.

  • Crucial details about correctness checking and latency profiling are missing from the paper. Section 4.1 and Appendix A mention CPU reference checks and repeated profiling, but never state the floating-point tolerances, number/distribution of test inputs, warmup iterations, timing repetitions, summary statistic (mean vs. median/min), or whether the winning kernel out of 210 trials is independently re-measured to avoid selection bias from timing noise. The slowdown threshold used to accept a Rewind variant is also described as "calibrated" without giving the number. Finally, Figure 5 shows an eye kernel rewrite that deletes off-diagonal zero writes under the assumption that the output buffer is already zero-initialized. The paper needs to clarify how that precondition is enforced and whether buffer zero-initialization time is included fairly when timing both the baseline and rewritten kernels.

Questions for authors
  • If you hold the source kernels, 20-evaluation offline budget, record format, retriever, and online optimizer completely fixed, how does a Rewind bank compare against a bank populated by slow–fast pairs collected during standard forward search (as in AccelOpt [R2]), or rules extracted directly from the reference kernels?

  • What are the exact numerical tolerances, test input counts, warmup/timing repetitions, and slowdown thresholds used in your harness, and do the speedups hold up under independent re-profiling of the selected kernels? In the Figure 5 eye kernel, is zero-initialization guaranteed by the operator contract or the test harness, and is its latency included in the timing numbers?

  • Are the "strongest available implementations" used as inputs to Rewind the exact same initial reference implementations given to all baselines, or did bank construction have access to stronger code?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.