acceptodds
Withdrawn from ICLR 2027

From Experience to Expertise: Adoption-Aware Memory Learning for Data-Scarce NPU Kernel Synthesis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

High-performance kernels underpin efficient accelerator execution but require expert tuning and lengthy manual optimization cycles. LLM coding agents promise automation, yet their CUDA knowledge transfers poorly to data-scarce domain-specific architectures (DSAs) such as NPUs, whose execution models and memory hierarchies differ substantially from those of GPUs. To address this transfer gap, post-training methods adapt LLMs to NPU programming but depend on scarce expert data and substantial training compute. Memory-learning agents instead adapt through external memory, but their uniform credit assignment gives adopted and unused experiences the same reward target, potentially biasing subsequent retrieval rankings. Moreover, when learned values guide only retrieval, high-value experiences that generalize across operators must be retrieved repeatedly rather than retained in context, thereby increasing retrieval overhead and weakening cross-task guidance. We therefore present SAGE, a persistent self-improving agent for NPU kernel synthesis. Adoption-Traced Utility estimation (ATU) combines explicit adoption records with kernel evaluation outcomes for adoption-aware credit assignment. Utility-Gated Consolidation (UGC) uses positive utility and repeated adoption across operators to select and abstract reusable rules into a bounded resident context. On NPUKernelBench, SAGE achieves a 95.5% execution rate versus 84.1% for the strongest controlled baseline, with 86.9% of solved operators outperforming torch_npu. With GLM-5.3, SAGE achieves a 43.99x speedup over the torch_npu reference on sparse flash attention. These results show that adoption-aware credit assignment and selective consolidation enable agents to accumulate and reuse hardware-specific knowledge across tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, Memory learning, Agent memory, Credit assignment, NPU kernel synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gifzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAGE, which studies sequential AscendC kernel generation with a frozen backbone model. It tracks whether retrieved experiences are adopted, assigns them outcome-based credit, and consolidates useful records into bounded resident memory. On a sequential 88-operator NPUKernelBench stream, it improves execution rate over controlled memory baselines across three backbones; for GLM-5.2 the reported rate is 84/88, versus 74/88 for Value Memory. On 16 unseen xCCL tasks, it obtains 11/16 versus 9/16, with memory updates still enabled. A separate frozen-bank cross-backbone experiment reuses the same 88 operators that built the bank.

Critical strengths
  1. The online protocol scores each task before updating memory from that task, directly testing whether accumulated experience helps later tasks. Shared-harness baselines, component ablations, and a fixed-bank ranking control make the main comparison informative.
  2. Adoption-aware credit and utility-gated consolidation address distinct parts of memory management. A fixed-bank reranking control isolates credit quality, and the frozen cross-backbone experiment provides evidence that learned memory is usable by another generator on the same task suite.
  3. The hardware timing protocol is comparatively clear: five warmups and 100 device-event-timed repetitions are reported.
Critical weaknesses
  1. The xCCL result combines source-memory reuse with continued learning on xCCL, exactly as the authors state. It shows adaptation to a new task library but cannot quantify how much benefit the initial source memory provides before xCCL updates. The separate frozen-memory cross-backbone test uses the same 88 operators that contributed to the bank, so direct frozen-memory transfer to unseen operators remains outside the demonstrated scope. This is a boundary on the transfer interpretation, rather than a flaw in the stated online-adaptation experiment.
  2. Table 7 freezes the final SAGE bank and replays historical SAGE adoption traces to create ranking variants. No separate evaluation set is identified, so its rates appear to come from the same 88-operator suite that produced the bank and traces. Holding contents fixed is useful, but this retrospective setup does not preserve the online pre-task information boundary and cannot alone isolate credit assignment's prospective contribution; the online Figure 5 ablation is more relevant for that claim. The paper does not specify clearly whether Table 7 reruns generation or how target-operator and future-operator records are handled during evaluation.
  3. The paper's own order comparison shows substantial Level-3 sensitivity: execution rate is 38.9% for Level-3 only, 61.1% for a mixed stream, and 77.8% when Level-2 precedes Level-3. The principal stream has only one fixed order. Appendix B.4 provides SAGE's per-operator xCCL speeds for GLM-5.2/5.3, but not a task-by-task SAGE-versus-Value-Memory correctness comparison behind the two-task xCCL margin. Order or seed uncertainty is also absent.
  4. A 10K-token resident-memory cap does not by itself establish efficiency. Actual prompt tokens, retrieval/model calls, and end-to-end time across baselines are not reported. Implementation and memory traces are also unavailable in the supplied materials.
Questions for authors
  1. Can you show per-task SAGE-versus-Value-Memory xCCL correctness outcomes and mark when each first retrieves or adopts xCCL-derived memory? Appendix B.4 already provides SAGE's per-operator speeds across two backbones; the requested comparison would explain the two-task execution-rate margin. A small frozen-source-memory control, if feasible, would separate initial transfer from later adaptation.
  2. For Table 7, were the three ranking variants evaluated in new generation runs or by replaying saved runs? When scoring a target operator, could its own records or outcomes from later operators in the stream appear in the frozen bank or ranking utilities? Please make the information boundary explicit so this control can be interpreted separately from the online ablation.
  3. Can you report actual resident/input tokens, retrieval and model calls, and wall-clock time for the compared methods? What order or seed variation is available for the main 88-task stream?
2 comments

Thank you for requesting the transfer protocols and per-task evidence.

W1 · Transfer scope

Fig. 3 measures online xCCL adaptation starting from benchmark-evolved memory. Fig. 4 tests whether another backbone can use a frozen donor bank on the same 88-operator suite.

Fig. 4 conditionProvenanceWrites during scoringDeepSeek / Qwen ER
InitialTable 1 Static RAG, shared seed bankOff37.5% / 21.6%
NativeTable 1 recipient’s online SAGEOn64.8% / 31.8%
GLM TransferFresh generation and on-device evaluation, GLM-5.2 donor bankOff72.7% / 40.9%

The Initial and GLM Transfer conditions use the same recipients, tasks, round cap and hardware evaluator. They compare a frozen seed bank with a frozen donor bank built on the same suite. Native provides the recipient’s online reference. We will clarify these settings in §3.5, Fig. 4 and Appendix D.3.

W2 · Interpretation of Table 7

The two new ranking runs use only records preceding the target operator. The ATU row comes from the online run in Table 1. Q2 gives the protocol for each row.

W3 · Order sensitivity and the xCCL margin

Table 2 compares interleaving with a sequential L2→L3 schedule on the same source and target sets. In the fixed-order GLM-5.2 xCCL run, SAGE solves two additional operators, sparse_flash_attention and gmm_deq_swiglu_quant_gmm_deq. Q1 gives the per-operator results.

W4 · Efficiency and reproducibility

The 10K cap applies to resident context. Appendix D.4 describes on-device kernel timing with 5 warmups and 100 timed repetitions. Appendix G gives example traces and memory record schemas. Q3 reports synthesis costs for nine matched xCCL tasks.

Q1 · Per-operator xCCL results and frozen-source control

Of the 16 GLM-5.2 xCCL tasks in Fig. 3, both methods solve nine and fail on five. SAGE alone solves the other two.

OperatorSAGEValue Memory
lightning_indexerPassPass
sparse_flash_attentionPassFail
mla_prolog_v3_xFailFail
compute_and_cache_digestPassPass
apply_rotary_pos_emb_partialPassPass
gmm_deq_swiglu_quant_gmm_deqPassFail
hc_post_xPassPass
hc_postPassPass
fused_gelu_sigmoidPassPass
fused_scatter_update_and_compute_digestPassPass
rms_norm_ropeFailFail
grouped_matmul_swiglu_quant_xFailFail
moe_dist_dispatch_mul_expFailFail
moe_gating_top_k_remapPassPass
rotary_pos_emb_partialFailFail
dequant_swiglu_clamp_quantPassPass
Total11/169/16

SAGE’s first recorded cross-task adoption was item 10121, written at round 23 of sparse_flash_attention and used at round 3 of gmm_deq_swiglu_quant_gmm_deq to avoid a cast on an intermediate activation. For Value Memory, the available record establishes only that this item was absent on the GMM task.

A run with the benchmark-end bank frozen would estimate the contribution of xCCL updates when paired with the online run. Comparing that frozen bank with the frozen initial bank on the same tasks would estimate the contribution of the source experience.

Q2 · What was rerun, and what could the ranker see?

Table 7 rowGeneration and hardware verificationER / L3 ER
Relevance OnlyFresh run84.1% / 61.1%
Uniform CreditFresh run89.8% / 72.2%
ATUReused online SAGE run from Table 195.5% / 88.9%

In each new run, the bank and utility replay at operator tt used only records from earlier operators. The reused ATU row comes from the online run, where each task’s record becomes available after that task is scored. We will identify ATU as the online reference in Table 7.

Q3 · What cost and run variation can be reported now?

The cost records cover nine matched GLM-5.2 xCCL tasks, including eight shared successes and one shared failure (rotary_pos_emb_partial).

Cost on the nine tasksSAGEValue Memory
Tasks solved8/98/9
Estimated prompt tokens, total1,938,3762,107,600
Retrieval calls2828
Logical model calls8288
Summed agent-side elapsed time≈5.0 h≈5.4 h

On these tasks, SAGE used 8.0% fewer estimated prompt tokens and 6.8% fewer model calls. The prompt estimates count cached prefixes incrementally within each task and include SAGE’s resident context, which held 818 tokens at the median and 832 at peak. The elapsed totals sum the agent-side intervals from each task’s first prompt to its logged verdict or round limit. Device stages have separate timing.

The principal 88-task stream uses one fixed-order run per configuration. Table 2 varies the curriculum on 18 Level-3 targets, with 7/18, 11/18 and 14/18 solved under L3-only, mixed L2+L3 and L2→L3, respectively.

Thank you again for your detailed questions. Our response clarifies the transfer and memory-evaluation protocols, identifies which conditions were rerun and which results were reused, and provides per-operator xCCL outcomes and synthesis-cost statistics for nine matched tasks.

As the discussion period draws to a close, we would appreciate hearing whether any concerns remain about the experimental setup or interpretation of the results. We would be happy to provide further clarification on any specific point.

Reviewer_hSBQOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents SAGE, a persistent self-improving agent for NPU (Ascend) kernel synthesis that keeps the backbone LLM frozen and adapts only an external memory. SAGE has two mechanisms. Adoption-Traced Utility estimation (ATU) replaces the uniform credit assignment of prior memory-learning agents (e.g., EvoKernel) by recording which retrieved experiences were actually adopted during diagnosis/revision, then distributing terminal verifier credit only among adopted items while penalizing unused retrievals. Utility-Gated Consolidation (UGC) promotes experiences that accumulate positive utility and repeated cross-operator adoption into a bounded "hot" resident context that bypasses retrieval, with usage-decay demotion under a fixed token budget.

Critical strengths

Well-motivated core insight. The distinction between retrieved and adopted experience is intuitive and addresses a concrete failure mode of uniform credit assignment: co-retrieved but unused experiences inheriting the same reward target and crowding out better candidates in later rankings. Grounding credit in evidence of use is a clean, defensible idea.

Strong and consistent empirical gains. The improvement over Value Memory is substantial and holds across three backbones of very different scale (Qwen3-Coder-Next 80B, DeepSeek-V4-Flash 284B, GLM-5.2 744B), suggesting the method is not tied to a single model. The widening advantage on Level-3 operators (61.1% → 88.9% ER with GLM-5.2) is the most convincing result.

Critical weaknesses

No system-level comparison to prior state of the art. This is deemed critical. There are many pappers on credit assignment. It is hard to claim credit, other than the specific application here.

Statistical rigor. All results appear to be single runs over a fixed sequential stream. Given stochastic LLM generation and the documented "local non-monotonicity" across the Refinement→Static RAG→Value Memory→SAGE progression, variance matters.

Questions for authors

Does the UGC hot memory ever displace rules that later prove useful (thrashing), and how often does demotion→re-promotion occur in practice?

2 comments

Thank you for asking us to clarify our contribution and baseline comparison.

W1a · Why compare with EvoKernel at the mechanism level?

The OpenReview supplement was our only available EvoKernel agent source. Its entry point requires missing YAML presets; the public artifacts contain generated kernels and reference tasks. A matched end-to-end comparison requires reconstructing presets and aligning its independent agent framework's task interface, verifier and budget with ours.

Value Memory ports EvoKernel's separate drafting/optimization values, per-round verifier updates and shared reward target for co-retrieved experiences. See our reply to ss8G Q1 for the mapping. Values guide retrieval; both methods use OpenCode for kernel start points. With the same backbone, seed bank, diagnostics, verifier and 30-round cap, SAGE improves GLM-5.2 solved counts from 74/88 to 84/88, and 11/18 to 16/18 at Level 3.

W1b · How is SAGE tailored to NPU kernel synthesis?

Credit assignment and external-memory learning are established ideas. SAGE enables a frozen agent to accumulate reusable Ascend C expertise from scarce programming knowledge and compiler/device feedback. Three synthesis challenges motivate its design.

1. Kernel-level verification does not attribute outcomes to retrieved experiences.

Table 8 documents a GatherV3 segmentation fault when host tiling dereferenced a device address. The repair moved the scalar read and dependent work partitioning to the device. Two of five retrieved items were adopted: 181 identified the address-space violation; 193 supplied the device-side read. The alternative ValueDepend strategy (item 61) was unused.

Uniform credit would give all five the same verifier target. ATU distributes the terminal score zz using adoption records (§2.3). For episode-deduplicated retrieved/adopted sets R,A\mathcal R,\mathcal A:

\xi_m=-\beta\quad(m\in\mathcal R\setminus\mathcal A).$$ These credits update retrieval utilities, with $\beta=0.2$. **Appendix E.1, Table 6** fixes available memory and resident rules, varying only cold-memory ranking: | Ranking signal | Overall ER | L3 ER | |---|---:|---:| | Relevance only | 84.1% | 61.1% | | Bulk utility | 89.8% | 72.2% | | ATU utility | **95.5%** | **88.9%** | ATU gains **16.7 percentage points** in L3 ER over bulk utility. **2. Useful feedback may emerge only after coordinated repairs.** The `Mla` trajectory in **Table 9** required repairs to Matmul configuration, cross-core synchronization and input layout before the first correct kernel at **round 23**. Optimization then reduced latency from **0.420 to 0.155 ms**. Intermediate failure need not invalidate adopted advice. ATU updates utility at episode termination, separating correctness and optimization (§2.3). For an improving episode, $T_{\mathrm{first}}$ and $T_{\mathrm{best}}$ are the first and best correct-kernel latencies: $$z=1-T_{\mathrm{best}}/T_{\mathrm{first}} =1-0.155/0.420\approx0.631.$$ The **63.1% reduction** measures progress relative to the operator's first correct kernel. **3. Cross-operator reuse requires selective residency.** The address-space constraint in **Table 8** can recur across operator families; a tiling recipe may depend on particular shapes. Retrieval-only memory must repeatedly retrieve shared constraints, while indiscriminate consolidation consumes context with narrow recipes. UGC requires positive utility and **at least three episode-level adoptions across at least three distinct operators** (§2.4). Resident rules and eligible candidates are ranked by $$U(m)=\frac{u_m\hat p_m}{L_m},$$ combining utility $u_m$, estimated access/use frequency $\hat p_m$ and token cost $L_m$ under a **10K-token budget**. **§3.6, Fig. 5** reports the GLM-5.2 ablation on 88 operators: | Configuration | Overall ER | L3 solved | |---|---:|---:| | Uniform Credit | 87.5% | 12/18 | | Without UGC | 92.0% | 14/18 | | Random UGC | 90.9% | 13/18 | | SAGE | **95.5%** | **16/18** | Random UGC uses the same resident budget. These controls support the joint design for NPU synthesis. We will clarify this scope in the revision. ### Q1 · Can a demoted rule return, and how often? Yes. **Rule 194 illustrates re-entry.** A GLM-5.2 xCCL trace excerpt (snapshots U1–U7) records three demotions and one return: | Source ID | Demoted | Outcome by U7 | |---|---|---| | 194 | U1 | Readmitted at U7 | | 238 | U3 | Still cold | | 10099 | U7 | Observation ended | Optimizer IDs start at **10001**, separately from generator correctness/debugging IDs; **10099 does not imply 10,000 stored experiences**. Demotion restores source retrieval eligibility (Algorithm 1). Rule 194 concerns an `OpDef` fix for an unsupported BF16 cast path. It was then adopted on `hc_post_x`, `moe_gating_top_k_remap` and `dequant_swiglu_clamp_quant` before readmission at U7.

Thank you again for your comments. We would like to briefly follow up on our response, which clarifies the mechanism-level comparison with EvoKernel and explains how SAGE’s design addresses NPU kernel repair and cross-operator knowledge reuse. We also highlighted the fixed-bank controls and mechanism ablations, and provided a concrete example of a demoted rule returning to resident memory.

We would appreciate your assessment of whether these explanations address your concerns, and would welcome any remaining questions while there is still time for discussion.

Reviewer_ss8GOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAGE, an external-memory learning method for NPU kernel synthesis with a frozen LLM. It uses adoption records and execution feedback to update experience utilities, then consolidates repeatedly useful experiences into a bounded resident context.

Critical strengths

The paper asks two meaningful questions: which experiences in a retrieved set should receive credit, and which knowledge should remain readily available across tasks? These are useful learning problems beyond simply storing more successful trajectories. Adoption records provide information about how the agent uses its context, while compilation and hardware feedback connect memory updates to a concrete task objective.

Critical weaknesses

My main reservation is the limited conceptual advance over EvoKernel. The setting and basic learning loop are already established: a frozen generator uses execution feedback to improve cross-task memory retrieval for NPU kernel synthesis. SAGE adds adoption-conditioned credit and a policy for keeping rules in context. These are concrete additions, but the paper needs to explain more clearly what new insight they provide about learning from experience. The main text should make the mechanism-level differences more precise, beyond the distinction between uniform and adoption-conditioned credit. Value Memory is a relevant adaptation, but the mapping to EvoKernel’s stage-specific values, selection policy, and reward/update schedule is not sufficiently clear to establish the extent of the advance.

Questions for authors
  1. What is the main insight beyond EvoKernel and existing memory-management methods? Please clarify which EvoKernel mechanisms Value Memory retains or changes, so the scope of the contribution is easier to judge.
  2. What interpretation of utility motivates sharing reward across all adopted experiences and assigning a fixed penalty to unused ones? When should non-adoption lower an item’s future retrieval priority, given redundancy, missed adoption records, or an agent choosing a different valid approach?
  3. Why should a source experience’s utility remain an appropriate value for its abstracted resident rule? What evidence helps distinguish frequently accessed knowledge from knowledge worth keeping in context? Observed rule usage, context occupancy, and promotion/demotion behavior would help make this argument concrete.
2 comments

Thank you for asking how adoption records inform credit and residency.

W1 · What is new beyond the EvoKernel loop?

EvoKernel established verifier-driven memory for NPU synthesis. A verifier evaluates the resulting kernel, even when the agent retrieves several experiences in the same round. SAGE records which experiences inform the diagnosis or revision. ATU uses that record to update the priority of retrieved experiences. UGC uses evidence of adoption across operators to decide which rules are worth keeping in context.

Table 8 shows a GatherV3 repair that moved the scalar read and axis-dependent work partitioning from host tiling to the device.

Retrieved experienceRelevanceUsed
Host cannot dereference the device pointer from GetData69.17Yes
Read the scalar in the kernel via a GM pointer67.42Yes
Expose the scalar through ValueDepend and TilingData57.43No
Use the struct macro for nested tiling data49.22No
Match host and kernel tiling field types32.47No

The unused ValueDepend item describes a plausible alternative. The first adopted item captures an address-space constraint that can recur across operator families.

Q1 · What does Value Memory preserve?

We adapted EvoKernel’s memory policy to the OpenCode harness shared with SAGE. The mapping follows EvoKernel’s supplementary code. EvoKernel, §§3.2–3.4

MechanismEvoKernel sourceValue Memory in SAGE
Stage valuesnote.py maintains separate drafting and optimization Q valuesPreserves separate correctness and optimization values
Credit and timingrun.py calls memory_curator.py after round verification. Co-retrieved entries receive a shared stage-specific reward targetPreserves per-round verifier-driven updates and the shared target for retrieved experiences
Context and start pointretriever.py selects context using stage values. CodeAgent.generate() also retrieves a verified refinement start point using optimization values when value-driven retrieval is enabledUses value to select experiences. The shared OpenCode loop controls the starting kernel

Both methods use the same generator, task interface, diagnostics, verifier and round cap. We will add the detailed mapping to the appendix.

Q2 · Why equal sharing and a fixed unused-item target?

A successful kernel can require complementary fixes in host tiling and device code. The verifier gives one outcome for their combined effect. ATU divides the terminal reward zz equally among the distinct adopted items in AA, so each receives z/∣A∣z/|A| and the total remains zz. Each retrieved item left unused receives −β-\beta (Eq. 6).

Utility estimates expected credit conditional on retrieval under the agent’s current behavior. Redundancy, a valid alternative or a missed annotation can all explain a recorded non-use. We therefore treat a single non-use as weak evidence. Repeated non-use can lower an item’s priority, and later successful adoption can raise it. With β=0.2\beta=0.2 and γ=0.3\gamma=0.3, utility rescales relevance by a factor between 0.94 and 1.30 after relevance-based candidate selection (Appendix F.3).

The trace records a rationale for each label, and evaluation waits until the labels are resolved. Of 50 audited positive labels, experts found 44 full matches to the code revision, 6 partial matches and 0 unrelated cases (Table 6).

Q3 · Why inherit utility, and when does a rule earn its context cost?

The source utility initializes the rule’s value because the abstraction is intended to preserve the original constraint and its preconditions. UGC requires positive utility and adoption across at least three distinct operators. It ranks rules by U(m)=ump^m/LmU(m)=u_m\hat p_m/L_m, which weighs utility by use and divides by the token cost. For a resident rule, p^m\hat p_m tracks observed use through an exponential moving average with μ=0.1\mu=0.1. This lets later tasks test whether the abstracted rule earns its space under the 10K-token cap. Fig. 5 compares the policies on the same 18 Level-3 operators.

GLM-5.2 conditionL3 solved
SAGE16/18
ATU with UGC disabled14/18
Random UGC under the same token cap13/18

Uniform Credit solves 12/18. This condition uses shared credit and runs UGC with previously recorded adoption evidence, without collecting new adoption labels.

In a seven-update GLM-5.2 xCCL diagnostic with an 850-token cap and consolidation every two episodes, resident occupancy ranged from 791 to 832 tokens, including the initial snapshot. The window recorded three demotions and later use of eight of the ten newly admitted rules. Rule 196, for example, was admitted at the first update and later used in gmm_deq_swiglu_quant_gmm_deq to synchronize the MTE2 and vector pipelines with SetFlag and WaitFlag.

Thank you again for your thoughtful feedback. In our response, we clarified SAGE’s distinction from EvoKernel, provided a source-level mapping of the Value Memory baseline, and explained the rationale for adoption-aware credit assignment and selective rule consolidation, supported by the reported ablations.

As the discussion period draws to a close, we would appreciate your feedback on whether these clarifications address your concerns. Please let us know if any points require further explanation; we would be happy to discuss them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.