acceptodds
Under review as a conference paper at ICLR 2027

ARTISAN: Unlocking LLM Kernel Optimization on Unfamiliar Hardware

Abstract

Rapidly evolving AI accelerators open a recurring gap between what the hardware can deliver and what its software actually realizes. LLM agents offer a promising way to automate kernel optimization, yet they struggle on unfamiliar hardware. A central limitation is missing hardware-specific knowledge: what to optimize on the new platform, and how to realize that optimization effectively. We introduce Artisan, which learns this knowledge automatically from a handful of expert kernels. Rather than imitating expert code, Artisan contrasts expert and agent kernels to learn reusable diagnostic tools for finding optimization opportunities and guides for realizing them on the target hardware. A task's artifacts are retained only if fresh agents that never see the expert can use the resulting support to achieve a verified improvement, filtering out knowledge that fails to transfer. On 40 unseen AWS Trainium 2 kernels, Artisan achieves a geometric-mean speedup over the vendor-provided NKIPy baseline, with every kernel verified correct, versus for the strongest baseline that learns from the same experts. Adding Artisan to four existing agentic optimization pipelines brings their reported speedups from – with 24–40/40 coverage to – with 40/40.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, kernel optimization, AI accelerators, hardware adaptation, code generation, performance engineering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ovf6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ARTISAN adapts a frozen kernel-writing agent to Trainium 2 by contrasting its implementations with expert kernels, extracting diagnostic tools and optimization guides, and testing the resulting support with fresh writers. A frozen selector delivers relevant artifacts at deployment. On 40 held-out tasks after practice on ten kernels, the paper reports 1.58x geometric-mean speedup over NKIPy with 40/40 verified coverage, versus 1.37x and 39/40 for GEPA.

Critical strengths

S1. Using execution feedback and reusable expert knowledge is a reasonable and promising direction for improving LLM-based kernel coding agents.

S2. The system is well-designed. Device-measured expert contrasts produce executable diagnostics and conditional guides, tested with fresh writers without fine-tuning.

S3. The support improves four pipelines to 1.55-1.60x with full coverage (Table 1). Ablations demonstrate that tools and guides individually improve the system.

Critical weaknesses

W1. While there can be merit in composing ideas in new ways, the evaluation only provides limited evidence for the claimed explanation of why agents fail. Appendix E.3 describes matched tests, including prose versus a worked matmul construction, but gives no condition-wise outcomes. E.1's counts and Section 4.4's transfer comparison concern different experiments.

W2. ARTISAN and GEPA get the same number of attempts, but ARTISAN uses more tokens per task (Table 1). A comparison at the same total cost, including the cost of preparing their tools or guidance, would help show whether ARTISAN's advantage remains.

W3. Were the final kernels checked on new inputs after optimization? Appendix B.2 says outputs and running times were checked, but does not specify the test inputs, acceptable numerical error, or how many runs were used to measure speed.

Questions for authors
  1. What are the trial counts and condition-wise outcomes for E.3?
  2. How do baselines perform under larger inference budgets? Do the gains persist after amortizing practice costs?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ovf6

W1 / Q1. The matched controls directly show why other agents fail and why ARTISAN succeeds. For each kernel and condition, we ran Opus 4.7 independently ten times. Within each comparison, the starting kernel, device feedback, and candidate budget are identical; only the support changes. Counts pool six candidates per run for diagnosis and 24 per run for implementation.

Failure mode 1—localization: raw execution feedback does not reveal where a known repair applies. Both conditions receive the same final-block repair rule and execution feedback. ARTISAN's diagnostic-tool report only identifies the affected region in the current candidate. Success requires passing both the target and held-out shapes.

KernelRule + feedback+ diagnostic-tool report
mha_qkv_proj5/6059/60
rmsnorm_proj1/6060/60
Total6/120119/120

Failure mode 2—realization: describing the correct target design does not mean the writer can implement it. Both conditions receive the same design. The agent-written-design condition uses the agent's own description; ARTISAN's practiced implementation guide shows how to realize it under NKI's coupled layout, tiling, placement, and ISA constraints.

KernelAgent-written designARTISAN practiced guideLatency
norm_qkproj11/240240/2402,112 → 1,390 µs
cumsum0/240240/240223 → 148 µs

These controls isolate the paper's two claims: the diagnostic tool connects device evidence to the affected code, while the practiced guide turns a correct design description into a viable implementation. The same mechanisms hold end to end: tools alone reach 1.45×, guides alone 1.51×, and both reach 1.58× with 40/40 verified (Table 2). The explanation now rests on matched experiments rather than comparisons across different studies. We added the complete protocol and condition-wise outcomes to Appendix E.3 and tied the claims in §1 directly to these controls.

W2 / Q2. ARTISAN remains best after accounting for its complete practice cost and after giving baselines substantially more inference. ARTISAN's full practice uses 1.32M input and 0.25M output tokens and costs $14, versus 0.89M/0.75M tokens and $22 for GEPA. Amortized over 40 held-out kernels, this is $0.35 per kernel, versus $0.55 for GEPA. Giving every practice baseline the corresponding additional deployment allowance leaves the ranking unchanged:

Practice methodCommon budgetAmortized budget
GEPA1.37×1.37×
DSPy/MIPROv21.35×1.38×
ExpeL1.32×1.35×
Voyager1.31×1.30×
AutoGuide1.28×1.30×

The strongest baseline reaches 1.38×, still well below ARTISAN's 1.58× with 40/40 verified. We also transfer ARTISAN's entire amortized practice allowance to the search-only baselines, then increase them further to 138–244 calls per kernel:

Online-search methodAmortized budgetExpanded budget
Autocomp1.04× (33 calls)1.21× (244 calls)
AccelOpt1.11× (31 calls)1.18× (138 calls)
AlphaEvolve-style1.14× (33 calls)1.25× (150 calls)

ARTISAN reaches 1.58× with 24 online calls. Thus the best expanded search remains far behind despite using over 6× as many calls; Autocomp uses over 10×. The issue is not insufficient budget. A self-generated search receives no external counterexample when its existing hardware model prevents it from trying the right mapping, so more calls continue exploring the same biased design space. Expert contrast supplies that missing external signal, and independent-use practice converts it into support that future writers can execute instead of rediscovering per kernel. Calls are the controlled quantity because trajectory token lengths vary; each allowance uses that method's measured tokens per call. We added practice-cost accounting in §4.2 and Appendix B.2, and the amortized and expanded results in Appendix F, Tables 13 and 15.

W3. Every final kernel was rechecked on new inputs after optimization. For each task, we generated five full-shape input sets with seeds 1001–1005, sampled floating-point inputs from a standard normal distribution, and cast them to the task-specified dtype. Outputs were compared elementwise with an FP32 PyTorch reference using torch.allclose with rtol=0.02 and atol=0.01. A kernel verifies only if all five inputs pass; all 40 ARTISAN kernels pass, giving 200/200 kernel–input checks. Each latency measurement uses five warm-up and 20 timed iterations, reporting the timed median. We added the complete verification and timing protocol to Appendix B.2 and linked it from §4.1.

AuthorsOpenReview ↗

Follow up to Reviewer ovf6

Thank you again for the constructive questions. As the discussion period is approaching its end, we wanted to briefly highlight how the new evidence addresses the central concern in W1: whether our explanation of agent failure is supported by controlled experiments.

The revision now provides three levels of causal evidence:

  1. Matched controls isolate the two failures. With the rule, starting kernel, feedback, and budget fixed, ARTISAN’s diagnostic-tool report raises successful localization from 6/120 to 119/120. With the target design fixed, replacing the agent’s own written description with a practiced guide raises correct realization from 11/480 to 480/480.

  2. The construction ablation explains why each practice step is necessary.

Support constructionNoneDirect diffMeasured ladderFull ARTISAN
Speedup0.88×1.31×1.42×1.58×
Transposed-weight reuse0%3%11%30%

Direct diff exposes expert differences; the measured ladder identifies which changes cause verified device gains; independent-use practice then determines whether a fresh writer can execute the lesson and revises it against observed failures.

  1. A controlled pipeline intervention confirms the mechanism end to end. Keeping AccelOpt’s search loop and budget unchanged, adding only ARTISAN’s frozen support raises it from 0.55× and 39/40 verified to 1.55× and 40/40. Thus the gain comes from correcting what the writer can diagnose and implement, rather than from changing the search algorithm or increasing its budget.

We hope the matched controls, component-wise ablation, and end-to-end intervention fully address your concern. We have also added the complete amortized-cost comparison and the five-input verification protocol requested in W2–W3. If any concern remains, we would be grateful for the opportunity to clarify it; otherwise, we would appreciate your reconsideration of the paper in light of the new evidence.

Reviewer_5NWVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a method for adapting a frozen LLM agent to an unknown accelerator (such as AWS trainum 2) by using a small set of expert kernels. They call their method ARTISAN. During a “practice” phase, a contrast agent builds progressions of verified changes from the agent’s best kernel toward the expert kernel. An extraction agent then turns each step into a task agnostic diagnostic tool and an optimization guide. They have heuristics for keeping the artifacts. A selector, also built during practice, delivers a smaller subset of the frozen library for each new task. On 40 held out NKI kernels, their method reports a 1.58x geometric mean speedup over NKIPy with 40/40 verified, v/s 1.37x for the best practice baseline (GEPA).

Critical strengths
  • The main problem itself is important and well timed. New hardware releases generally outpace model updates, and the “minimal knowledge set” formulation is decently realistic.
  • The contrast progression ladder is a principled way to attribute an expert’s advantage. Each change is applied in isolation to the agent’s own kernel and kept only if it stays correct along with getting faster. This thus filters out expert changes that don’t help the writer.
  • The baseline comparsions are wide: five practice methods, four NKI pipelines, LATS, and two GPU workflows ported to NKI. All are rerun under the same model, harness, and budget.
  • The composability result is good for practicality. The same frozen library improves four different search pipelines.
  • The ablations cover the main design choices: tools vs. guides, selective vs. full delivery, practice-set size, single kernel practice, and writer-model swaps.
Critical weaknesses
  1. The main metric compares geometric means over different verified sets: Speedup is computed over only the verified kernels, while coverage ranges from 22/40 to 40/40. For example, “No artifacts” scores 0.88x on 22 kernels and AlphaEvolve style search 1.27x on 24. These numbers are not comparable to 1.58x on 40. Objective (1) scores a failed task as 1x but the reported metric in the paper does not. It’d be better to see all of Table 1 and Table 2 to be rescored under objective (1), along with paired comparisons on the common verified subset. Appendix B.2 says that paired comparisons exist but does not have them.

  2. The starting condition is unclear: Appendix C.1 says SAMPLE writers receive “a correct starting kernel.” However the no-artifact condition verifies only 22/40, and several baselines score much below 1x over NKIPy. If writers start from a correct kernel or the NKIPy reference is available, why would a run return a verified kernel slower than the reference? Does the evaluation start from scratch or from the reference? Is the reference itself an eligible candidate?

  3. No variance is reported: Each condition reports only the median speedup run of three repeats, with no spread or confidence intervals. Many of the main ablation gaps are small. Bootstrap CIs over kernels and the per repetition range would show which differences are actually reliable.

  4. The practice loop’s components are not ablated. The ablations show that raw experts plus docs (1.19x) underperform the practiced library. They do not isolate the parts of practice:

    • artifacts extracted once from the ladder, without the independent-use loop
    • guides written by an LLM directly from the expert/agent diff, without the measured ladder
    • These would show which parts of the pipeline carry the gain.
  5. Smaller clarifications:

    • “Unfamiliar” hardware: Trainium 2 has been public since 2024 and is probably in the model’s pretraining in some capacity. How much of the gain is actually truly unfamiliar hardware versus partly represented hardware?
    • Leakage audit: Is the audit for operator names, shape constants, and expert code automated or manual?
Questions for authors
  1. What are the Table 1 and Table 2 results under objective (1) scoring, and on the common verified subset?
  2. Do evaluation writers start from scratch, from NKIPy, or from another correct kernel? Why do some methods return verified kernels slower than the reference?
  3. What is the variance across the three repetitions?
  4. How do the baselines perform at their native budgets?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 5NWV

W1 / Q1 / W3 / Q3. ARTISAN's advantage holds under both requested scorings and repeated runs. Because ARTISAN verifies all 40 tasks, its common verified subset with each method is exactly that method's verified task set. Eq. (1) assigns 1× to each failure.

Method/conditionEq. (1), all 40 tasksARTISAN on the same verified tasksPaired ARTISAN/method [bootstrap 95% CI]
GEPA1.361.551.13 [1.05,1.24]
DSPy/MIPROv21.341.591.18 [1.08,1.31]
ExpeL1.311.551.17 [1.09,1.33]
Voyager1.301.591.21 [1.07,1.27]
AutoGuide1.271.581.23 [1.07,1.49]
Autocomp0.991.611.63 [1.26,2.09]
AccelOpt0.561.592.89 [2.04,4.15]
NKI-Agent0.461.583.43 [2.37,5.00]
AlphaEvolve1.151.461.15 [1.07,1.25]
Guides only1.511.581.05 [1.00,1.10]
Tools only1.451.581.09 [1.03,1.15]
Experts + docs1.181.591.34 [1.17,1.55]
Full library1.331.631.19 [1.11,1.29]
Random guides1.291.571.19 [1.08,1.31]
Three attempts1.241.651.29 [1.09,1.65]
One attempt1.161.491.22 [1.11,1.36]
Sonnet 4.61.291.591.22 [1.10,1.38]

On every identical (V), ARTISAN has the higher geometric mean, and no paired bootstrap interval extends below 1. Under Eq. (1), ARTISAN remains 1.58×, versus 0.93× without artifacts and 1.15× for AlphaEvolve; its three runs span 1.56–1.58×. Tables 1–2 now add ARTISAN on each row's (V); Appendix B.2, Table 8 gives the full rescoring, intervals, and run range.

W2 / Q2. Every held-out evaluation starts from scratch. Each method receives only the task specification and its own support. NKIPy is hidden and used only as a latency reference—never as a starting kernel or candidate—so a correct generated kernel may still be slower. Only the ten disjoint practice tasks use correct starts to isolate optimization gains; held-out evaluation never does. ARTISAN verifies all 40 from scratch. We clarified both conditions in §4.1 and Appendix C.1.

W4. The measured ladder and independent-use loop are both necessary.

Support constructionNoneDirect diffMeasured ladderFull ARTISAN
Speedup0.88×1.31×1.42×1.58×
Transposed-weight reuse0%3%11%30%

Direct diff exposes expert ideas but cannot separate causal optimizations from case-specific edits. The ladder changes one decision at a time and retains only correct, device-measured gains. Writers can still misread a valid lesson through CPU/GPU assumptions, so independent-use practice tests it with a fresh writer and revises the lesson against observed failures. Although the same expert contains transposed-weight reuse throughout, adoption rises from 3% after direct diff to 11% after the ladder and 30% after full practice. Likewise, the agent's own correct lesson realizes a design in only 11/240 attempts, versus 240/240 after practice. The ladder establishes which lesson works; independent use establishes whether a new writer can execute it. We added these ablations in §4.3, Table 3 and the matched control in Appendix E.3.

Q4. ARTISAN's lead persists even with the baselines' much larger native online budgets.

MethodOnline calls/kernelSpeedup
Autocomp2441.21×
AccelOpt1381.18×
AlphaEvolve-style1501.25×
ARTISAN241.58×

Thus 6–10× more online search does not recover ARTISAN's gain. In a transpose-heavy case, the baselines retain a familiar CPU/GPU-style mapping and spend 30–31 ms on the vector engine, versus 3.4 ms for ARTISAN, despite similar 3–4 ms tensor-engine work. The five offline-learning baselines define no NKI-native deployment budget; even after amortizing their learning cost into 2–5 extra calls, the best only reaches 1.38×. We added these results in §4.2 and Appendix F, Tables 13–15.

W5. Prior exposure to Trainium 2 does not explain the gain. To test this directly, we repeated the comparison on newer hardware with no launchable instance in any public AWS region and obtained access directly through AWS. Holding Opus 5.5 fixed and giving every method the same six vendor kernels, ARTISAN reaches 1.71× versus 1.44× on Trainium 3—a 19% lead, larger than its 14% Trainium 2 lead. The advantage grows rather than disappears as prior exposure decreases (§4.5, Table 4).

For Trainium 2, the December 2024 instance announcement preceded NKI's stable public release in April 2026. More importantly, the 11/240-versus-240/240 control above directly shows that documentation and even a correct written lesson do not supply the missing implementation model.

The automated leakage audit checks 94 held-out names, shapes, IDs, and code patterns: zero evaluation matches and all 233 planted controls detected. Practice reads only the fixed ten-task split. We added the audit in Appendix B.2.

Thank you for the detailed response. My main concerns about comparisons, starting state and ablations have been addressed. The new construction ablation is also helpful. I have raised my score accordingly.

There's a point that I would still ask you to answer in the final version:

  • AWS announced general availability of Trn3 UltraServers in Dec 2025, so please clarify what you mean by no Trn3 instance is launchable publicly. Also, Trn3 and its NKI documentation has been public for ~ a year, so that by itself doesn't directly say much about pretraining exposure, especially since this experiment uses Opus 5.5 which is a newer model, which might have more exposure to Trn3 data.
AuthorsOpenReview ↗

Clarification on Trainium 3

Thank you for your careful reassessment and for pointing out the potential issues.

First, by “not publicly launchable,” we did not mean that Trainium 3 had not been announced. AWS announced general availability of Trn3 UltraServers in December 2025. However, during our experiments, no Trainium 3 capacity could be launched through public self-service access in any AWS Region. We therefore submitted an experimental-access request to AWS, which provisioned the four Trainium 3 systems used in our study.

Second, we agree that Opus 5.5 may have seen public Trainium 3 and NKI materials. We have therefore removed the implication of zero pretraining exposure. Our conclusion does not require that assumption. The relevant controlled result is that, with Opus 5.5 fixed and every method given the same six AWS-provided expert kernels, ARTISAN reaches 1.71× versus 1.44×, a 19% lead. Thus, whatever public information the model may have encountered was insufficient to eliminate the implementation gap; expert-grounded practice still provides a substantial advantage.

We have revised §4.5 and Appendix F to distinguish the Trainium 3 announcement from self-service hardware availability and to state the pretraining interpretation precisely. Thank you again for helping us clarify this point.

By the way, we appreciate that you indicated raising the score. However, the rating still appears as the previous 2 on our side. Could you please confirm that the score has been raised accordingly?

Thank you for the clarification. This distinction and the revision addresses my concerns. I have updated my rating.

Reviewer_YgrVOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper aim to address the critical challenge of how to use LLM agents to optimize kernels for new AI accelerators. In these cases, there is limited optimization knowledge or access to documentation. Its key observation is that agents often know standard optimization strategies but lack the hardware-specific knowledge needed to identify bottlenecks from execution feedback and implement effective changes. This missing knowledge can be extracted from a small set of expert kernels and reused across tasks. ARTISAN learns this knowledge through a practice phase that contrasts agent-generated kernels with expert implementations. This information is used to optimize unseen kernels.

Critical strengths

The paper tackles an important and unsolved challenge: using agents is a promising approach to address the issue of how to generate and optimize kernels for new hardware. The lack of training samples is a significant challenge

The paper demonstrates very strong results. The authors also compare against a range of different mechanisms and demonstrates the effectiveness of their approach.

Some parts of the paper are quite verbose but many key aspects of the paper are clearly written.

Critical weaknesses

The major concern I have with this work is that it generalizes the observations of why kernel generation on unseen hardware is challenging only based on some experiments with Trainium 2. It is not clear if this is specific to Trainium 2 and its compilation infrastructure. It is unclear whether observation hold across coding agents, and as they evolve. There needs to be more evaluation in a different setting before we can meaningfully draw any conclusions about the observations made and efficacy of the proposed tools.

The novelty of the proposed work is somewhat limited and conceptually the novelty over the vast number of kernel generation papers is unclear. For example, papers like AccelOpt do try to address the same challenges, albeit in a different way (using memory). The novelty of the proposed mechanism needs to be stated more clearly.

The results show that ARTISAN performs better but it does not drive the point clearly as it what it does well that’s missing from the other approaches. It would be useful to see a demonstrable example of how other approaches fail in optimzing a kernel, but the strategies employed by ARTISAN addresses this.

Questions for authors

The far right sub-figure in Figure 2 is confusing. Does it imply that all different mechanisms for kernel generation are brought to a 1.5x speedup with ARTISAN?

Is there some specific optimization that is being missed by other approaches that your approach does well? Or is it a range of optimizations? If the later which are they and how many of each in your evaluation?

1 comment
AuthorsOpenReview ↗

Response to Reviewer YgrV

W1. Stronger agents still need to adapt their hardware mental model. This gap does not disappear with the next model. On Trainium 2, moving from Opus 4.7 to 5.5 improves every method, yet ARTISAN still leads the strongest baseline by 14%. Our matched matmul experiment explains why. The agent first compared its kernel with the expert and wrote down the correct target design. Fresh writers received those lessons, yet realized the design in only 11/240 candidates. They interpreted and implemented the lesson through assumptions learned from familiar CPU/GPU code. ARTISAN instead tests the lesson through fresh writers, observes where layout, tiling, and placement still go wrong, and revises the guide. This raises realization to 240/240. A stronger agent can therefore describe the correct target design while still lacking the hardware model needed to implement it.

The finding also survives substantial changes in the target hardware. The latest-generation Trainium 3 changes the decisions a kernel writer must make: MX matmul requires packed operands and separate scale tiles; its vector engine can produce the exact quantized layout consumed by the tensor engine; and background transpose changes which data-movement schedules overlap with matmul. We obtained access to the experimental Trainium 3 configuration through AWS (Trainium 3 is not open to the public yet), and all methods received the same six AWS-provided practice kernels. Holding Opus 5.5 fixed, ARTISAN’s lead grows from 14% on Trainium 2 to 19% on Trainium 3.

Target / writerARTISANBest baselineLead
Trainium 2 / Opus 4.71.58×1.37×15%
Trainium 2 / Opus 5.51.65×1.45×14%
Trainium 3 / Opus 5.51.71×1.44×19%

ARTISAN therefore adapts the writer when hardware rules change, rather than memorizing Trainium 2 optimizations. We added the cross-generation study in §4.5, Table 4 and the matched control in §4.2 and Appendix E.3.

W2. ARTISAN addresses a blind spot of self-generated memory. AccelOpt stores successful rewrites found by its own search. If the writer’s existing hardware model prevents it from trying a mapping, that mapping cannot enter memory as a success. ARTISAN breaks this loop in two steps: expert contrast identifies, through device measurement, which missing decisions improve the writer’s own kernel; independent practice then verifies that fresh writers can actually apply them.

We test this distinction directly. Keeping AccelOpt’s search loop and budget unchanged, ARTISAN support raises it from 0.55× and 39/40 verified to 1.55× and 40/40. Directly extracting lessons from an expert–writer diff reaches 1.31×; measuring the changes through the ladder reaches 1.42×; testing and refining them with fresh writers reaches 1.58×. ARTISAN adds measured external contrast and learning from failed use. We clarified this mechanism in §4.2 and added the ablation in §4.3, Table 3.

W3 / Q2. Other methods miss a range of hardware-specific mappings. Across all 11 competing methods, we audited ten representative strategies spanning instructions, layouts, numerical formats, and engine scheduling.

KernelARTISAN mappingOther methods using the mappingARTISAN vs. median without it
mambaNative scan11,413 vs. 6,015 µs
sdattentionCross-block PSUM accumulation5329 vs. 915 µs
attention_blockFull-row instruction tile42,055 vs. 3,788 µs
csa_16384Skip fully masked tiles0136 vs. 174 µs
mamba_ssmPrefetch during computation390 vs. 116 µs
layernormNative bn_stats194 vs. 116 µs
falcon_mlpDMA transpose and copy59,401 vs. 11,681 µs
gated_projBF16 operands, FP32 accumulation21,907 vs. 2,187 µs
llama7b_layer_decodeStationary reused operand41,084 vs. 1,202 µs
norm_qkprojFused activation/scale/reduction21,263 vs. 1,394 µs

ARTISAN is faster than 72/75 verified alternatives that omit the corresponding mapping. Across all 40 kernels, it uses specialized instructions in 18, versus 7 for AccelOpt and 0 for the unassisted writer, and explicit engine scheduling in 10. We added the full audit in Appendix E.5, Table 12, linked from §4.2.

Q1. Yes—the four search pipelines reach 1.55–1.60× after receiving ARTISAN support. Each arrow in Figure 2 connects the same pipeline before and after adding the frozen tools and guides; e.g., AccelOpt moves from 0.55× to 1.55×. All four reach 40/40 verified. The fifth row compares the unassisted writer with standalone ARTISAN. We clarified the endpoints, reference, and coverage in the Figure 2 caption.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Revision Summary

We thank reviewer YgrV for prompting stronger-model, newer-hardware, and missed-mapping analyses; reviewer 5NWV for requesting fair rescoring and component ablations; and reviewer ovf6 for requesting matched mechanism controls and complete cost and verification protocols. These suggestions materially strengthened the revision. Specific responses to each reviewer are provided under each reviewer's comment.

Why ARTISAN is necessary. LLM agents have much stronger programming experience and implementation priors for mature hardware such as CPUs and GPUs. A new accelerator changes the relevant ISA, tensor layouts, memory placement, and division of work across engines. An agent may therefore know a general strategy such as reuse or fusion while implementing it through the wrong hardware model.

This is the blind spot of self-generated search and memory. If the agent's prior prevents it from trying the right mapping, that mapping cannot become a successful trajectory stored in memory. Even after an agent compares its kernel with an expert and writes the correct lesson, a fresh agent may read that lesson through the same familiar-hardware assumptions. More self-generated experience can preserve this error rather than correct it.

ARTISAN adds the missing calibration. Expert kernels expose decisions outside the agent's search space. The measured ladder tests each decision on the agent's kernel and retains only correct, device-measured gains. Independent-use practice then gives the lesson to a fresh writer that never saw the expert and revises it against observed failures. ARTISAN thus learns support that another writer can execute, not merely a description of what the expert did.

The new evidence directly validates this explanation. Holding the repair rule, starting kernel, feedback, and budget fixed, adding only ARTISAN's diagnostic-tool report raises success from 6/120 to 119/120: it connects execution evidence to the affected code. Holding the target design fixed, replacing the agent's written description with a practiced guide raises realization from 11/480 to 480/480: it turns the right idea into a viable implementation. End to end, no artifacts, direct diff, measured ladder, and full ARTISAN reach 0.88×, 1.31×, 1.42×, and 1.58×. These controls explain why documentation, raw experts, one-pass diff, and self-generated memory are not equivalent to practice.

Stronger models, newer hardware, and more search do not remove the need. Moving every method from Opus 4.7 to 5.5 improves performance, yet ARTISAN retains a 14% lead on Trainium 2. To test whether this was specific to a partly represented platform, we obtained direct AWS access to Trainium 3; we found no launchable instance in any public AWS region. Trainium 3 changes key decisions, including MX operand/scale layouts, on-device quantization layouts, and transpose–matmul overlap. With Opus 5.5 fixed and the same six AWS-provided kernels for every method, ARTISAN reaches 1.71× versus 1.44×, expanding its lead to 19%. The advantage grows as the hardware becomes less familiar.

The result is also not a budget artifact. After amortizing practice, the strongest practice baseline reaches 1.38× versus ARTISAN's 1.58×; ARTISAN's complete practice costs $14 versus $22 for GEPA. Online-search baselines use 138–244 calls per kernel—6–10× ARTISAN's 24—but reach only 1.18–1.25×. More search cannot recover a mapping that the agent's hardware model never proposes; expert-grounded practice corrects that model and transfers the correction across kernels.

Revision summary.

  • Mechanism: matched diagnostic and implementation controls, plus direct-diff, ladder, and independent-practice ablations (§4.3, Table 3; Appendix E.3).
  • Generalization: Opus 5.5, Trainium 3, and a ten-strategy audit of missed hardware mappings (§4.5, Table 4; Appendix E.5, Table 12).
  • Rigor: all-task/common-subset rescoring, bootstrap intervals, repetition ranges, full cost and budget comparisons, 200/200 final input checks, and an automated leakage audit (Appendix B.2, Table 8; Appendix F, Tables 13–15).

The revision therefore strengthens the central claim: unfamiliar-hardware optimization requires an externally grounded, execution-validated practice process that corrects the agent's hardware model and verifies that new writers can use the resulting knowledge. ARTISAN provides this adaptation layer and improves standalone generation and four existing pipelines to full verified coverage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.