- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
ARTISAN adapts a frozen kernel-writing agent to Trainium 2 by contrasting its implementations with expert kernels, extracting diagnostic tools and optimization guides, and testing the resulting support with fresh writers. A frozen selector delivers relevant artifacts at deployment. On 40 held-out tasks after practice on ten kernels, the paper reports 1.58x geometric-mean speedup over NKIPy with 40/40 verified coverage, versus 1.37x and 39/40 for GEPA.
Critical strengths
S1. Using execution feedback and reusable expert knowledge is a reasonable and promising direction for improving LLM-based kernel coding agents.
S2. The system is well-designed. Device-measured expert contrasts produce executable diagnostics and conditional guides, tested with fresh writers without fine-tuning.
S3. The support improves four pipelines to 1.55-1.60x with full coverage (Table 1). Ablations demonstrate that tools and guides individually improve the system.
Critical weaknesses
W1. While there can be merit in composing ideas in new ways, the evaluation only provides limited evidence for the claimed explanation of why agents fail. Appendix E.3 describes matched tests, including prose versus a worked matmul construction, but gives no condition-wise outcomes. E.1's counts and Section 4.4's transfer comparison concern different experiments.
W2. ARTISAN and GEPA get the same number of attempts, but ARTISAN uses more tokens per task (Table 1). A comparison at the same total cost, including the cost of preparing their tools or guidance, would help show whether ARTISAN's advantage remains.
W3. Were the final kernels checked on new inputs after optimization? Appendix B.2 says outputs and running times were checked, but does not specify the test inputs, acceptable numerical error, or how many runs were used to measure speed.
Questions for authors
- What are the trial counts and condition-wise outcomes for E.3?
- How do baselines perform under larger inference budgets? Do the gains persist after amortizing practice costs?
2 comments
Follow up to Reviewer ovf6
Thank you again for the constructive questions. As the discussion period is approaching its end, we wanted to briefly highlight how the new evidence addresses the central concern in W1: whether our explanation of agent failure is supported by controlled experiments.
The revision now provides three levels of causal evidence:
-
Matched controls isolate the two failures. With the rule, starting kernel, feedback, and budget fixed, ARTISAN’s diagnostic-tool report raises successful localization from 6/120 to 119/120. With the target design fixed, replacing the agent’s own written description with a practiced guide raises correct realization from 11/480 to 480/480.
-
The construction ablation explains why each practice step is necessary.
| Support construction | None | Direct diff | Measured ladder | Full ARTISAN |
|---|---|---|---|---|
| Speedup | 0.88× | 1.31× | 1.42× | 1.58× |
| Transposed-weight reuse | 0% | 3% | 11% | 30% |
Direct diff exposes expert differences; the measured ladder identifies which changes cause verified device gains; independent-use practice then determines whether a fresh writer can execute the lesson and revises it against observed failures.
- A controlled pipeline intervention confirms the mechanism end to end. Keeping AccelOpt’s search loop and budget unchanged, adding only ARTISAN’s frozen support raises it from 0.55× and 39/40 verified to 1.55× and 40/40. Thus the gain comes from correcting what the writer can diagnose and implement, rather than from changing the search algorithm or increasing its budget.
We hope the matched controls, component-wise ablation, and end-to-end intervention fully address your concern. We have also added the complete amortized-cost comparison and the five-input verification protocol requested in W2–W3. If any concern remains, we would be grateful for the opportunity to clarify it; otherwise, we would appreciate your reconsideration of the paper in light of the new evidence.
Response to Reviewer ovf6
W1 / Q1. The matched controls directly show why other agents fail and why ARTISAN succeeds. For each kernel and condition, we ran Opus 4.7 independently ten times. Within each comparison, the starting kernel, device feedback, and candidate budget are identical; only the support changes. Counts pool six candidates per run for diagnosis and 24 per run for implementation.
Failure mode 1—localization: raw execution feedback does not reveal where a known repair applies. Both conditions receive the same final-block repair rule and execution feedback. ARTISAN's diagnostic-tool report only identifies the affected region in the current candidate. Success requires passing both the target and held-out shapes.
mha_qkv_projrmsnorm_projFailure mode 2—realization: describing the correct target design does not mean the writer can implement it. Both conditions receive the same design. The agent-written-design condition uses the agent's own description; ARTISAN's practiced implementation guide shows how to realize it under NKI's coupled layout, tiling, placement, and ISA constraints.
norm_qkprojcumsumThese controls isolate the paper's two claims: the diagnostic tool connects device evidence to the affected code, while the practiced guide turns a correct design description into a viable implementation. The same mechanisms hold end to end: tools alone reach 1.45×, guides alone 1.51×, and both reach 1.58× with 40/40 verified (Table 2). The explanation now rests on matched experiments rather than comparisons across different studies. We added the complete protocol and condition-wise outcomes to Appendix E.3 and tied the claims in §1 directly to these controls.
W2 / Q2. ARTISAN remains best after accounting for its complete practice cost and after giving baselines substantially more inference. ARTISAN's full practice uses 1.32M input and 0.25M output tokens and costs $14, versus 0.89M/0.75M tokens and $22 for GEPA. Amortized over 40 held-out kernels, this is $0.35 per kernel, versus $0.55 for GEPA. Giving every practice baseline the corresponding additional deployment allowance leaves the ranking unchanged:
The strongest baseline reaches 1.38×, still well below ARTISAN's 1.58× with 40/40 verified. We also transfer ARTISAN's entire amortized practice allowance to the search-only baselines, then increase them further to 138–244 calls per kernel:
ARTISAN reaches 1.58× with 24 online calls. Thus the best expanded search remains far behind despite using over 6× as many calls; Autocomp uses over 10×. The issue is not insufficient budget. A self-generated search receives no external counterexample when its existing hardware model prevents it from trying the right mapping, so more calls continue exploring the same biased design space. Expert contrast supplies that missing external signal, and independent-use practice converts it into support that future writers can execute instead of rediscovering per kernel. Calls are the controlled quantity because trajectory token lengths vary; each allowance uses that method's measured tokens per call. We added practice-cost accounting in §4.2 and Appendix B.2, and the amortized and expanded results in Appendix F, Tables 13 and 15.
W3. Every final kernel was rechecked on new inputs after optimization. For each task, we generated five full-shape input sets with seeds 1001–1005, sampled floating-point inputs from a standard normal distribution, and cast them to the task-specified dtype. Outputs were compared elementwise with an FP32 PyTorch reference using
torch.allclosewithrtol=0.02andatol=0.01. A kernel verifies only if all five inputs pass; all 40 ARTISAN kernels pass, giving 200/200 kernel–input checks. Each latency measurement uses five warm-up and 20 timed iterations, reporting the timed median. We added the complete verification and timing protocol to Appendix B.2 and linked it from §4.1.