- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ForgeKernel, a self-evolving GPU kernel optimization agent designed to overcome the architectural and knowledge-retention limits of single-run coding agents. ForgeKernel profiles sealed reference kernels to infer architectural decisions, guides a coding agent through measurement-driven optimization milestones, accumulates verified hardware knowledge across campaigns, and promotes its own best kernels as references for subsequent generations. Across Hopper and Blackwell, the system matches or surpasses most expert-library kernels and all evaluated kernels from prior generation systems, while self-evolution further improves already optimized kernels.
Critical strengths
- The paper explores a timely and ambitious direction for autonomous research systems: rather than treating kernel generation as a one-shot optimization problem, it explicitly targets long-horizon self-improvement through reference promotion, architectural re-planning, and persistent knowledge accumulation.
- The empirical results are strong and broad for this setting. ForgeKernel is evaluated across multiple operator families and GPU architectures, including Hopper, Blackwell, and Ascend, and is compared against both expert libraries and prior kernel-generation systems, with several cases reaching or surpassing highly optimized references.
Critical weaknesses
This paper presents very strong empirical results and an ambitious vision for autonomous kernel optimization. However, I believe the paper still requires substantial writing and methodological clarification before acceptance, as several important parts of the system and evaluation are difficult to follow.
(1) The manuscript appears insufficiently polished and is difficult to follow in several places. For example, the main text points readers to Appendix H for the cost of each campaign, but the corresponding cost table appears before the Appendix H heading, while the Appendix H section itself is effectively empty before Appendix I begins.
(2) The concrete organization of the hardware knowledge base is underspecified. The paper gives a high-level description of each recipe as containing a mechanism, trigger symptom, runnable example, and known gotchas, but it does not clearly describe the category taxonomy, the concrete representation of an entry, or provide a complete end-to-end example of how a piece of knowledge is stored and maintained.
(3) The comparison with prior kernel-generation systems such as KDA and CAKE is not controlled for optimization cost. ForgeKernel is evaluated against the final published kernels produced by these systems, but it is unclear whether those kernels were obtained under comparable model, token, GPU-time, or wall-clock budgets. Therefore, the results demonstrate stronger final kernel performance, but do not establish that ForgeKernel is a more effective or efficient optimization system under comparable resources.
(4) The mechanism and contribution of cross-operator knowledge reuse are not sufficiently characterized. The paper reports that recipes learned from earlier campaigns are prescribed to later ones, but it is unclear how relevant entries are selected, what subset of the knowledge base is exposed to the Arch Agent, and how applicability across operators is determined. Moreover, the paper reports extensive reuse statistics but does not provide a controlled with-KB vs. without-KB ablation, making it difficult to quantify the causal contribution of knowledge reuse to the final performance gains.
Questions for authors
(1) Could the authors provide a more concrete description of the hardware knowledge base? In particular, what are the nine knowledge categories, what does a complete entry look like in practice, and how are entries represented and maintained as the knowledge base grows?
(2) For the comparisons against KDA and CAKE, were the systems evaluated under comparable optimization budgets, such as agent turns, tokens or wall-clock time?
(3) How exactly is cross-operator knowledge reuse implemented? Specifically, how are relevant recipes retrieved or selected for a new turn, what subset of the knowledge base is exposed to the Arch Agent, and how is applicability across operators determined?
(4) While /goal serves as a stateless single-episode baseline, it differs from ForgeKernel in several other components besides the knowledge base. The paper lacks a controlled ablation that isolates the contribution of persistent cross-campaign knowledge reuse. Could the authors also provide a controlled with-KB vs. without-KB ablation to quantify the contribution of knowledge reuse?
2 comments
Thank you for your clarification. However, I'm still negative about the paper in its current form and will maintain my original rating.
We thank the reviewer for asking us to clarify the knowledge base, comparison, and ablations. We will make the final-artifact target, campaign accounting, and mechanism evidence explicit, and add controlled experiments for workflow components currently shown through traces.
R1: Presentation and appendix organization
The apparently empty cost section was an arXiv-cleanup error: the campaign table was removed while references remained. We will restore the campaign-ledger section and repair cross-references. We will also define “architecture” earlier with a concrete example.
R2: What the hardware knowledge base contains
The knowledge base stores validated hardware mechanisms, not just code snippets or operator-specific recipes. Each entry records:
The SM90 snapshot contains 71 entries covering pipeline/barrier scheduling, asynchronous-copy/TMA organization, warp-specialized movement, register/shared-memory pressure, predication/selection, and instruction packing/layout. We will add a category table and one complete entry.
The lifecycle is: observe a bottleneck; propose and independently probe a mechanism; admit it with applicability conditions; retrieve matching entries; cite them in milestones; and check predicted behavior, correctness, and performance. Later tasks can reject or refine entries when their boundaries fail; a successful rollout is not treated as universally reusable.
R3: KDA/CAKE and optimization cost
The original KDA/CAKE comparison evaluates released final-artifact quality, not optimization resource efficiency. We will state this scope, add a transparent campaign ledger with budgets and stopping conditions, and redo the comparison under a matched protocol when task, reference, input, correctness gate, model, and optimization budget can be aligned.
Representative active budgets are 9.9 h for H100 GQA forward, 56.0 h for H100 GQA backward, 10.7 h for H100 MLA decode, 30.1 h for B200 GDN varlen, and 97.8 h for B200 recurrent KDA. The revised comparison will report the full protocol and stopping rule rather than conflate final performance with optimization cost.
R4: Cross-operator knowledge reuse
The architecture agent does not place the entire KB in every context. It first measures task characteristics such as memory traffic, occupancy, resource pressure, instruction mix, synchronization, and profiling/disassembly symptoms, then retrieves entries with matching hardware and applicability predicates.
Only the selected subset enters the architecture context. Each entry must be cited by a milestone, whose acceptance check tests the predicted effect. The coding agent receives a conditional mechanism hypothesis and verification obligation, not an unfiltered history of tricks.
R5: KB ablation and causal interpretation
We will strengthen the KB evaluation with a matched factorial experiment. The experiment will hold model, reference, inputs, correctness gate, and budget fixed while comparing full, frozen-KB, empty-KB, no-reopening, and matched continued-search arms. This will separately quantify the effects of KB reuse, reopening, and continued search instead of relying on an unmatched module ablation.
The SM90 provenance already records cross-operator reuse, including reciprocal DSA--GQA and GEMM--MHA exchanges. The matched experiment will add quantitative comparison to this traceable evidence.
The DSA case provides a concrete mechanism trace. Continued tuning within the original architecture gives about 4.5% improvement over the previous generation. A store-removal probe changes runtime by about 6.0%; the subsequent reopening adopts a persistent loop, and the confirmed generation improves the previous one by about 7.2%. The matched experiment will provide the corresponding quantitative attribution.
Measurement clarification
All 27 released kernels were audited under the
/goalgate: all 26 applicable cells pass, while DSA indexer backward is inapplicable because the reference itself disagrees with the FP32 oracle. Same-input paired timing used five processes and 15 repetitions per process, and reproduces the main performance improvements. We will include this protocol and its reproducibility statement in the revised paper.