- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents AscendAgent, an LLM-guided kernel-optimization system that keeps its search harness fixed while updating a persistent knowledge base across workloads. External documentation, community implementations, and technical discussions are organized into conditional transformations with expected effects and provenance. Within a workload, the system performs evolutionary search over a program graph, compiling and measuring every candidate under a 40-compilation budget. After source workloads, replay compares expected and observed effects and proposes changes to transformation priorities, applicability conditions, and reusable combinations. Updates must pass independent source-side confirmation before they are transferred to held-out targets. The paper evaluates five language models and five search methods on CANN-Bench, KernelBench, MultiKernelBench, and TritonBench-G across Ascend NPUs and NVIDIA GPUs.
Critical strengths
-
The paper is unusually careful about separating workload-local search from cross-workload knowledge transfer. Frozen-KB Harness and AscendAgent share the same target-time controller, program graph, evaluation reuse, model, tasks, and compilation budget; only the inherited knowledge differs. Program archives and local statistics are reset between workloads, and held-out target outcomes cannot update or select the knowledge snapshot.
-
The experimental scope is broad. It covers five models, four benchmarks, two hardware ecosystems, and three repeated runs. Failed compilations and final confirmation consume the same budget, and the paper reports terminal speedup, first-hit cost, success rate, run variation, threshold sensitivity, and inclusive learning cost. The results are also presented with appropriate qualifications: AscendAgent has strong terminal speedups, while AB-MCTS-M often reaches the threshold earlier or on more targets.
Critical weaknesses
-
The reusable knowledge may contain benchmark-specific solution information, but the paper does not provide a sufficient overlap audit. The initial context is collected from community implementations, public repositories, documentation, and technical discussions, while the evaluated benchmarks also contain public kernel implementations and common operator families. The pilot study explicitly uses community implementations for the same types of workloads. No repository-level deduplication, source-to-benchmark mapping, near-duplicate analysis, or experiment removing community code is reported. A retrieved transformation or implementation close to a benchmark solution could directly improve search and would be difficult to distinguish from generalizable optimization knowledge.
-
The knowledge-evolution mechanism is only evaluated through aggregate downstream outcomes. The paper does not report how many updates are proposed, accepted, rejected, narrowed, merged, or transferred; which transformation families change; or whether the stated mechanism predictions are confirmed. Table 3 shows that updated knowledge outperforms frozen knowledge in terminal speedup, but it does not reveal whether the gain comes from refined conditions, changed priorities, new advice, or repeated source-side tuning. Since source learning and confirmation cost up to 932 minutes, stronger comparisons with alternative ways to use the same source-workload budget would be needed to attribute the gain specifically to replay-based knowledge revision.
-
The main benchmark matrix supports a more limited conclusion than the framing suggests. AscendAgent does not dominate all objectives or models. With Kimi K3, AB-MCTS-M has better first-hit cost and success rate on CANN-Bench. With GPT-5.6 Sol, AscendAgent is not the best CANN method, and other methods lead several CUDA cells. The main table reports only three runs and no paired statistical tests. Differences are often small after rounding, particularly in the macro speedup. The evidence supports a competitive method with strong terminal quality, but not a generally superior optimizer.
-
Reproducibility is currently weak for a result matrix of this scale. The paper explicitly states that the current package is a manuscript draft and not a validated benchmark archive. It does not provide the implementation, prompts, initial and evolved knowledge bases, source ordering, accepted update records, benchmark manifests, or measured traces required to verify the results. The method depends on many implementation choices, including proposal allocation, repair, profiling, evidence compatibility, replay, and confirmation. Formal pseudocode is helpful, but it is not enough to reproduce the reported millions of candidate-level decisions and hardware measurements.
Questions for authors
See Weaknesses