- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes EvoCompile, an LLM agent that improves how deep learning compilers such as torch.compile are used on a given training workload, without modifying compiler internals. The agent profiles the workload, looks up matching optimization schemes in an experience memory, and turns them into compilation-, program- and kernel-level interventions. Execution feedback refines each recipe, and both successful and failed attempts update the memory. EvoCompile is compared with the PyTorch benchmark's default compilation workflow, built-in compiler modes and the Cursor coding agent on HuggingFace, TIMM and TorchBench.
Critical strengths
-
EvoCompile works without modifying compiler internals. It changes compilation scopes, applies fullgraph to suitable regions and registers new kernels through torch.library, so it can be used with existing compilers. The paper applies it to Inductor, OpenXLA and Thunder.
-
The agent's changes are constrained. Instead of freely deciding what to modify, the Optimizer fills in the operations of a validated scheme, and every recipe is evaluated against the frozen vanilla baseline. This guards against edits that miss the compilation bottleneck or break the workload's correctness.
Critical weaknesses
-
Without the experience memory, EvoCompile performs close to naive compile (Table 4), so the results depend on what the memory contains. Sec. 3.3.3 says the memory is constructed from previously observed cases, but not which cases these were. Since the paper asks whether experience can be reused on new workloads (Sec. 1), it would help to state whether the evaluated workloads were among them.
-
The improvement over the default compilation workflow is a few percent (Table 1). EvoCompile gets 10 optimization iterations on each workload, while each compiler baseline is a single fixed configuration. Compilation-level interventions alone reach the full HuggingFace result (Table 4). A simple search over these interventions with the same budget and no LLM would be a useful baseline.
Questions for authors
-
Did the reported runs start from the experience memory in the supplementary material? It includes cases with recipes for evaluated workloads, for example XLNetLMHeadModel and demucs. If so, was each workload's own case in the memory when that workload was optimized?
-
Sec. 1 says later workloads benefit from the outcomes of earlier ones. In what order were the 119 workloads processed? How do the results on one suite change when the memory is built only from the other suites?
-
With the same 10 optimization iterations, what does a simple search over compilation-level interventions achieve without the LLM, for example random search? What does EvoCompile's search cost per workload, in LLM calls and time?
-
In Table 2, EvoCompile matches TorchTitan on Llama3-8B at both sequence lengths. The gains appear on the debug variants of about 0.32B and 33M parameters. What limits EvoCompile on the full-size model?
-
How was the Cursor baseline set up, for example its starting point, prompt, iteration budget and access to execution feedback?