- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the NPU microarchitecture design space exploration (DSE) task, whose goal is to find low-latency NPU configurations for a specified LLM workload under resource budgets. Recent LLM agentic design methods can learn through evaluation results (simulation results) and improve the designs, but it is hard to transfer the experience across different target LLM workloads, because different LLM workloads have significantly different requirements and different resource budgets. To solve this problem, this paper proposes RecurNPU. It recursively accumulates design experience from multiple source workloads and reuses the experience on new target workloads. RecurNPU summarizes the LLM workload, constraints, and evaluation results in an experience card. For a new LLM workload, it selects up to two relevant cards, uses an adaptor LLM to analyze the relevance to the new workload, and then uses a designer LLM to propose the new design. This paper also builds a new benchmark with 96 settings to fill the gap of lacking existing good benchmarks. The new benchmark is dedicated to evaluating transfer-workload performance. Experiment results demonstrate that RecurNPU achieves better performance than baseline methods, including directly asking the LLM to summarize past experience and guidance.
Critical strengths
1, The motivation is good, the studied problem is very meaningful, and the paper constructs a useful benchmark. It is interesting to explore LLM agents in the NPU design domain, and it is beneficial for improving the transfer-learning ability of LLM agents.
2, The paper first demonstrates the limitations of a naïve solution, then proposes a better method. The finding that “more memory does not necessarily lead to better transfer-setting results” is interesting. And the experimental results demonstrate that RecurNPU achieves better performance.
Critical weaknesses
1, The experiment results are not convincing enough.
(1) RecurNPU’s improvement over baseline methods might be insignificant in typical cases. Section 4.1 mentions that RecurNPU improves TTFT over SimpleMem by 1.2% at the median and 15.5% in the geometric mean of setting-wise comparisons. The median improvement among all settings is 1.2%, which is not very significant.
(2) In the main experiments, Raw-Mem uses a different designer prompt from RecurNPU, which might lead to an unfair comparison. Nonetheless, the appendix conducts experiments on Raw-Mem using the same designer prompt as RecurNPU and shows consistent conclusions, which is good.
(3) The ablation studies show that, in most settings, removing a model component does not change the results very much.
2, The method’s novelty over existing methods might be limited. It is essentially a rule-based memory retrieval method with a structured memory design.
Questions for authors
Clarification requests:
1, Will you publicly share your code and the benchmark dataset if accepted?
2, One of the components of the experience card is "parameter comparisons". What is a typical number of parameter comparison pairs you can get in the experiments?
Suggestions for future work:
1, I am curious about whether LLM mainly benefits from the best design in the experience card or from other designs as well. It would be interesting if you could add an ablation study where you remove the parameter comparisons and other search results in the experience card.