acceptodds
Withdrawn from ICLR 2027

Recursive Experience Improves NPU Microarchitecture Design

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

NPU microarchitecture design space exploration seeks low latency configurations that satisfy power and memory constraints. Recent LLM design agents can learn useful hardware choices through simulator feedback, but this experience usually remains within the current workload. When another LLM workload arrives, the agent may spend additional simulator calls rediscovering similar choices. Reusing past experience could reduce this effort, yet a choice that helps one model may not help a larger model or remain feasible under a tighter power budget. We propose RecurNPU, a method for recursively accumulating and reusing design experience across LLM workloads. RecurNPU summarizes what each search learned about hardware choices in an experience card, retaining the workload and constraints behind those observations. It then selects relevant cards and adapts their guidance to the next workload. This gives the Designer a starting point based on earlier results, while measurements on the current workload help it revise choices that no longer work. Each completed source search adds new experience for later searches, without retraining the language model. We evaluate RecurNPU on 96 settings spanning changes in LLM family, model scale, and power budget. Across the 84 shifted settings, RecurNPU reduces negative transfer from 44.0% for an agent carrying notes from preceding searches (Raw-Mem) to 20.2%. Compared with Raw-Mem, time to first token improves by 5.4% at the median and 23.4% in the geometric mean. After one target evaluation, RecurNPU is within 1% of or faster than Raw-Mem's final latency in 51 of these settings.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
NPU, DSE

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wvstOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the NPU microarchitecture design space exploration (DSE) task, whose goal is to find low-latency NPU configurations for a specified LLM workload under resource budgets. Recent LLM agentic design methods can learn through evaluation results (simulation results) and improve the designs, but it is hard to transfer the experience across different target LLM workloads, because different LLM workloads have significantly different requirements and different resource budgets. To solve this problem, this paper proposes RecurNPU. It recursively accumulates design experience from multiple source workloads and reuses the experience on new target workloads. RecurNPU summarizes the LLM workload, constraints, and evaluation results in an experience card. For a new LLM workload, it selects up to two relevant cards, uses an adaptor LLM to analyze the relevance to the new workload, and then uses a designer LLM to propose the new design. This paper also builds a new benchmark with 96 settings to fill the gap of lacking existing good benchmarks. The new benchmark is dedicated to evaluating transfer-workload performance. Experiment results demonstrate that RecurNPU achieves better performance than baseline methods, including directly asking the LLM to summarize past experience and guidance.

Critical strengths

1, The motivation is good, the studied problem is very meaningful, and the paper constructs a useful benchmark. It is interesting to explore LLM agents in the NPU design domain, and it is beneficial for improving the transfer-learning ability of LLM agents.

2, The paper first demonstrates the limitations of a naïve solution, then proposes a better method. The finding that “more memory does not necessarily lead to better transfer-setting results” is interesting. And the experimental results demonstrate that RecurNPU achieves better performance.

Critical weaknesses

1, The experiment results are not convincing enough.

(1) RecurNPU’s improvement over baseline methods might be insignificant in typical cases. Section 4.1 mentions that RecurNPU improves TTFT over SimpleMem by 1.2% at the median and 15.5% in the geometric mean of setting-wise comparisons. The median improvement among all settings is 1.2%, which is not very significant.

(2) In the main experiments, Raw-Mem uses a different designer prompt from RecurNPU, which might lead to an unfair comparison. Nonetheless, the appendix conducts experiments on Raw-Mem using the same designer prompt as RecurNPU and shows consistent conclusions, which is good.

(3) The ablation studies show that, in most settings, removing a model component does not change the results very much.

2, The method’s novelty over existing methods might be limited. It is essentially a rule-based memory retrieval method with a structured memory design.

Questions for authors

Clarification requests:

1, Will you publicly share your code and the benchmark dataset if accepted?

2, One of the components of the experience card is "parameter comparisons". What is a typical number of parameter comparison pairs you can get in the experiments?

Suggestions for future work:

1, I am curious about whether LLM mainly benefits from the best design in the experience card or from other designs as well. It would be interesting if you could add an ablation study where you remove the parameter comparisons and other search results in the experience card.

Reviewer_1Hc4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RecurNPU transfers experience from previous workloads to NPU microarchitecture search. It stores workload conditions, feasible designs, and measured comparisons that change one hardware parameter at a time. Each source search uses earlier cards and contributes a new card, which later searches can retrieve and adapt. Experiments evaluate transfer across workload families, model scales, and power budgets.

Critical strengths
  1. Recording the conditions of a measured hardware comparison is a sensible way to make search experience reusable. The sequential transfer protocol also provides a setting for studying when experience from one workload helps or misleads the next.
  2. Appendix G.2 provides a repeated comparison with the same Designer and model. RecurNPU reduces the geometric mean of setting-median TTFT by 15.3% relative to Raw-Mem on this subset. These results support a practical benefit, and the paper also checks the method with other simulators.
Critical weaknesses
  1. My main concern is the incremental contribution to learning from experience. The method applies established memory accumulation, retrieval, and adaptation ideas to parameter search. Single-parameter measured comparisons are a concrete addition, but Table 3 does not isolate their value: median ablation ratios are close to one, the Adapter has little effect, and the variants rebuild different source histories. The experiments also do not distinguish recursive source conditioning from pooling independent source searches. The matched comparison supports the complete system, but leaves unclear which new capability comes from the memory representation or recursive loop.
  2. For NPU design, the benchmark quantifies transfer across workload and power-budget shifts, but provides less insight into which hardware observations remain useful and why. The 96 settings reuse eight target configurations, and the gains vary across simulators. These results support benefits in the evaluated settings more clearly than reusable NPU design lessons. I would find the domain contribution stronger if the analysis better explained when a measured parameter effect survives a change in workload or hardware constraints.
Questions for authors
  1. What evidence shows that retaining single-parameter comparisons improves transfer over existing experience-memory representations built from the same source trajectories?
  2. Does recursive conditioning of the source searches improve target performance beyond pooling experience from independent source searches?
Reviewer_Ke6tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how experience from previous searches can be reused in LLM-based NPU design-space exploration. It proposes RecurNPU, a memory-management workflow that stores workload conditions, search outcomes, and single-parameter comparisons in structured experience cards, then retrieves and adapts relevant cards for new workloads. Experiments across changes in model family, model scale, and power constraints show that RecurNPU reduces negative transfer compared with directly carrying raw search notes and often finds competitive designs with fewer target evaluations. The paper highlights the importance of condition-aware experience reuse, although the contribution is primarily a workflow-level design rather than a major algorithmic innovation.

Critical strengths
  1. NPU-specific memory management. RecurNPU provides a structured workflow for storing and selectively reusing experience from previous NPU design searches. By retaining workload conditions, constraints, and measured parameter comparisons, it addresses the important problem of negative transfer from raw memory. The reported TTFT improvements indicate that this domain-specific organization of experience can provide a useful starting point for subsequent searches.

  2. Benchmark design and empirical coverage. The paper constructs a meaningful benchmark over a large NPU parameter space, covering shifts in model family, model scale, power constraints, and source-history length. The experiments, including ablations and repeated runs on a subset, provide reasonable evidence that the proposed memory workflow reduces negative transfer and improves TTFT in at least the evaluated setting. This makes the work a useful empirical contribution even though the methodological innovation is mainly workflow-level.

Critical weaknesses
  1. Limited algorithmic novelty. The paper proposes an NPU-specific memory-management workflow, but its main components—structured memory, retrieval, adaptation, and feedback-guided search—are primarily a system-level combination of existing ideas rather than a substantial algorithmic innovation. This limits the paper’s methodological contribution.

  2. Insufficiently controlled and statistically limited evaluation. Most memory-based settings are evaluated with only one run, whereas No-Mem is represented by the median of twelve runs. The main Raw-Mem comparison also does not use exactly the same Designer setup, and it is not fully clear whether all memory baselines use identical harness and execution conditions. Given the stochasticity of LLM-guided search, these differences weaken the attribution of the reported gains to RecurNPU itself.

  3. Oversimplified treatment of NPU design quality. Single-parameter comparisons cannot fully capture the strong interactions among compute, memory, frequency, and communication parameters. Moreover, the evaluation optimizes only TTFT, without considering decode latency, throughput or others. The results therefore support the narrower claim that RecurNPU can find configurations with favorable TTFT under the evaluated simulator, but not that it can identify generally optimal NPU designs.

Questions for authors
  1. Could the authors clarify whether Raw-Mem, ReME, SimpleMem, and RecurNPU use exactly the same harness, proposal-validation procedure, model version, temperature, and target-search budget?
  2. Why are most memory-based settings evaluated with only one run, while No-Mem is represented by the median of twelve runs? Could the authors provide repeated runs and confidence intervals for the full benchmark, or at least for a larger representative subset?
  3. How does RecurNPU model interactions among NPU parameters? Single-parameter comparisons may miss effects such as the dependence of additional compute units on HBM bandwidth or the interaction between clock frequency and power feasibility. Have the authors considered recording pairwise parameter interactions or uncertainty estimates?
  4. Why is TTFT used as the only optimization objective? Do the configurations found by RecurNPU also perform well in terms of decode latency, time per output token, throughput, area?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.