Recursive Experience Improves NPU Microarchitecture Design

NPU microarchitecture design space exploration seeks low latency configurations that satisfy power and memory constraints. Recent LLM design agents can learn useful hardware choices through simulator feedback, but this experience usually remains within the current workload. When another LLM workload arrives, the agent may spend additional simulator calls rediscovering similar choices. Reusing past experience could reduce this effort, yet a choice that helps one model may not help a larger model or remain feasible under a tighter power budget. We propose RecurNPU, a method for recursively accumulating and reusing design experience across LLM workloads. RecurNPU summarizes what each search learned about hardware choices in an experience card, retaining the workload and constraints behind those observations. It then selects relevant cards and adapts their guidance to the next workload. This gives the Designer a starting point based on earlier results, while measurements on the current workload help it revise choices that no longer work. Each completed source search adds new experience for later searches, without retraining the language model. We evaluate RecurNPU on 96 settings spanning changes in LLM family, model scale, and power budget. Across the 84 shifted settings, RecurNPU reduces negative transfer from 44.0% for an agent carrying notes from preceding searches (Raw-Mem) to 20.2%. Compared with Raw-Mem, time to first token improves by 5.4% at the median and 23.4% in the geometric mean. After one target evaluation, RecurNPU is within 1% of or faster than Raw-Mem's final latency in 51 of these settings.