From Practice to Model Weights: Consolidating the Skill Library of a Self-Improving LLM Agent into LoRA

LLM agents can improve from practice without weight updates. A harness extracts skills from failed attempts, tests them, and stores the useful ones in a skill library retrieved for later problems. Because the model never changes, every stored skill consumes context and must be read again on every problem. We study when such a RAG library should be consolidated into model weights. Our harness approves candidate skills through an acceptance gate and diversity-aware selection. A heuristic trigger, LNIS, built from retrieval and outcome statistics the harness already logs, decides when to consolidate, training a LoRA adapter on the retained skills and emptying the library. We compare it with two retrieval-only harnesses, with and without periodic skill management. On LiveCodeBench, consolidation cuts input context per held-out problem by a factor of $5.6$–$5.8$ and practice tokens by $34$–$39\%$, with LoRA training already included, and $45$–$55\%$ less practice wall-clock time with LoRA training included. Held-out accuracy is $19.8\%$ for our approach against $17.0\%$ for both baselines. On MATH-500, consolidation still uses the fewest practice tokens, though it takes $18$–$26\%$ longer, since the library never grows large enough for resets to pay back their training time. Managing the library separately, without ever consolidating it, produces no held-out benefit, so the gain traces to consolidation itself.