acceptodds
Under review as a conference paper at ICLR 2027

From Practice to Model Weights: Consolidating the Skill Library of a Self-Improving LLM Agent into LoRA

Abstract

LLM agents can improve from practice without weight updates. A harness extracts skills from failed attempts, tests them, and stores the useful ones in a skill library retrieved for later problems. Because the model never changes, every stored skill consumes context and must be read again on every problem. We study when such a RAG library should be consolidated into model weights. Our harness approves candidate skills through an acceptance gate and diversity-aware selection. A heuristic trigger, LNIS, built from retrieval and outcome statistics the harness already logs, decides when to consolidate, training a LoRA adapter on the retained skills and emptying the library. We compare it with two retrieval-only harnesses, with and without periodic skill management. On LiveCodeBench, consolidation cuts input context per held-out problem by a factor of – and practice tokens by –, with LoRA training already included, and – less practice wall-clock time with LoRA training included. Held-out accuracy is for our approach against for both baselines. On MATH-500, consolidation still uses the fewest practice tokens, though it takes – longer, since the library never grows large enough for resets to pay back their training time. Managing the library separately, without ever consolidating it, produces no held-out benefit, so the gain traces to consolidation itself.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi Agent Systems, Agent Harness, Learning from Practice, Skill Memory, Self Improving Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Je1kOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies when an agent’s growing skill library should be transferred from retrieved context into model weights. During practice, an extractor proposes skills from failed attempts; these skills are tested, selected using cost-normalized gain and diversity-aware ranking, and stored for retrieval. The proposed LNIS (Local Net-Improvement Saturation) trigger uses recent outcome and retrieval statistics to initiate LoRA training on skill-conditioned examples, followed by a conditional library reset.

Critical strengths
  1. The paper asks a meaningful question beyond whether agents can accumulate experience: when should useful experience remain in context, and when is parameter consolidation worth its training cost? This gives the work a clear operational objective.
  2. The three variants share skill extraction and selection, while progressively adding management and consolidation. This provides an informative initial comparison and shows that the reported benefits cannot simply be attributed to the particular pruning procedure alone.
Critical weaknesses
  1. The central question of when to consolidate remains insufficiently tested. The experiments compare LNIS-triggered consolidation with no consolidation, but not with alternative consolidation schedules. This establishes the potential usefulness of the full package, not the value of LNIS as a timing policy.
  2. The headline efficiency claims require stronger measurement and clearer accounting. The reported merged-adapter wall-clock results are recalibrated estimates, not measurements from an actual merged-adapter execution.
Questions for authors
  1. What evidence specifically supports LNIS over simpler consolidation schedules?
  2. At the same consolidation checkpoints, could the authors compare keeping the library, clearing it without training, training while retaining it, and training followed by reset?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Je1k (1/1)

The central question of when to consolidate remains insufficiently tested. The experiments compare LNIS-triggered consolidation with no consolidation, but not with alternative consolidation schedules.

Q1. What evidence specifically supports LNIS over simpler consolidation schedules?

Weakness 1 && Question 1: We thank the reviewer for the precise requests. As detailed in the general response, the 77- versus 38-window spacing cannot come from one fixed interval, and three of the four triggers fired on large rises in context cost. Consolidating on a detected change rather than at fixed points is the established approach when no task boundaries exist [1].

The headline efficiency claims require stronger measurement and clearer accounting. The reported merged-adapter wall-clock results are recalibrated estimates, not measurements from an actual merged-adapter execution.

Weakness 2: Our primary claim is in tokens, which do not depend on hardware and need no recalibration (Section 4.3); Table 5 and Figure 2 give the stage-by-stage accounting, including training and data construction. For wall-clock, the unmerged adapter's overhead was measured directly (78 to 82%, Appendix A.13); because merging folds the adapter into the base weights and adds no inference latency [2], we estimate merged-adapter time from the measured base-model speed; using the directly measured overhead instead changes no total by more than 0.2 h (Appendix A.14). We will label these values explicitly as estimates in the revision.

Q2. At the same consolidation checkpoints, could the authors compare keeping the library, clearing it without training, training while retaining it, and training followed by reset?

Question 2: These four arms are reported in Table 20 at the final consolidation checkpoint:

ArmLiveCodeBench (50)MATH-500 (50)
Keep library, no training22.0%54.0%
Clear library, no training16.0%64.0%
Train and keep library22.0%68.0%
Train and reset24.0%68.0%

Training followed by a reset is best or tied on both benchmarks while using no context for skills.

The same comparison is available at every earlier consolidation checkpoint from Appendix A.8 and A.9, on the ten most recent practice problems:

CheckpointKeep library, no trainingClear library, no trainingTrain and reset
LiveCodeBench, window 375/105/106/10
LiveCodeBench, window 1143/107/105/10
MATH-500, window 329/109/109/10
MATH-500, window 707/107/107/10

On these recent problems, the trained model without the library matches or exceeds the library it replaces at every checkpoint, which is the condition for the reset.

References:

[1] Aljundi, R., Kelchtermans, K., and Tuytelaars, T. "Task-Free Continual Learning." CVPR, 2019.

[2] Hu, E. J., et al. "LoRA: Low-Rank Adaptation of Large Language Models." ICLR, 2022.

Thank you for providing a detailed and sufficient reply. I keep my score unchanged.

AuthorsOpenReview ↗

Edited on [11/17/26]: added the measurements for Weakness 2 and Question 2 below.

Thank you for reading our response and for your time. In case it is useful, we have since posted a general comment with a new fixed-schedule baseline: on LiveCodeBench, LNIS used 10.8% fewer practice tokens than consolidating only at the end, at the same held-out context cost. If a specific concern still stands in the way of a higher score, we would be glad to address it during the remaining discussion period.

Added: measurements for Weakness 2 and Question 2, on a consolidated checkpoint from a new run of the same code (MATH-500, consolidation at the end of practice).

Weakness 2: measured, not estimated. On the same 50 held-out problems, decoding cost per output token is:

Modelms per output token
Base model12.18
Merged adapter12.09
Unmerged adapter21.70

Merging removes the adapter's overhead entirely (−0.7% versus the base model), confirming the estimate of Appendix A.14 with a direct measurement.

Question 2: the four arms on the full held-out set, three seeds.

ArmSeed 0Seed 1Seed 2Mean
Clear library, no training56%64%62%60.7%
Keep library, no training60%58%60%59.3%
Train and keep library68%66%62%65.3%
Train and reset66%64%66%65.3%

In every seed, training followed by a reset matches or exceeds both untrained arms; on average it adds 4.7 points over clearing the library alone (95% CI [0.7, 9.3]), and keeping the library after training adds nothing. Table 20 reported a single sampling run of a different consolidated checkpoint; its base-model value (64.0%) is one of the values observed here, and both evaluations rank the arms identically, with the two trained arms on top.

AuthorsOpenReview ↗

Final follow-up before the discussion closes

Thank you again for engaging with our response. Since your 15 November assessment, we added results for each of your requests (see our updated reply above and the general comment "Update: LNIS versus fixed schedules and versus clearing the library"), all marked in blue in the revised PDF:

  • Simpler schedules (Q1): on LiveCodeBench, LNIS used 5.8% fewer practice tokens than midpoint consolidation and 10.8% fewer than end-only consolidation (single runs).
  • Measured merged adapter (W2): 12.09 ms per output token, versus 12.18 for the base model and 21.70 unmerged.
  • Training versus clearing (Q2): on all 50 MATH-500 held-out problems with three sampling seeds, train-and-reset averaged 65.3% versus 60.7% for clearing without training.

With the discussion ending within a day, we would be grateful to know whether a specific concern still keeps the paper below the acceptance threshold.

Reviewer_rbhTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for consolidating procedural skills acquired by self-improving LLM agents from an external skill library into model parameters using LoRA. The framework extracts skills from failed trajectories, evaluates and selects useful skills based on their contribution to task performance and context cost, and maintains them in a retrieval-based library. The authors introduce Local Net-Improvement Saturation (LNIS), a heuristic trigger that determines when the accumulated skills should be consolidated into model weights.

The approach is evaluated on LiveCodeBench and MATH-500 against two retrieval-based baselines, with and without periodic skill management. The results show substantial reductions in input-context usage and practice-token consumption while maintaining comparable held-out accuracy. The proposed approach also reduces overall practice time on LiveCodeBench, although not on MATH-500.

Critical strengths
  1. Interesting and practically relevant problem.

The paper addresses an important challenge in self-improving LLM agents: how to avoid the increasing context overhead associated with accumulating external skills. The idea of periodically consolidating useful skills into model parameters is interesting and potentially valuable for agents that continuously acquire and reuse knowledge.

  1. Substantial efficiency improvements and detailed cost analysis.

The reported reductions in context usage and practice-token consumption are substantial. In particular, the proposed method achieves a 5.6–5.8× reduction in held-out input context on LiveCodeBench while maintaining comparable accuracy. The paper also provides detailed computational cost breakdowns and discusses the conditions under which consolidation can amortize its training overhead.

Critical weaknesses
  1. The effectiveness and necessity of the proposed LNIS trigger are insufficiently validated.

The main methodological contribution is LNIS, which adaptively determines when to consolidate the skill library into model weights. However, the experiments only compare the proposed framework against retrieval-based baselines that do not perform any consolidation.

A straightforward and important baseline would be periodic LoRA consolidation, using the same skill selection procedure, comparable training budgets, and a matched number of consolidation events. Without such a comparison, it is unclear whether LNIS provides any meaningful advantage over simpler consolidation schedules.

Consequently, the experimental results demonstrate the potential benefits of consolidation itself, but provide limited evidence for the effectiveness of the proposed adaptive triggering mechanism. Since LNIS is presented as a central contribution, I consider this a significant weakness.

  1. The evaluation does not sufficiently establish the internalization and generalization of reusable skills.

The experiments are conducted only on LiveCodeBench and MATH-500, which primarily evaluate standalone coding and mathematical reasoning abilities rather than the acquisition and reuse of procedural agent skills. The authors justify their selection by contrasting large and small practice pools, but do not adequately explain why these benchmarks are appropriate for evaluating skill internalization. Moreover, the benchmarks differ not only in practice-pool size but also in task domain and policy model, making it difficult to isolate the effect of practice-pool size.

The improvements attributable to the retrieved skills themselves are relatively small (approximately 2–3 percentage points on the practice sets). This raises the question of whether the framework is consolidating genuinely useful and transferable skills or mainly eliminating the context overhead of weakly beneficial instructions.

Furthermore, the LoRA adapters are trained through supervised fine-tuning on problem-solution pairs derived from the selected skills. Without comparison to standard SFT on comparable problem-solution data, it remains unclear whether explicit skill extraction and selection provide benefits beyond ordinary task-specific fine-tuning.

Finally, the evaluation provides limited evidence of generalization beyond the benchmarks used during practice. Although the appendix includes additional checkpoint diagnostics, evaluation on independent benchmarks would be valuable for determining whether the consolidated model acquires transferable procedural skills and whether it preserves capabilities outside the training domain. The use of only Qwen2.5-based 7B policy models and a single practice run per variant further limits the generality of the conclusions.

Questions for authors

Q1. How does LNIS compare against a simple periodic consolidation baseline under matched consolidation counts and computational budgets? Would the reported efficiency gains remain substantially different?

Q2. How does the proposed skill-based consolidation compare with standard LoRA fine-tuning on successful or verified corrected problem-solution pairs, using comparable training budgets? What evidence demonstrates the necessity of the intermediate skill library?

Q3. Could the authors further justify the choice of LiveCodeBench and MATH-500 for evaluating procedural skill internalization? In particular, does the consolidated model preserve or improve performance on independent benchmarks that were not used during skill acquisition?

Q4. How robust are the reported results across different practice seeds or stronger and more recent policy models? Given the stochastic nature of skill extraction and consolidation, additional evidence of stability would help establish the generality of the proposed approach.

3 comments
AuthorsOpenReview ↗

Response to Reviewer rbhT (1/2)

The effectiveness and necessity of the proposed LNIS trigger are insufficiently validated. […] A straightforward and important baseline would be periodic LoRA consolidation, using the same skill selection procedure, comparable training budgets, and a matched number of consolidation events. Without such a comparison, it is unclear whether LNIS provides any meaningful advantage over simpler consolidation schedules.

Q1. How does LNIS compare against a simple periodic consolidation baseline under matched consolidation counts and computational budgets? Would the reported efficiency gains remain substantially different?

Weakness 1 && Question 1: We thank the reviewer for the concrete baseline proposed. As detailed in the general response, a fixed interval cannot match the spacing LNIS chose on both benchmarks, and a signal-based trigger is the established answer to deciding when to consolidate without fixed boundaries [1]. Every consolidation uses the identical procedure (16 examples per skill, the same LoRA settings), so a matched-count periodic schedule would differ from LNIS only in timing; our claim concerns choosing that timing from observed cost rather than from an interval fixed in advance.

The evaluation does not sufficiently establish the internalization and generalization of reusable skills. […] The improvements attributable to the retrieved skills themselves are relatively small (approximately 2–3 percentage points on the practice sets). […] Without comparison to standard SFT on comparable problem-solution data, it remains unclear whether explicit skill extraction and selection provide benefits beyond ordinary task-specific fine-tuning. […] Evaluation on independent benchmarks would be valuable […]. The use of only Qwen2.5-based 7B policy models and a single practice run per variant further limits the generality of the conclusions.

Q2. How does the proposed skill-based consolidation compare with standard LoRA fine-tuning on successful or verified corrected problem-solution pairs, using comparable training budgets? What evidence demonstrates the necessity of the intermediate skill library?

Weakness 2 && Question 2: Our question is when an agent's accumulated knowledge should move from its library into the weights, not whether skill-based training beats standard fine-tuning. The design therefore holds the knowledge source fixed (the same gated skills, in context versus in weights); fine-tuning on raw solutions would change both the representation and the training data. Prior work also finds that direct fine-tuning on collected experience recovers only 3.8% of the in-context gain, while distilling in-context experience retains at least 64.8% [2]. The library also keeps training compact: 528 and 224 examples for the two LiveCodeBench consolidations (Table 9). The training data also contain hindsight corrections of failed attempts, which self-training on already-solved problems cannot provide: at the first MATH-500 consolidation, 21 of the 73 verified examples were corrected failures (Appendix C.1). On whether we consolidate transferable skills or remove the overhead of weakly helpful ones: on these benchmarks mostly the latter, which is precisely our efficiency claim. We therefore make no comparative claim against standard fine-tuning on solved problems.

AuthorsOpenReview ↗

Response to Reviewer rbhT (2/2)

Q3. Could the authors further justify the choice of LiveCodeBench and MATH-500 for evaluating procedural skill internalization? In particular, does the consolidated model preserve or improve performance on independent benchmarks that were not used during skill acquisition?

Weakness 2 && Question 3: The skills are procedural by construction: each holds a trigger, a numbered corrective procedure, and a verification step (Section 3.2, Table 17). The two benchmarks contrast a large and a small practice pool (949 versus 450 problems), and our claims are made per benchmark. Because the benchmarks also differ in domain and policy model, we compare variants within each benchmark and do not attribute differences between the two benchmarks to pool size alone. The same runs also evaluated the LiveCodeBench model (Qwen2.5-Coder-7B) on an independent benchmark never used in practice, 50 Omni-MATH problems: the consolidated model solved 5 problems versus 2 for Baseline 1 (10.0% versus 4.0%), with 144 versus 2,414 input tokens per problem. We will add this to the appendix. On forgetting, Appendix E.2 shows that no previously solved problem among those checked was lost at any consolidation.

Q4. How robust are the reported results across different practice seeds or stronger and more recent policy models?

Weakness 2 && Question 4: Held-out results average three sampling seeds (Appendix E.4), and two separate practice runs (the full variant and Baseline 1) first trigger at windows 37 and 36 (Table 7). For compute-heavy pipelines, few-run evaluation with interval estimates is the accepted practice [3], and statistical studies of LLM-agent training likewise average their main results over two seeds [4]. Newer policy models are a natural next step.

References:

[1] Aljundi, R., Kelchtermans, K., and Tuytelaars, T. "Task-Free Continual Learning." CVPR, 2019.

[2] Gou, C., et al. "Sample-Efficient Learning from Agent Experience." arXiv:2607.21051, 2026.

[3] Agarwal, R., et al. "Deep Reinforcement Learning at the Edge of the Statistical Precipice." NeurIPS, 2021.

[4] Vattikonda, D., et al. "How to Train Your LLM Web Agent: A Statistical Diagnosis." NeurIPS, 2025.

AuthorsOpenReview ↗

Final follow-up before the discussion closes

Thank you again for your detailed review. With the discussion ending within a day, we point to the results added for your questions. For Question 3, a consolidated checkpoint scored 94.7% on GSM8K against 93.6% and 93.3% for the two library-based systems, and 70.1% on HumanEval against 61.8% and 74.8%, with 14 to 29 times less input context (see our general comment "Update: unseen benchmarks and forgetting"). For Question 1, LNIS used 5.8% to 17.8% fewer practice tokens than three alternative schedules on LiveCodeBench (see "Update: LNIS versus fixed schedules and versus clearing the library"). All additions are marked in blue in the revised PDF.

If one comparison remains most important for your recommendation, a short note would let us respond before the deadline.

Reviewer_RVQROpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when an LLM agent should consolidate an external skill library into model weights instead of repeatedly retrieving skills in context. The proposed framework extracts and filters reusable skills from practice, manages the resulting library, and introduces LNIS, a heuristic trigger that decides when to distill the active skills into a LoRA adapter and reset the library. Experiments on LiveCodeBench and MATH-500 show that consolidation can substantially reduce context and practice-token costs while largely preserving task accuracy, although the wall-clock benefit depends on whether the practice horizon is long enough to amortize the cost of LoRA training.

Critical strengths
  1. The paper identifies an important and underexplored systems problem in experience-based LLM agents.
  2. The efficiency results and cost accounting are strong and practically meaningful.
Critical weaknesses
  1. The experiments establish the value of consolidation more clearly than the value of LNIS as a consolidation trigger.
  2. The main practice runs use only one practice seed, while consolidation changes the subsequent learning trajectory.
  3. The comparison between Baseline 2 and the full method does not isolate only the representation of knowledge.
  4. The reset criterion is weakly validated and can accept updates that reduce no-skill accuracy.
Questions for authors

1.Could the authors provide a comparison against at least one simple consolidation-trigger baseline? 2.How stable are LNIS trigger times and the resulting efficiency gains across different practice seeds? 3.How should the causal claim that the gain “traces to consolidation itself” be interpreted given that consolidation also resets the library and changes future skill acquisition?

3 comments
AuthorsOpenReview ↗

Response to Reviewer RVQR (1/1)

The experiments establish the value of consolidation more clearly than the value of LNIS as a consolidation trigger.

Q1. Could the authors provide a comparison against at least one simple consolidation-trigger baseline?

Weakness 1 && Question 1: We thank the reviewer for separating the value of consolidation from the value of LNIS. As detailed in the general response, no single fixed interval matches the 77- and 38-window spacing LNIS chose on the two benchmarks, and its triggers coincide with sharp rises in context cost. Triggering consolidation from a detected change in recent statistics, rather than at fixed points, follows the established approach for learning without task boundaries [1].

The main practice runs use only one practice seed, while consolidation changes the subsequent learning trajectory.

Q2. How stable are LNIS trigger times and the resulting efficiency gains across different practice seeds?

Weakness 2 && Question 2: Two separate practice runs reach the first trigger at nearly the same point: window 37 in the full variant and window 36 when LNIS is evaluated on Baseline 1 (Table 7). As these are two different variants, this shows consistent first-trigger timing rather than a replicate of the full method. The efficiency gain is mechanistic rather than stochastic: a small library makes every candidate evaluation cheaper (34.88M versus 14.67M tokens, Table 5). Sampling variation in the held-out evaluation is covered separately: results average three sampling seeds, confirmed by the paired three-seed tests in Appendix E.4, and we report interval estimates, the recommended practice when compute allows only a few runs [2]. How consolidation changes the later learning trajectory is addressed in our answer to Weakness 3 and Question 3.

The comparison between Baseline 2 and the full method does not isolate only the representation of knowledge.

Q3. How should the causal claim that the gain "traces to consolidation itself" be interpreted given that consolidation also resets the library and changes future skill acquisition?

Weakness 3 && Question 3: Consolidation indeed changes both where knowledge lives and what the agent learns next. Table 20 separates the representation at the final checkpoint: training raises accuracy without the library from 16.0% to 24.0% (LiveCodeBench) and from 64.0% to 68.0% (MATH-500), and keeping the library after training does not improve on the trained model without it (22.0% versus 24.0% on LiveCodeBench, 68.0% versus 68.0% on MATH-500). The reset's effect on later learning is shown in Section 5.3: after a reset, 28% of novel candidates pass the gate versus 7 to 9% in the growing libraries. Both effects belong to the method, so we will revise "the gain traces to consolidation itself" to "consolidation followed by a library reset".

The reset criterion is weakly validated and can accept updates that reduce no-skill accuracy.

Weakness 4: The reset criterion ensures that the new weights reproduce the library's benefit before the library is removed, accadapter≥0.7⋅accRAG\mathrm{acc}_{\text{adapter}} \geq 0.7 \cdot \mathrm{acc}_{\text{RAG}} (Table 10). Its practical safety is shown in Appendix E.2: on the earlier practice problems checked, no previously solved problem was lost at any consolidation. No-skill accuracy rose or held at three of the four consolidations (Table 11); at the fourth (LiveCodeBench window 114, 7/10 to 5/10 on ten recent problems), every earlier-solved problem in the retention check was kept (3 of 3, Table 21).

References:

[1] Aljundi, R., Kelchtermans, K., and Tuytelaars, T. "Task-Free Continual Learning." CVPR, 2019.

[2] Agarwal, R., et al. "Deep Reinforcement Learning at the Edge of the Statistical Precipice." NeurIPS, 2021.

AuthorsOpenReview ↗

New result for Weakness 2 && Question 2: LNIS trigger timing across practice runs (1/1)

We ran a new re-run of the full variant with the same seed, code and settings. Because the extractor's outputs and GPU numerics are not exactly reproducible, it followed its own practice trajectory, giving a run-to-run comparison of when LNIS first triggers.

BenchmarkRunFirst LNIS trigger (window)
LiveCodeBenchFull variant, original run37
LiveCodeBenchFull variant, new re-run31
LiveCodeBenchBaseline 1 (LNIS evaluated, not acted on)36
MATH-500Full variant, original run32
MATH-500Full variant, new re-run41

On LiveCodeBench, three separate practice runs reach their first trigger within six windows of each other; on MATH-500, the two runs are nine windows apart. This covers trigger timing; the re-run's efficiency figures are not yet complete. This is not a multi-seed study.

AuthorsOpenReview ↗

Final follow-up before the discussion closes

Thank you again for your review. With the discussion period ending within 24 hrs, we briefly summarize the evidence added since our response, in case it helps your final assessment.

On LiveCodeBench, LNIS used 5.8% fewer practice tokens than a single consolidation at the midpoint and 17.8% fewer than clearing the library at the same windows without training, with no lower held-out accuracy in these runs (see our general comment "Update: LNIS versus fixed schedules and versus clearing the library"). These are single runs, and the midpoint schedule consolidates once rather than twice. We revised the paper's attribution to "consolidation followed by a library reset"; all additions are marked in blue in the revised PDF.

If one concern remains decisive for your recommendation, a short note would let us address it before the deadline.

Reviewer_nZDUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when an LLM agent’s self-generated skill library should be consolidated into model weights. During practice, a harness extracts skills from failures, accepts them through a net-improvement gate, selects them by cost-normalized gain and MMR, and retrieves them for later problems. A heuristic trigger, LNIS, decides when to train a LoRA adapter on the retained skills and reset the library. Compared with two retrieval-only baselines, consolidation reduces held-out context by 5.6–5.8× and practice tokens by 34–39% on LiveCodeBench, while held-out accuracy is similar.

Critical strengths

The question of when in-context experience should move into weights is timely and well motivated. The incremental three-variant design cleanly separates the effect of library management from that of consolidation. The cost accounting is detailed and stage-by-stage, and the authors are candid about limitations and non-significant results.

Critical weaknesses

Missing a no-library baseline, and the benchmarks may not be skill-sensitive. In my view, the skill library brings little benefit on the chosen benchmarks. The baselines’ held-out accuracy with their libraries is close to that of the base model, and the remaining skills add nothing after consolidation (Table 20). The paper does not report a no-library baseline under the same held-out protocol, which is essential here. More importantly, if the skill library itself cannot meaningfully improve performance on a benchmark, then consolidating it into LoRA, and the efficiency claims built on top of that, are not very meaningful. I would encourage the authors to evaluate on skill-sensitive benchmarks where the library clearly helps, so that the consolidation question is actually worth answering. No variance across runs. Each variant’s practice is run only once (§6), so there is no estimate of how much the results vary between runs. The reported improvements are small and come from small held-out sets: 17.0% → 19.8% on LiveCodeBench (106 problems, i.e., 3 more problems solved, McNemar p = 0.61) and 62% → 64% on MATH-500 (50 problems, i.e., 1 more problem). Multiple independent practice runs are needed to show that these results are robust. I’m not convinced LNIS is doing much. EffDecl fires on any increase in tokens per injection, with no minimum threshold (§3.4), so it is hard to tell a real signal from noise. It ended up triggering three of the four consolidations, while AbilDecl and CtxFull never fired at all (§5.3). As a result, half of the trigger design is never actually tested, and its value over a simple fixed schedule remains unclear.

Questions for authors

N/A

6 comments
AuthorsOpenReview ↗

Response to Reviewer nZDU (1/2)

Missing a no-library baseline, and the benchmarks may not be skill-sensitive. In my view, the skill library brings little benefit on the chosen benchmarks. The baselines' held-out accuracy with their libraries is close to that of the base model, and the remaining skills add nothing after consolidation (Table 20). The paper does not report a no-library baseline under the same held-out protocol, which is essential here. More importantly, if the skill library itself cannot meaningfully improve performance on a benchmark, then consolidating it into LoRA, and the efficiency claims built on top of that, are not very meaningful. I would encourage the authors to evaluate on skill-sensitive benchmarks where the library clearly helps, so that the consolidation question is actually worth answering.

Weakness 1: We thank the reviewer for this important point. The no-library baseline under the held-out protocol is in Table 20 (all 50 MATH-500 held-out problems and a fixed random subset of 50 of the 106 LiveCodeBench held-out problems, Appendix E.1): the base model without a library scores 64.0% on MATH-500 and 16.0% on the LiveCodeBench subset, while the consolidated model without a library scores 68.0% and 24.0% on the same problems. The library's in-context gain is small (+2 to +3 points, Table 2), and our claim does not rest on it: even a weakly helpful library adds 1,731 input tokens to every LiveCodeBench problem (Table 6), and consolidation removes that cost. The libraries were active on both benchmarks, repairing 27 and 10 practice problems (Table 12). On 50 Omni-MATH problems never used in practice, the consolidated LiveCodeBench model solved 5 problems versus 2 for Baseline 1 (10.0% versus 4.0%), with 144 versus 2,414 input tokens per problem. That the remaining skills add nothing in Table 20 is the expected outcome: they had just been consolidated into the evaluated weights, so the model no longer needs them in context.

No variance across runs. Each variant's practice is run only once (§6), so there is no estimate of how much the results vary between runs. The reported improvements are small and come from small held-out sets: 17.0% → 19.8% on LiveCodeBench (106 problems, i.e., 3 more problems solved, McNemar p = 0.61) and 62% → 64% on MATH-500 (50 problems, i.e., 1 more problem). Multiple independent practice runs are needed to show that these results are robust.

Weakness 2: Held-out accuracy averages three sampling seeds, and Appendix E.4 repeats MATH-500 with paired three-seed tests (63.3% for both the full variant and Baseline 1, exact McNemar p = 1.0). At the trajectory level, two separate practice runs (the full variant and Baseline 1) reach the first LNIS trigger at windows 37 and 36 (Table 7). As the reviewer notes, 106 and 50 problems cannot resolve a 2 to 3 point difference; small test sets are underpowered for such differences [1]. We therefore claim only efficiency, and the token savings are not a sampling effect: they come from evaluating each candidate against a small library, which cut candidate-evaluation tokens from 34.88M to 14.67M on LiveCodeBench (Table 5). For compute-heavy pipelines, few-run evaluation with interval estimates is the accepted practice [2], which is what we report (Wilson intervals and paired exact tests, Appendix A.1 and A.2); even dedicated statistical studies of LLM-agent training average their main results over two seeds [3].

AuthorsOpenReview ↗

Response to Reviewer nZDU (2/2)

I'm not convinced LNIS is doing much. EffDecl fires on any increase in tokens per injection, with no minimum threshold (§3.4), so it is hard to tell a real signal from noise. It ended up triggering three of the four consolidations, while AbilDecl and CtxFull never fired at all (§5.3). As a result, half of the trigger design is never actually tested, and its value over a simple fixed schedule remains unclear.

Weakness 3: Three of the four consolidations fired on large changes: +140% and +67% in tokens per injection, and a MATH-500 trigger that met all four saturation conditions. Only one (LiveCodeBench window 114, +7.4%) depends on the missing threshold, and every threshold we tested up to 20% leaves the first consolidation on both benchmarks unchanged. AbilDecl and CtxFull were evaluated at every deployment window and stayed silent because their conditions never occurred: every LiveCodeBench problem received at least one skill, the share of uncovered MATH-500 problems fell to zero, and prompts peaked at 72.5% of the context, below the 90% threshold. On the value over a fixed schedule, please see the general response; consolidating on a detected plateau rather than at fixed points is the established approach when no task boundaries exist [4]. We will add this threshold analysis to the appendix.

The trigger behind each consolidation, from the run logs:

BenchmarkWindowSignalChange
LiveCodeBench37EffDecl+140% tokens per injection
LiveCodeBench114EffDecl+7.4% tokens per injection
MATH-50032EffDecl+67% tokens per injection
MATH-50070Saturationall four conditions held

References:

[1] Card, D., et al. "With Little Power Comes Great Responsibility." EMNLP, 2020.

[2] Agarwal, R., et al. "Deep Reinforcement Learning at the Edge of the Statistical Precipice." NeurIPS, 2021.

[3] Vattikonda, D., et al. "How to Train Your LLM Web Agent: A Statistical Diagnosis." NeurIPS, 2025.

[4] Aljundi, R., Kelchtermans, K., and Tuytelaars, T. "Task-Free Continual Learning." CVPR, 2019.

AuthorsOpenReview ↗

New result for Weakness 3 (EffDecl as signal or noise) and Weakness 2 (variation across runs)

We ran a new re-run of the full variant with the same seed, code and settings; since the extractor's outputs and GPU numerics are not exactly reproducible, it followed its own practice trajectory. The table gives the tokens per injection behind each first trigger of the full variant.

BenchmarkRunFirst trigger (window)Tokens per injection, before → after
LiveCodeBenchFull variant, original run371.22 → 2.92
LiveCodeBenchFull variant, new re-run313.47 → 15.42
MATH-500Full variant, original run3259.91 → 100.11
MATH-500Full variant, new re-run4131.49 → 49.30

In the new re-runs, the first trigger followed a rise of 12 and 18 tokens per injection, and on MATH-500 the original trigger followed a rise of 40. The original LiveCodeBench trigger was small in absolute terms (1.22 to 2.92), but it fired when prompts already filled 72.5% of the context (Table 8), the point at which consolidation pays off.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response for All Reviewers (1/1)

We thank all four reviewers and the Area Chair. Our central contribution is a mechanism for deciding when accumulated knowledge should move from an external skill library into model weights, and we evaluate it primarily by efficiency: consolidation removes the library's context cost, with 5.6 to 5.8 times fewer held-out input tokens on LiveCodeBench, 3.1 to 3.6 times on MATH-500, and 34% fewer practice tokens than Baseline 1 (Table 2), at no measurable accuracy loss. We do not claim a statistically established accuracy gain; accuracy point estimates are reported only to show that the efficiency comes at no measurable loss.

Why an adaptive trigger rather than a fixed schedule.

A fixed schedule must fix its interval in advance, but the right interval depends on how fast the library grows, which differs across tasks. LNIS spaced its two consolidations 77 windows apart on LiveCodeBench and 38 on MATH-500 (Table 7); no single fixed interval reproduces both. Our claim is that LNIS sets the timing from observed context cost without an interval chosen in advance, not that it outperforms every fixed schedule. Its first triggers (windows 37 and 32) coincided with sharp rises in context cost per injection, +140% and +67%, when prompts already filled 72.5% of the context (Table 8). Evaluated on Baseline 1, a separate practice run, LNIS would first have fired at window 36 (Table 7), next to the full variant's window 37. The logs show why one interval cannot serve both tasks. Without consolidation (Baseline 1), the LiveCodeBench library fills 70% of the context by window 9 and stays at that ceiling for the remaining 110 windows, while the MATH-500 library never exceeds 50% of the context in 91 windows. A fixed interval short enough for LiveCodeBench would retrain MATH-500's small library many times, each a full training run (about 0.6 h, Table 6) for a handful of skills; one long enough for MATH-500 would hold LiveCodeBench at its context ceiling for most of practice. With LNIS, LiveCodeBench context fell from 72% to 39 to 50% after the first consolidation and stayed there. Signal-based triggers are the established answer to deciding when to consolidate without fixed boundaries: task-free continual learning consolidates when a sliding window of recent losses reaches a plateau [1], the same principle as LNIS's comparison of recent regions. A benchmark-specific interval would have to be fixed before practice, from a growth rate that is only observed during practice; LNIS reads that rate online.

References:

[1] Aljundi, R., Kelchtermans, K., and Tuytelaars, T. "Task-Free Continual Learning." CVPR, 2019.

AuthorsOpenReview ↗

General Comment: a fixed-schedule baseline (new result**) (1/1)

Several reviewers asked how LNIS compares with a simple fixed consolidation schedule. We ran the simplest such schedule on LiveCodeBench with the same code and settings: no consolidation during practice, one consolidation at the end.

ScheduleConsolidationsPractice tokensHeld-out input tokens / problemHeld-out accuracy
No consolidation (Baseline 1)067.06M2,99017.0%
Fixed: once, at the end150.38M53516.0%
LNIS244.54M53019.8%

Practice tokens count every consolidation's data construction for both schedules; this is why the LNIS figure is 0.16M (in Table 2).

At the same held-out context cost, LNIS used 11.6% fewer practice tokens than consolidating at the end. A schedule that waits carries the growing library through all of practice, so every candidate is evaluated against it; LNIS consolidates when that cost rises. The accuracy difference (19.8% vs 16.0%) is not significant on 106 problems, and we do not claim it; the result shows no accuracy cost of consolidating earlier. This is one run of each schedule.

AuthorsOpenReview ↗

Update: training versus clearing the library on LiveCodeBench (new result**)

Edited: added the midpoint schedule.

We extended our fixed-schedule comparison (all reviewers; Reviewer Je1k, Question 1; Reviewer RVQR, Question 1) and added a run that clears the library at LNIS's windows without training (Reviewer Je1k, Question 2; Reviewer RVQR, Weakness 3 && Question 3). All runs use the same code and settings; practice tokens exclude the end-of-practice step for every schedule.

ScheduleConsolidations during practicePractice tokensHeld-out input tokens / problemHeld-out accuracy
Reset without training, at LNIS's windows053.99M1,05616.0%
Fixed: only at the end049.74M53516.0%
Fixed: once at the midpoint (window 59)147.13M53516.0%
LNIS244.38M53019.8%

Among the schedules completed so far, LNIS used the fewest practice tokens: 5.8% fewer than consolidating at the midpoint, 10.8% fewer than consolidating only at the end, and 17.8% fewer than clearing the library at the same windows without training, with no loss in accuracy. The ordering follows the mechanism: the longer a schedule carries the growing library, the more every candidate evaluation costs, and clearing the library without training forfeits what was learned. Clearing without training is evaluated with the skills gathered after its last reset still in the prompt, hence its longer held-out prompts.

AuthorsOpenReview ↗

Update: unseen benchmarks and forgetting (new results**)

Consolidation should keep what the library provides while removing its context cost. We tested this on benchmarks never used in practice (Reviewer rbhT, Question 3), evaluating a consolidated checkpoint from a new run of the same code (consolidation at the end of practice) against the library-based systems it replaces, three seeds each:

Benchmark (model)Baseline 1 (library)Baseline 2 (managed library)Consolidated (no library)Input tokens / problem
GSM8K, 500 problems (math model)93.6%93.3%94.7%90 vs 2,433 to 2,613
HumanEval, 164 problems (code model)61.8%74.8%70.1%194 vs 2,770 to 2,800

On both unseen benchmarks, the consolidated model stays within the range of the library-based systems: above both on GSM8K, and on HumanEval significantly above Baseline 1 (paired exact test, p = 0.02) and statistically indistinguishable from Baseline 2 (p = 0.11), while using 14 to 29 times less context. The gap between the two baselines on HumanEval is consistent with skill management pruning skills that do not transfer (37 retained versus 48).

On forgetting (Reviewer RVQR, Weakness 4), no previously solved problem was lost at this consolidation: 43 of 60 earlier practice problems were solved both before and after.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.