- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies when an agent’s growing skill library should be transferred from retrieved context into model weights. During practice, an extractor proposes skills from failed attempts; these skills are tested, selected using cost-normalized gain and diversity-aware ranking, and stored for retrieval. The proposed LNIS (Local Net-Improvement Saturation) trigger uses recent outcome and retrieval statistics to initiate LoRA training on skill-conditioned examples, followed by a conditional library reset.
Critical strengths
- The paper asks a meaningful question beyond whether agents can accumulate experience: when should useful experience remain in context, and when is parameter consolidation worth its training cost? This gives the work a clear operational objective.
- The three variants share skill extraction and selection, while progressively adding management and consolidation. This provides an informative initial comparison and shows that the reported benefits cannot simply be attributed to the particular pruning procedure alone.
Critical weaknesses
- The central question of when to consolidate remains insufficiently tested. The experiments compare LNIS-triggered consolidation with no consolidation, but not with alternative consolidation schedules. This establishes the potential usefulness of the full package, not the value of LNIS as a timing policy.
- The headline efficiency claims require stronger measurement and clearer accounting. The reported merged-adapter wall-clock results are recalibrated estimates, not measurements from an actual merged-adapter execution.
Questions for authors
- What evidence specifically supports LNIS over simpler consolidation schedules?
- At the same consolidation checkpoints, could the authors compare keeping the library, clearing it without training, training while retaining it, and training followed by reset?
4 comments
Thank you for providing a detailed and sufficient reply. I keep my score unchanged.
Edited on [11/17/26]: added the measurements for Weakness 2 and Question 2 below.
Thank you for reading our response and for your time. In case it is useful, we have since posted a general comment with a new fixed-schedule baseline: on LiveCodeBench, LNIS used 10.8% fewer practice tokens than consolidating only at the end, at the same held-out context cost. If a specific concern still stands in the way of a higher score, we would be glad to address it during the remaining discussion period.
Added: measurements for Weakness 2 and Question 2, on a consolidated checkpoint from a new run of the same code (MATH-500, consolidation at the end of practice).
Weakness 2: measured, not estimated. On the same 50 held-out problems, decoding cost per output token is:
| Model | ms per output token |
|---|---|
| Base model | 12.18 |
| Merged adapter | 12.09 |
| Unmerged adapter | 21.70 |
Merging removes the adapter's overhead entirely (−0.7% versus the base model), confirming the estimate of Appendix A.14 with a direct measurement.
Question 2: the four arms on the full held-out set, three seeds.
| Arm | Seed 0 | Seed 1 | Seed 2 | Mean |
|---|---|---|---|---|
| Clear library, no training | 56% | 64% | 62% | 60.7% |
| Keep library, no training | 60% | 58% | 60% | 59.3% |
| Train and keep library | 68% | 66% | 62% | 65.3% |
| Train and reset | 66% | 64% | 66% | 65.3% |
In every seed, training followed by a reset matches or exceeds both untrained arms; on average it adds 4.7 points over clearing the library alone (95% CI [0.7, 9.3]), and keeping the library after training adds nothing. Table 20 reported a single sampling run of a different consolidated checkpoint; its base-model value (64.0%) is one of the values observed here, and both evaluations rank the arms identically, with the two trained arms on top.
Final follow-up before the discussion closes
Thank you again for engaging with our response. Since your 15 November assessment, we added results for each of your requests (see our updated reply above and the general comment "Update: LNIS versus fixed schedules and versus clearing the library"), all marked in blue in the revised PDF:
- Simpler schedules (Q1): on LiveCodeBench, LNIS used 5.8% fewer practice tokens than midpoint consolidation and 10.8% fewer than end-only consolidation (single runs).
- Measured merged adapter (W2): 12.09 ms per output token, versus 12.18 for the base model and 21.70 unmerged.
- Training versus clearing (Q2): on all 50 MATH-500 held-out problems with three sampling seeds, train-and-reset averaged 65.3% versus 60.7% for clearing without training.
With the discussion ending within a day, we would be grateful to know whether a specific concern still keeps the paper below the acceptance threshold.
Response to Reviewer Je1k (1/1)
Weakness 1 && Question 1: We thank the reviewer for the precise requests. As detailed in the general response, the 77- versus 38-window spacing cannot come from one fixed interval, and three of the four triggers fired on large rises in context cost. Consolidating on a detected change rather than at fixed points is the established approach when no task boundaries exist [1].
Weakness 2: Our primary claim is in tokens, which do not depend on hardware and need no recalibration (Section 4.3); Table 5 and Figure 2 give the stage-by-stage accounting, including training and data construction. For wall-clock, the unmerged adapter's overhead was measured directly (78 to 82%, Appendix A.13); because merging folds the adapter into the base weights and adds no inference latency [2], we estimate merged-adapter time from the measured base-model speed; using the directly measured overhead instead changes no total by more than 0.2 h (Appendix A.14). We will label these values explicitly as estimates in the revision.
Question 2: These four arms are reported in Table 20 at the final consolidation checkpoint:
Training followed by a reset is best or tied on both benchmarks while using no context for skills.
The same comparison is available at every earlier consolidation checkpoint from Appendix A.8 and A.9, on the ten most recent practice problems:
On these recent problems, the trained model without the library matches or exceeds the library it replaces at every checkpoint, which is the condition for the reset.
References:
[1] Aljundi, R., Kelchtermans, K., and Tuytelaars, T. "Task-Free Continual Learning." CVPR, 2019.
[2] Hu, E. J., et al. "LoRA: Low-Rank Adaptation of Large Language Models." ICLR, 2022.