- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SRT, which generates and maintains skill-level regression tests from historical trajectories and uses them to accept, repair, or reject updates. Experiments across three benchmarks and four evolution operators show improved retention with a small loss in adaptation performance.
Critical strengths
- The paper addresses how agents can retain existing capabilities as their skills evolve. This is a central problem in continual agent learning that arises repeatedly in practice.
- The framework is simple and easy to follow. Using regression tests, repair, and rollback provides a straightforward and effective way to manage skill updates.
- The experiments are fairly extensive, covering multiple benchmarks, evolution operators, and ablations. Overall, the results provide good support for the authors’ main claims.
Critical weaknesses
- Test generation, regression checking, and repair are well-established ideas. SRT mainly brings them together for skill evolution, so I find the methodological novelty somewhat limited. What it adds beyond existing testing and repair mechanisms could be clearer.
- I would expect the framework to depend heavily on how diverse and representative its history is. The paper uses a buffer of 64 trajectories, but does not sufficiently explain how they are selected, what information is retained, or how performance changes with the amount and coverage of the history.
- The framework trades off retention, adaptation, and computational cost. The ablations provide some support for the defaults, but the selection procedure for the rejection threshold, repair budget, and other parameters remains unclear. Were these values tuned on a separate validation set, why were they chosen, and would different tasks require different settings?
Questions for authors
Please see the questions raised in the weaknesses. I would consider raising my score if the authors can satisfactorily address these concerns.
4 comments
Rebuttal to Reviewer KSrP (2/2)
AW2(2/2). We have added buffer-size and selection-policy sweeps in Table 8 (Lines 790–800) ↗.
| Buffer size | 8 | 16 | 32 | 64 (default) | 128 |
|---|---|---|---|---|---|
| Coverage ↑ | 0.66 | 0.75 | 0.83 | 0.86 | 0.87 |
| Erosion ↓ | 10.4 | 8.9 | 7.7 | 7.3 | 7.2 |
| CPI ↑ | 116.1 | 117.9 | 119.4 | 119.9 | 119.8 |
| Hidden pass ↑ | 82.5 | 86.4 | 89.4 | 90.3 | 90.6 |
| Overhead ↓ | 13.1% | 13.3% | 13.6% | 14.0% | 15.2% |
| Selection (size 64) | FIFO | Reservoir (default) | Success/failure-stratified | Diversity (k-center) |
|---|---|---|---|---|
| Erosion ↓ | 7.9 | 7.3 | 7.0 | 7.1 |
| CPI ↑ | 119.2 | 119.9 | 120.1 | 120.1 |
Performance saturates at 32–64 trajectories; doubling to 128 adds 1.2 points of overhead without gain, and even 8 trajectories cut erosion from 18.7 (vanilla) to 10.4. Selection matters less than size: stratified and diversity selection are 0.2 CPI above the default, within noise, while FIFO is worst because it discards early failed trajectories, the source of negative predicates (consistent with the 4.1-point erosion increase without negatives, Table 22).
Values are Lifelong×SkillRL means over three seeds. In the main runs, skills that start with short histories show 9.8 erosion vs. 6.9 for full buffers; this comparison is descriptive, since rarely used skills also differ in how often they are edited (Lines 805–806 ↗).
W3. The framework trades off retention, adaptation, and computational cost. [...] Were these values tuned on a separate validation set, why were they chosen, and would different tasks require different settings?
AW3(1/2). We have stated the selection protocol in Section 5 (Lines 349–352) ↗ and Appendix B (Lines 753–755) ↗. All defaults (β=0.5, K=3, L=8, T=50, buffer 64) were selected once on a validation split of 400 Lifelong AgentBench old tasks with SkillRL, disjoint from the held-out retention set. They were then fixed for all twelve benchmark–operator cells without per-setting tuning.
AW3(2/2). We have added hyperparameter sweeps and a cross-benchmark check in Table 30 (Lines 1350–1365) ↗.
| Repair budget K | 0 | 1 | 2 | 3 (default) | 5 |
|---|---|---|---|---|---|
| Repair success ↑ | — | 36% | 58% | 73.2% | 79% |
| A_adapt ↑ | 57.7 | 59.3 | 60.3 | 60.9 | 61.0 |
| CPI ↑ | 116.4 | 118.2 | 119.3 | 119.9 | 119.9 |
| Overhead ↓ | 12.3% | 13.2% | 13.7% | 14.0% | 14.9% |
| Threshold β | 0.3 | 0.5 (default) | 0.7 |
|---|---|---|---|
| Erosion ↓ | 5.8 | 7.3 | 9.1 |
| A_adapt ↑ | 59.3 | 60.9 | 61.4 |
| CPI ↑ | 119.8 | 119.9 | 118.6 |
| Test CPI | Lifelong | SealQA | OfficeQA Pro |
|---|---|---|---|
| Configuration selected per benchmark | default | β=0.3, K=3 | default |
| Default | 119.9 | 72.9 | 130.4 |
| Per-benchmark selection | 119.9 | 73.3 | 130.4 |
The defaults sit on plateaus: K=5 adds no CPI at 0.9 points more overhead, K=2 is a reasonable cheaper choice, and β in [0.3, 0.5] gives nearly equal CPI while 0.5 keeps more adaptation. Per-benchmark selection changes test CPI by at most 0.4, within one seed standard deviation, so a single default suffices in our benchmarks.
Sweeps use Lifelong×SkillRL with three seeds; in the cross-benchmark check each benchmark selects on its own validation split. Appendix M.1 (Lines 1341–1349) ↗ gives the details, and Table 33 (Lines 1442–1452) ↗ adds the joint β–K surface with audit error rates.
Thank you for your rebuttal, which has addressed most of my concerns. I will raise my score to 3.
Thank you
We thank the reviewer for the follow-up and for raising the score. We are glad the buffer, hyperparameter, and novelty analyses addressed the concerns, and we will keep them in the final version.
Rebuttal to Reviewer KSrP (1/2)
AW1. We have added an evaluation of off-the-shelf transplants in Table 15 (Lines 965–970) ↗ and rewritten the end of Section 2 (Lines 152–160) ↗.
Each established technique, transplanted directly, fails on evolving skills in a measurable way. Because a skill is a natural-language and code artifact executed by a stochastic policy without a fixed code oracle, preservation needs paraphrastic inputs with typed equivalence, trace and negative predicates mined from failed trajectories, and a suite frozen across versions and during repair so that the edit under review cannot shape its own tests.
Hidden pass is from Table 23 (Lines 1147–1155) ↗, F1 from the 600-update audit in Table 25 (Lines 1188–1195) ↗, CPI is the micro-average over the six settings of Table 3 (Lines 432–443) ↗, and the CPE row is Lifelong×SkillRL at λ=0.3 / 0.7 (Table 2). Table 14 (Lines 949–959) ↗ compares the closest work along seven axes.
AW2(1/2). We have specified the buffer in Section 4 (Lines 201–205) ↗ and Appendix B.1 (Lines 785–788) ↗. Each skill keeps up to 64 trajectories in which it was retrieved, selected by reservoir sampling over its retrieval history. Each entry stores the task input, full tool-call trace, final answer, verifier outcome, and skill version used. Skills with fewer than 64 trajectories use all available ones; at the start of the stream this applies to 41% of skills.
(AW2(2/2) and W3 continue in Part 2/2.)