- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SLIM treats an agent's external skill set as something to manage during RL training, not a library that only grows or only shrinks. Its main contribution is a skill lifecycle that can add and remove skills over time. GRPO updates alternate with periodic audits. In each audit, every skill gets a leave-one-out "marginal external contribution" score (MEC, Eq. 3) on routed dev tasks, smoothed with an EMA. Based on that score the skill is kept or retired, and new skills are added where coverage is missing. The paper also gives a theoretical argument for when audit decisions are reliable (Appendix A). With Qwen3-4B, SLIM† reaches 87.5 on ALFWorld and 41.0 on SearchQA. Table 2 has ablations, and the appendix adds seeds, other backbones and WebShop.
Critical strengths
S1. The reframing is clean. "Accumulate" and "eliminate" become special cases of one lifecycle controller, and the controls target the obvious confounds. Fixed Active Set Size (75.6) covers "it's just a shorter prompt", and the frozen Qwen3-4B creator (86.7, Table A7) covers "it's just distillation from o3".
S2. The endpoint behavior holds up across settings. SearchQA ends with an almost empty set, while ALFWorld and WebShop keep compact ones, and set sizes are stable across seeds and backbones (Table A5). That gives the "task-dependent boundary" claim some real support.
Critical weaknesses
W1. The ALFWorld numbers, which carry most of the average gain, don't add up. The Avg. entries and the Table A4 run means are all multiples of 1/128, which points to a 128-episode test set rather than the 134-game official unseen split. The prompt baselines and SLIM without skills all fit one per-type split (35/5/32/13/23/20). The headline SLIM† row does not: 92.9 is not k/35, 78.3 is not k/13, and so on. That is odd, since SLIM and SLIM† use the same policy and test split. EvolveR's Avg. (39.8) also disagrees with its own per-type entries (about 46.1). On top of this, Look has 5 episodes, the bootstrap CI for the main gap is roughly ±9 points, and the Table 2 ablations are single runs. I need to know how every Table 1 number was produced, and I'd like seeds for the ablations.
W2. I can't tell what MEC adds over Skill0. Appendix B.2 describes Skill0's filter as the performance difference with and without each skill, which is essentially Eq. 3. SLIM also drops the zero target and adds expansion, and Table 2 doesn't separate these pieces. Random Audit is a weak control. At 32 dev tasks per audit, a skill's routed subset is a handful of episodes, so the Hoeffding-based bound in Appendix A is vacuous at the thresholds used. Decisions hinge on one or two episodes. The missing controls are a Skill0-style filter with SLIM's schedule and expansion, and SLIM with frequency-based retirement.
Questions for authors
- [Clarification] How were the ALFWorld entries in Table 1 computed (episodes per type, single run vs. average)? Why don't the SLIM† and several baseline rows fit the split the other rows imply? This answer matters most for my Soundness score.
- [Clarification] Why 128 test episodes rather than the 134 official unseen games, and which split does the 32-task dev set come from?
- [Clarification] What is the typical number of routed dev episodes per audited skill, and how many retire/retain decisions rest on a one- or two-episode difference?
- [Suggestion] Could you run the two controls in W2? If MEC clearly beats both, my concern largely goes away.
- [Clarification] SkillRL uses top-6 retrieval with KL and SLIM uses top-3 without. Does the 12.5-point gap survive matched settings?
8 comments
Rebuttal by Authors (2/5)
Review (W1 - Part 3): ... the bootstrap CI for the main gap is roughly ±9 points ...
Response: Table A6 quantifies sampling variation across the 128 binary test episodes. At the observed success rates on binary outcomes, the standard independent-proportion standard error is , which gives a 95% confidence interval half-width of 9.4 to 9.5 percentage points, matching our bootstrap intervals. Crucially, both 95% confidence intervals strictly exclude zero, with gains of +13.3 points and a 95% CI of over Skill0, and +12.6 points with a 95% CI of over SkillRL$^\dagger$.
Furthermore, Table A4 separately measures training stability across 3 independent training seeds. With Qwen3-4B, SLIM$^\dagger$ achieves , compared to for SkillRL$^\dagger$ and for Skill0, maintaining mean advantages of +11.72 and +14.58 points. With Qwen3-8B, SLIM$^\dagger$ achieves versus for SkillRL$^\dagger$. On SearchQA, the bootstrap 95% CI over SkillRL$^\dagger$ is much tighter at . Together, these results confirm that the gains are consistent across evaluation episodes, independent training seeds, and model scales.
Review (W1 - Part 4): ... and the Table 2 ablations are single runs. I'd like seeds for the ablations.
Response: Following your suggestion, we have extended all ALFWorld ablation variants in Table 2 to 3 independent training seeds and report the mean success rate and sample standard deviation below.
| Method | ALFWorld Success Rate (Mean Std %) |
|---|---|
| SLIM$^\dagger$ (Full) | 88.02 1.63 |
SLIM w/o Retirement | 73.70 1.19 |
SLIM w/o Expansion | 79.43 0.90 |
Random Audit | 67.19 1.56 |
Fixed Active Set Size | 75.00 0.78 |
The 3-seed results confirm all ablation conclusions from Table 2, with component gaps of 8.6 to 20.8 points that far exceed the run-to-run standard deviations. We have updated Table 2 in Section 6.3 of the revised paper with these multi-seed ablation results.
Rebuttal by Authors (3/5)
Review (W2 - Part 1 & Q3): What is the typical number of routed dev episodes per audited skill, and how many retire/retain decisions rest on a one- or two-episode difference?
Response: We have analyzed all 126 individual MEC audit records from the main ALFWorld training run, excluding the separately conducted general-group audits. Across these 126 audits, each audited skill is associated with an average of 19.7 routed evaluation records, with a median of 21.5 and a range of 7 to 31, including repeated evaluations on the same underlying development tasks.
To directly test sensitivity to one- or two-episode outcome flips, we retrospectively enumerate all worst-case perturbations to at most 1 or 2 binary rollout outcomes across the paired with-skill and skill-disabled evaluations for all 126 audit decisions, holding routing, exposure, and prior audit history fixed.
| Outcome Perturbations per Audit | Total Audit Decisions Changed | Observed Retirements Reversed |
|---|---|---|
| Up to 1 episode outcome flip | 3 / 126 (2.4%) | 0 / 33 (0.0%) |
| Up to 2 episode outcome flips | 5 / 126 (4.0%) | 2 / 33 (6.1%) |
Thus, 97.6% and 96.0% of all lifecycle decisions remain unchanged under worst-case one- and two-episode perturbations, and all observed retirements are invariant to single-episode flips due to EMA smoothing with , minimum exposure , and patience . We have included this perturbation analysis in Appendix D.4 of the revised paper.
Review (W2 - Part 2): I can't tell what MEC adds over Skill0. Appendix B.2 describes Skill0's filter as the performance difference with and without each skill, which is essentially Eq. 3. SLIM also drops the zero target and adds expansion, and Table 2 doesn't separate these pieces...
Response: Thanks for raising this point. First, Skill0 and SLIM evaluate fundamentally different counterfactual baselines. Skill0 evaluates each skill individually against a zero-skill baseline on a matched sub-task and ranks skills under a forced shrinking schedule toward zero-skill inference. In contrast, SLIM's leave-one-skill-out MEC removes only the audited skill from the current active set to form on its routed subset , measuring the marginal value of conditioned on the remaining active skills and refreshing baselines sequentially after each removal.
Second, Table 2 already isolates the lifecycle without expansion via SLIM w/o Expansion, which makes zero creator calls and still achieves across 3 seeds on ALFWorld, outperforming Skill0 at across 3 seeds (Table A4) by +5.99 points. Our new matched controls in Review (W2 - Part 4 & Q4) below further isolate the exact contribution of leave-one-skill-out MEC when paired with the identical SLIM schedule and expansion module.
Rebuttal by Authors (1/5)
We thank Reviewer JLUo for the rigorous review and constructive suggestions. We address every question below and have incorporated the unified evaluation table and new control experiments into the revised paper.
Response: Thank you for catching this aggregation inconsistency. We have thoroughly audited our evaluation logs and confirm that all methods in Table 1 are evaluated on the exact same 128 test episodes, with per-category counts of Pick at 35, Look at 5, Clean at 32, Heat at 13, Cool at 23, and Pick2 at 20, using a single evaluation run with one rollout per episode partitioned into 8 equal batches of 16 episodes.
The discrepancy in per-category decimals arises because two different category-aggregation formulas are used across the standalone inference script and the online-RL evaluation logger. Specifically, the standalone evaluation script used for prompt baselines and the post-training no-skill pass of SLIM computes pooled episode-level category success rates SRcpool=∑bnbc∑bsbc, which always equal integer numerators divided by the category totals 35,5,32,13,23, and 20. In contrast, the batched online-RL evaluator used for SLIM$^\dagger$, GRPO, GRPO$^\dagger$, EvolveR, SkillRL$^\dagger$, and Skill0 computes the category success rate within each 16-episode batch and averages those rates across the batches Bc containing category c, namely SRcbatch=∣Bc∣1∑b∈Bcnbcsbc. When category counts nbc vary across batches, SRcbatch differs slightly from SRcpool; for instance, batch counts [2,1,1,1] with successes [1,0,0,1] give a batch-averaged score of 37.5% versus a pooled episode score of 2/5=40.0%. Meanwhile, because all 8 batches contain exactly 16 episodes, the batch-averaged overall score and episode-pooled overall score are mathematically identical and always equal total successes divided by 128.
For EvolveR, its per-category entries in Table 1 are computed by the same 8-batch evaluator from the raw per-batch outcomes below, which sum to 59/128=46.1% overall success, whereas
39.8in Table 1 is a transcription error from an earlier checkpoint log.[8, 5, 5, 4, 4, 3, 3, 3][7, 1, 5, 0, 4, 1, 3, 3][0, 0, 0, 1, 1, 0, 2, 1][0, 0, 0, 0, 0, 0, 1, 1][5, 5, 4, 3, 3, 8, 2, 2][2, 5, 3, 2, 3, 1, 0, 0][1, 1, 1, 1, 2, 1, 2, 4][0, 0, 0, 0, 0, 1, 0, 1][1, 4, 3, 5, 3, 2, 3, 2][1, 1, 0, 4, 0, 0, 1, 1][1, 1, 3, 2, 3, 2, 4, 4][0, 0, 0, 2, 2, 2, 0, 1]In the revised paper, we have unified all rows in Table 1 under the same batch-averaged aggregation rule from the raw evaluation logs. This unification leaves all other overall
Avg.scores and all experimental conclusions completely unchanged. Thank you again for pointing out this critical issue.Response: Look contains 5 episodes because we preserve the exact 128-game test set from the released SkillRL [1] benchmark. Because Look accounts for only 5/128=3.91% of the overall test set, one additional solved Look episode changes the overall
Avg.score by only 0.78 percentage points. In Table 1, SLIM$^\dagger$ outperforms SkillRL$^\dagger$ by 12.5 points, corresponding to 16 additional solved episodes out of 128. Even in the extreme case where all 5 Look episodes contribute to this gap, at least 11 additional solved episodes come from the remaining categories, led by substantial improvements on Clean and Cool. Thus, variation on Look cannot account for SLIM$^\dagger$'s overall performance gain.