- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper defines skill debt as a performance gap between a deployed skill library and a budget-matched counterfactual alternative. It proposes Counterfactual Skill Stewardship (CSS), which uses paired skill-availability ablations, a temporally cross-fitted utility estimator, and active/frozen/retired states to manage a growing library. SkillDebtBench combines synthetic interventions with lifelong and interactive agent environments. The reported results suggest improved reward and recurrence retention, with auditing costs amortized over long streams. The problem is relevant, but the retirement criterion and the specification of audit evidence and costs contain issues that undermine confidence in the central claims.
Critical strengths
-
The paper correctly distinguishes retrieval or invocation frequency from a skill's incremental effect on task performance. Removing one skill and rerunning retrieval under matched budgets can expose competition at the top-K boundary, making the proposed measurements more informative than historical success alone.
-
Preserving recoverable frozen skills is a sensible response to recurring task distributions. The evaluation also includes useful controls such as information-matched prediction, compute-matched reranking, and deletion instead of freezing, and reports a setting where CSS does not outperform SkillOps. This is a promising experimental framework if the underlying decision rules and accounting can be validated.
Critical weaknesses
-
The retirement rule does not establish the repeated harmful utility that it claims to require. Equation (4), Appendix B.3, and Algorithm 1 all retire eligible skills when their LOWER confidence bounds are below -epsilon in two audits. A negative lower bound does not provide confident evidence of a negative effect; the relevant directional criterion would normally use an UPPER bound below -epsilon. With the stated beta=1.645 and epsilon=0.02, a utility estimate of +0.02 with standard error 0.05 has LCB=-0.06225 and UCB=+0.10225. It satisfies the harmfulness inequality while remaining compatible with substantial benefit, and its standard error passes the stated 0.10 uncertainty gate. With sufficient observations and no protective predicates, two such audits permit retirement. Because this rule affects subsequent library contents and retention, resolving it may require rerunning the central experiments rather than correcting an isolated notation error.
-
The audit specification is insufficient to validate the reported evidence and efficiency. Appendix B.3 specifies 31 paired comparisons every 250 tasks, but calls this 124 additional shadow executions per 1,000 tasks. Algorithm 1 executes both branches, which would give 124 pairs or 248 executions unless factual branches are reused or shared under a valid current-library replay rule. Table 11 additionally reports the quantity as extra calls. The distinction matters for the 8.6% overhead and compute-matched baselines. Separately, Equations (1)/(6) give observed-or-imputed averages, while propensity weighting and the combination of uncertainty components are only described verbally. Successive 1,000-task windows overlap by 750 tasks, leaving the required independence of consecutive audits unexplained. These gaps prevent checking both calibrated retirement evidence and the claimed cost of obtaining it.
Questions for authors
-
Algorithm clarification: Does the implementation use LCB or UCB for retirement? Please provide the exact predicate and representative decision records. If it uses the printed LCB rule, how do cumulative reward, false retirement, and recurrence retention change under a criterion that establishes negative utility?
-
Accounting clarification: Does the number 124 denote pairs, complete trajectory executions, or individual model calls? How are factual replay reuse and multi-call trajectories charged, and do the reported overhead and compute-matched baselines include both branches under the same current library?
-
Estimator clarification: Please provide the exact propensity-weighted estimator and confidence-interval construction, including clipping and refitting, and explain how the two retirement audits obtain independent evidence despite overlapping task windows. Are coverage and harmful-skill decisions evaluated on held-out randomized audits?