- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates "silent staleness," a critical lifecycle issue where repository-specific agent skills become obsolete and misleading after software releases without raising explicit errors. The authors introduce Repo2Skill-Evo, a benchmark of 105 official release transitions across 57 repositories, pairing fixed skill sets with release patches and patch-verified obsolete line annotations to evaluate whether frontier LLM agents can reliably maintain externalized procedural knowledge.
Critical strengths
- Well-written paper and easy to follow. The core concept of framing agent skills as versioned knowledge assets is practical, timely, and well-motivated.
- Strong motivating study showing that fixed skills match source-code access in downstream utility while reducing token consumption by over 80%, clearly justifying the value of skill maintenance.
- High-quality dataset construction across 57 real-world repositories with manually verified gold obsolete lines, complemented by thorough trace-level failure diagnostics and oracle localization ablations.
Critical weaknesses
-
The physical line-based removal metric is overly rigid and penalizes benign paraphrasing or formatting adjustments of valid context lines as "off-target edits," which conflicts with the high retention scores observed in the NL Judge.
-
The empirical story lacks an end-to-end evaluation loop on downstream tasks. Without testing the stale skills versus agent-maintained skills on actual tasks, it remains unclear how severely silent staleness degrades real execution or whether a ~70% maintenance is already sufficient in practice.
-
Missing comparison against a "re-distillation from scratch" baseline. It is unclear how incremental patch-based maintenance compares to simply re-running the Repo2Skill distillation pipeline (or equipping the maintenance agent with the static Grounding Gate from Section 3.1) on the repository in terms of token cost and final quality.
Questions for authors
Refer to the Weaknesses
2 comments
Thank you for the detailed response. My concerns have been addressed, I will maintain my original score.
Maintenance Metrics, Downstream Utility, and Regeneration Cost
We thank the reviewer for the constructive feedback. Our additional experiments connect maintenance F1 to downstream utility and compare incremental maintenance with full V2 regeneration.
1. F1 reflects the requirement for minimal, patch-grounded updates
The maintenance prompt explicitly requires minimal, patch-grounded edits and prohibits unrelated rewriting, reformatting, or cosmetic changes. Under this objective, rewriting valid content without release evidence counts as an off-target modification even when the original information remains intact. Precision measures the fraction of removed or revised V1 lines belonging to the gold stale set; Recall measures how much of that set is covered.
2. Maintained skills improve downstream V2 utility
We evaluated one V2 artifact-generation task for each of the same ten repositories used in the motivating study. All conditions use the same GPT-5.4 executor, tools, budget, and V2 scoring rules. Ten conditions × ten tasks × three runs yield 300 artifact evaluations.
F1 values are existing avg@3 macro scores on this subset. Utility measures the correctness and usefulness of generated V2 task artifacts.
All six maintainers improve mean utility over stale V1 skills. Claude and GPT achieve gains of 1.78 and 1.60 points, respectively. These gains show practical benefits at the observed F1 levels; the adequacy of an update depends on which instructions a downstream task requires.
For the 180 maintained outputs, we compute Spearman correlation within each transition across six models × three runs, then average across ten transitions. The resulting mean ρ = 0.81 indicates a strong positive association between F1 and utility within matched tasks.
This association supports retaining F1 as the primary metric across 105 transitions. F1 is computed deterministically from output differences and fixed, patch-grounded annotations. Downstream utility also depends on task choice, executor behavior, sampling, and grading, making it a complementary measure of practical value.
3. Regeneration improves mean utility at substantially higher preparation cost
We regenerated skills directly from V2 repositories through fully autonomous runs of the original Repo2Skill multi-stage pipeline. Regeneration reconstructs skill packages from V2 source; maintenance updates existing V1 skills using the official release patch.
Values are mean logged input-plus-output tokens for V2 skill preparation over the same ten transitions and three runs. Ratios and savings are calculated before rounding.
Regeneration improves mean utility over same-model maintenance by 0.26 points for GPT and 0.22 for Claude. Maintenance saves 75.3% and 72.4% of preparation tokens, respectively. These results quantify the trade-off between the additional utility of reconstruction and the lower token cost of updating existing assets.