- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RASO uses a public skill corpus (GitSkills) as a prior for optimizing an agent's skill document. Skills are split into sections and retrieved with BM25. A "Cross-Harness Adaptation" LLM step turns retrieved sections into lessons phrased in the target task's and harness's vocabulary. RASI builds an initial skill without rollouts: it decomposes the task and harness descriptions into procedure steps and queries, then retrieves, adapts and synthesizes. RASU iteratively refines the skill from failed rollouts: it produces textual gradients and queries, retrieves, adapts, updates, and keeps an update only if validation improves. Across OfficeQA, SpreadsheetBench, ALFWorld and WebShop, with GPT-5.6-Luna and Qwen-3.5-9B, RASI beats retrieval-free initialization and SkillRouter retrieval. RASO beats TextGrad, GEPA, SkillOpt and WikiSkill under matched rollout budgets. Ablations cover init/update, adaptation, K, corpus size and cost.
Critical strengths
- A simple, broadly applicable idea, well controlled. All methods share the same task and harness descriptions and the same initial skill, use matched or smaller rollout budgets, and accept updates only on validation. A 2×2 init/update ablation, a fixed-RASI-init comparison against every update baseline, an adaptation on/off ablation, and K and corpus-size sweeps make the contribution of each stage visible.
- The benefits are largest where skill optimization struggles. With the 9B model, retrieval-free optimizers often barely move or even degrade the skill (Tables 6–7), while RASO improves WebShop from 13.4 to 24.7 and ALFWorld from 43.5 to 51.0. With Qwen-3.5-9B, RASI at zero rollouts already beats every retrieval-free rollout-based optimizer on all four benchmarks (Table 1).
Critical weaknesses
- The roles of the (agent-generated but highly procedural) task and harness descriptions, and of corpus contamination, are not separated from the effect of retrieval. The SpreadsheetBench H prescribes header=None, selective fill-forward, groupby(sort=False), safe_cell_value-style comparisons and reverse-order delete_rows. The ALFWorld H spells out a visited-ledger search policy, and the WebShop H gives the exact reward formula. Moving from No-skill to RFSI already takes OfficeQA from 11.4 to 40.1. Because adaptation keeps only claims "corroborated by H", a rich H may especially benefit RASO. The paper says H was not revised after observing test performance, but not whether it was refined using train or validation rollouts during development. Contamination control is a keyword blocklist at repository level. It would miss forum-derived or generic agent skills, which matters because SpreadsheetBench itself comes from public Excel-forum questions. No statistics are given on corpus size or removed repositories, and there is no overlap probe between retrieved sections and test instances or solutions.
- The evidence that retrieval specifically helps is weaker than the framing. RFSI omits lessons entirely. The missing control is the same number of adaptation calls producing lessons from parametric knowledge only, or from random sections, at equal LLM compute. Several "consistent" gains are within about 1–2 SE with 3 seeds and small test sets: GPT ALFWorld +1.49 (SE ≈1.0/0.9), WebShop +1.07, OfficeQA +3.49 (SE 1.85/2.05), Qwen Spreadsheet +1.79. Paired or bootstrap CIs are needed. The SkillRouter baseline (one whole skill, no adaptation) is weak by construction.
Questions for authors
Clarifications:
- How were the H and T descriptions developed? Were they ever edited after looking at train or validation rollouts or RASO outputs? How does performance change with a minimal H (tools and I/O contract only)?
- Corpus statistics after sectioning, repositories and sections removed per benchmark, the most-retrieved source repositories per benchmark, and an n-gram or embedding overlap check against test instructions and gold solutions.
- Results for "RFSI + parametric lessons" (same adaptation prompt, no retrieved text) and "random sections + adaptation".
- Is RFSU exactly RASU minus retrieval and adaptation?
- Do learned skills transfer across target models (optimizer ≠ target)? Suggestions: analyze harness-mismatch errors before and after adaptation quantitatively. Move the 50+ pages of verbatim prompts and skills to the supplement. The GoS preprocessing-cost extrapolation in Appendix A.6 is tangential to the claims.
5 comments
[W2 - 3] Role of the SkillRouter baseline SkillRouter represents the standard setting in the retrieval-based skill reuse literature: reusing a whole skill without adaptation. Its role is to show that direct reuse fails across harnesses. The stronger control, section-level retrieval with only adaptation removed, is the no-adaptation variant in Table 3.
[Q4] Definition of RFSU RFSU is exactly RASU with retrieval and adaptation removed. RFSU used textual gradients derived from the train rollouts to refine the current skill, without access to an external skill corpus.
[Q5 - 1] Cross-model transfer of optimized skills Skills learned by RASO transfer across target models. Skills optimized with Qwen and evaluated on GPT and vice-versa outperform GPT and Qwen’s performance with No skill on both benchmarks.
| Source → Target | OQA | SSB |
|---|---|---|
| Qwen → GPT | 46.32 ± 0.84 | 53.57 ± 3.00 |
| GPT No Skill | 11.44 ± 0.70 | 32.98 ± 0.12 |
| GPT → Qwen | 36.43 ± 0.85 | 30.12 ± 1.34 |
| Qwen No Skill | 33.14 ± 1.2 | 28.81 ± 0.60 |
[Q5 - 2] Harness-mismatch errors before and after adaptation Cross-Harness Adaptation removes harness-mismatch errors. We counted retrieved sections that require unexecutable commands or unavailable tools in GPT-5.6-Luna. As the table below shows, this rate drops from 40.0–72.7% before adaptation to 0% on every benchmark after adaptation.
| Benchmark | Before adaptation | After adaptation |
|---|---|---|
| OQA | 40/55 (72.7%) | 0/55 (0%) |
| SSB | 28/64 (43.8%) | 0/64 (0%) |
| ALF | 24/60 (40.0%) | 0/60 (0%) |
| WS | 36/57 (63.2%) | 0/57 (0%) |
[Q5 - 3] Presentation improvements We thank the reviewer for these suggestions. In the revision, we will move the verbatim prompts and skills to the supplementary material and condense or remove the GoS preprocessing-cost extrapolation in Appendix A.6.
Dear Reviewer JKbq,
Thank you again for your review.
With the discussion deadline approaching, we would appreciate your feedback on our rebuttal, particularly the minimal-harness experiments, lesson-generation controls, and additional contamination and statistical analyses addressing your main concerns. Please let us know if any points would benefit from further clarification.
We thank the reviewer for the careful and thorough review.
[W1, Q1 - 1] Development of task and harness descriptions Task description T and harness description H were developed without any rollouts. As stated in the paper, we generated them with Codex from the benchmark source code. During development, we used an LLM-as-a-judge at each stage of RASI to check that the intended behavior was produced. No train or validation rollouts and no test performance were used to refine T or H.
[W1, Q1 - 2] RASO performance with minimal harness description The advantage of RASO does not depend on the content of the harness description. With a minimal harness description, the ordering RFSI < RASI < RASO holds on all benchmarks (table below) and still outperforms the strong baseline, SkillOpt. For OfficeQA (OQA), the original harness description in Appendix A.8 is already minimal. Our rollout analysis suggests that OQA’s 28.7-point gain from No skill to RFSI stems mainly from producing answers in the normalized format required by the exact-match scorer. On SpreadsheetBench, all three variants improved with the minimal description. Our rollout analysis suggests that some of the original guidance led generated skills to recommend actions that hindered execution.
[W1, Q2 - 1] Corpus statistics and blocklist details The corpus contains 242,732 repositories, 1,877,981 canonical files, and 17,183,585 sections. The table below reports the repositories and sections removed by the blocklist, together with the most-retrieved source repository for each benchmark.
majiayu000/claude-skill-registry-datatools-only/X-Skillshiyenwong/ai_collectiondvcrn/openclaw-skills-marketplace[W1, Q2 - 2] Overlap analysis against test data Retrieved sections share no content with the test data. Across all benchmarks, the exact 8-gram and 12-gram overlap between retrieved sections and the test instructions or gold solutions is 0. This also holds for SpreadsheetBench, whose tasks originate from public Excel forums.
[W2, Q3] Contribution of retrieved content The gains of RASI come from the retrieved content itself, not from the extra generation step. We compare two controls that share RASI’s per-requirement generation step. Random section replaces retrieved sections with randomly sampled sections from the corpus. Retrieval-free (self-generated) lesson generates lessons from the LLM’s own knowledge without retrieval. The table below shows the results. Parametric lesson trails RASI by 5.89 points on OQA and 4.10 points on SSB, and it even falls below RFSI on OQA. The extra generation step alone therefore does not explain the gains. Random section improves over RFSI but still trails RASI by 5.10 and 4.39 points. This shows that external procedural knowledge provides information the LLM lacks, and that relevance-based retrieval adds further gains.
[W2 - 2] Statistical significance of RASO gains RASO outperforms the strongest baseline in all 8 settings (sign test, p = 0.008). Per-setting 95% CIs from a paired bootstrap over test instances exclude zero in 5 of 8 settings, including GPT WebShop. Pooled over all settings, the mean gain is +4.72 points (95% CI [+2.78, +6.66]).