acceptodds
Under review as a conference paper at ICLR 2027

Retrieval-Augmented Skill Optimization via Cross-Harness Adaptation

11 views

Abstract

An agent skill is a reusable, actionable natural-language artifact that guides an agent to perform a task effectively under a given harness. Recent studies have explored the optimization of agent skills, contributing to a growing collection of publicly available skills spanning diverse tasks, domains, and harnesses. Despite millions of publicly shared skills, existing skill optimization methods largely overlook this accumulated knowledge, instead relying solely on expensive agent rollouts to iteratively refine skills for a target task. To address this, we propose Retrieval-Augmented Skill Optimization (RASO), a framework that leverages an external skill corpus as prior knowledge throughout skill optimization. RASO retrieves relevant knowledge from existing skills and adapts it to the target task and harness via Cross-Harness Adaptation, accounting for mismatches in both domain and harness. RASO comprises two complementary stages: Retrieval-Augmented Skill Initialization (RASI) constructs a knowledge-grounded initial skill without requiring agent rollouts, while Retrieval-Augmented Skill Update (RASU) iteratively refines the skill by retrieving external knowledge guided by execution feedback. Across four agent benchmarks and two models, extensive experiments show that RASO consistently outperforms baselines without retrieval-augmented skill initialization and updating.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Skill, Retrieval-Augmented Skill Optimization, Cross-harness adaptation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_JKbqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RASO uses a public skill corpus (GitSkills) as a prior for optimizing an agent's skill document. Skills are split into sections and retrieved with BM25. A "Cross-Harness Adaptation" LLM step turns retrieved sections into lessons phrased in the target task's and harness's vocabulary. RASI builds an initial skill without rollouts: it decomposes the task and harness descriptions into procedure steps and queries, then retrieves, adapts and synthesizes. RASU iteratively refines the skill from failed rollouts: it produces textual gradients and queries, retrieves, adapts, updates, and keeps an update only if validation improves. Across OfficeQA, SpreadsheetBench, ALFWorld and WebShop, with GPT-5.6-Luna and Qwen-3.5-9B, RASI beats retrieval-free initialization and SkillRouter retrieval. RASO beats TextGrad, GEPA, SkillOpt and WikiSkill under matched rollout budgets. Ablations cover init/update, adaptation, K, corpus size and cost.

Critical strengths
  1. A simple, broadly applicable idea, well controlled. All methods share the same task and harness descriptions and the same initial skill, use matched or smaller rollout budgets, and accept updates only on validation. A 2×2 init/update ablation, a fixed-RASI-init comparison against every update baseline, an adaptation on/off ablation, and K and corpus-size sweeps make the contribution of each stage visible.
  2. The benefits are largest where skill optimization struggles. With the 9B model, retrieval-free optimizers often barely move or even degrade the skill (Tables 6–7), while RASO improves WebShop from 13.4 to 24.7 and ALFWorld from 43.5 to 51.0. With Qwen-3.5-9B, RASI at zero rollouts already beats every retrieval-free rollout-based optimizer on all four benchmarks (Table 1).
Critical weaknesses
  1. The roles of the (agent-generated but highly procedural) task and harness descriptions, and of corpus contamination, are not separated from the effect of retrieval. The SpreadsheetBench H prescribes header=None, selective fill-forward, groupby(sort=False), safe_cell_value-style comparisons and reverse-order delete_rows. The ALFWorld H spells out a visited-ledger search policy, and the WebShop H gives the exact reward formula. Moving from No-skill to RFSI already takes OfficeQA from 11.4 to 40.1. Because adaptation keeps only claims "corroborated by H", a rich H may especially benefit RASO. The paper says H was not revised after observing test performance, but not whether it was refined using train or validation rollouts during development. Contamination control is a keyword blocklist at repository level. It would miss forum-derived or generic agent skills, which matters because SpreadsheetBench itself comes from public Excel-forum questions. No statistics are given on corpus size or removed repositories, and there is no overlap probe between retrieved sections and test instances or solutions.
  2. The evidence that retrieval specifically helps is weaker than the framing. RFSI omits lessons entirely. The missing control is the same number of adaptation calls producing lessons from parametric knowledge only, or from random sections, at equal LLM compute. Several "consistent" gains are within about 1–2 SE with 3 seeds and small test sets: GPT ALFWorld +1.49 (SE ≈1.0/0.9), WebShop +1.07, OfficeQA +3.49 (SE 1.85/2.05), Qwen Spreadsheet +1.79. Paired or bootstrap CIs are needed. The SkillRouter baseline (one whole skill, no adaptation) is weak by construction.
Questions for authors

Clarifications:

  1. How were the H and T descriptions developed? Were they ever edited after looking at train or validation rollouts or RASO outputs? How does performance change with a minimal H (tools and I/O contract only)?
  2. Corpus statistics after sectioning, repositories and sections removed per benchmark, the most-retrieved source repositories per benchmark, and an n-gram or embedding overlap check against test instructions and gold solutions.
  3. Results for "RFSI + parametric lessons" (same adaptation prompt, no retrieved text) and "random sections + adaptation".
  4. Is RFSU exactly RASU minus retrieval and adaptation?
  5. Do learned skills transfer across target models (optimizer ≠ target)? Suggestions: analyze harness-mismatch errors before and after adaptation quantitatively. Move the 50+ pages of verbatim prompts and skills to the supplement. The GoS preprocessing-cost extrapolation in Appendix A.6 is tangential to the claims.
5 comments

We thank the reviewer for the careful and thorough review.

[W1, Q1 - 1] Development of task and harness descriptions Task description T and harness description H were developed without any rollouts. As stated in the paper, we generated them with Codex from the benchmark source code. During development, we used an LLM-as-a-judge at each stage of RASI to check that the intended behavior was produced. No train or validation rollouts and no test performance were used to refine T or H.

[W1, Q1 - 2] RASO performance with minimal harness description The advantage of RASO does not depend on the content of the harness description. With a minimal harness description, the ordering RFSI < RASI < RASO holds on all benchmarks (table below) and still outperforms the strong baseline, SkillOpt. For OfficeQA (OQA), the original harness description in Appendix A.8 is already minimal. Our rollout analysis suggests that OQA’s 28.7-point gain from No skill to RFSI stems mainly from producing answers in the normalized format required by the exact-match scorer. On SpreadsheetBench, all three variants improved with the minimal description. Our rollout analysis suggests that some of the original guidance led generated skills to recommend actions that hindered execution.

BenchmarkSettingRFSIRASIRASOSkillOpt
OQAOriginal (= Minimal)40.11 ± 0.5845.74±1.4049.03±1.8545.54 ± 2.05
SSBOriginal44.40 ± 2.7449.17 ± 1.5263.33 ± 2.6957.02 ± 3.60
SSBMinimal58.21 ± 1.6463.45 ± 2.2367.62 ± 2.0762.74 ± 2.52
ALFOriginal69.40 ± 1.5572.64 ± 1.6374.13 ± 1.0072.64 ± 0.90
ALFMinimal69.90 ± 0.6671.89 ± 0.2573.88 ± 0.8671.90 ± 1.18
WSOriginal43.89 ± 1.1745.06 ± 0.4046.61 ± 0.1145.17 ± 0.23
WSMinimal42.21 ± 1.8944.72 ± 0.8246.38 ± 0.3245.12 ± 0.28

[W1, Q2 - 1] Corpus statistics and blocklist details The corpus contains 242,732 repositories, 1,877,981 canonical files, and 17,183,585 sections. The table below reports the repositories and sections removed by the blocklist, together with the most-retrieved source repository for each benchmark.

BenchmarkRemoved reposRemoved sectionsMost-retrieved repository
OQA10245,102majiayu000/claude-skill-registry-data
SSB3140tools-only/X-Skills
ALF9441,466hiyenwong/ai_collection
WS8439,691dvcrn/openclaw-skills-marketplace

[W1, Q2 - 2] Overlap analysis against test data Retrieved sections share no content with the test data. Across all benchmarks, the exact 8-gram and 12-gram overlap between retrieved sections and the test instructions or gold solutions is 0. This also holds for SpreadsheetBench, whose tasks originate from public Excel forums.

[W2, Q3] Contribution of retrieved content The gains of RASI come from the retrieved content itself, not from the extra generation step. We compare two controls that share RASI’s per-requirement generation step. Random section replaces retrieved sections with randomly sampled sections from the corpus. Retrieval-free (self-generated) lesson generates lessons from the LLM’s own knowledge without retrieval. The table below shows the results. Parametric lesson trails RASI by 5.89 points on OQA and 4.10 points on SSB, and it even falls below RFSI on OQA. The extra generation step alone therefore does not explain the gains. Random section improves over RFSI but still trails RASI by 5.10 and 4.39 points. This shows that external procedural knowledge provides information the LLM lacks, and that relevance-based retrieval adds further gains.

OQASSB
RFSI40.11 ± 0.5844.40 ± 2.74
random section40.64 ± 1.0344.78 ± 0.94
self-generated lesson39.85 ± 0.5145.07 ± 1.83
RASI45.74 ± 1.4049.17 ± 1.52

[W2 - 2] Statistical significance of RASO gains RASO outperforms the strongest baseline in all 8 settings (sign test, p = 0.008). Per-setting 95% CIs from a paired bootstrap over test instances exclude zero in 5 of 8 settings, including GPT WebShop. Pooled over all settings, the mean gain is +4.72 points (95% CI [+2.78, +6.66]).

BackboneBenchmarkΔ (RASO-Next best)95% bootstrap CIP(Δ≤0)
GPT-5.6-LunaOQA+3.49[−0.39, +7.36]0.040
GPT-5.6-LunaSSB+6.31[+3.57, +9.05]<0.001
GPT-5.6-LunaALF+1.49[−1.74, +4.98]0.200
GPT-5.6-LunaWS+1.07[+0.11, +2.00]0.015
Qwen-3.5-9BOQA+4.84[+1.16, +8.53]0.005
Qwen-3.5-9BSSB+1.79[−2.14, +5.60]0.184
Qwen-3.5-9BALF+7.46[+2.49, +12.69]0.002
Qwen-3.5-9BWS+11.30[+9.23, +13.42]<0.001

[W2 - 3] Role of the SkillRouter baseline SkillRouter represents the standard setting in the retrieval-based skill reuse literature: reusing a whole skill without adaptation. Its role is to show that direct reuse fails across harnesses. The stronger control, section-level retrieval with only adaptation removed, is the no-adaptation variant in Table 3.

[Q4] Definition of RFSU RFSU is exactly RASU with retrieval and adaptation removed. RFSU used textual gradients derived from the train rollouts to refine the current skill, without access to an external skill corpus.

[Q5 - 1] Cross-model transfer of optimized skills Skills learned by RASO transfer across target models. Skills optimized with Qwen and evaluated on GPT and vice-versa outperform GPT and Qwen’s performance with No skill on both benchmarks.

Source → TargetOQASSB
Qwen → GPT46.32 ± 0.8453.57 ± 3.00
GPT No Skill11.44 ± 0.7032.98 ± 0.12
GPT → Qwen36.43 ± 0.8530.12 ± 1.34
Qwen No Skill33.14 ± 1.228.81 ± 0.60

[Q5 - 2] Harness-mismatch errors before and after adaptation Cross-Harness Adaptation removes harness-mismatch errors. We counted retrieved sections that require unexecutable commands or unavailable tools in GPT-5.6-Luna. As the table below shows, this rate drops from 40.0–72.7% before adaptation to 0% on every benchmark after adaptation.

BenchmarkBefore adaptationAfter adaptation
OQA40/55 (72.7%)0/55 (0%)
SSB28/64 (43.8%)0/64 (0%)
ALF24/60 (40.0%)0/60 (0%)
WS36/57 (63.2%)0/57 (0%)

[Q5 - 3] Presentation improvements We thank the reviewer for these suggestions. In the revision, we will move the verbatim prompts and skills to the supplementary material and condense or remove the GoS preprocessing-cost extrapolation in Appendix A.6.

Dear Reviewer JKbq,

Thank you again for your review.

With the discussion deadline approaching, we would appreciate your feedback on our rebuttal, particularly the minimal-harness experiments, lesson-generation controls, and additional contamination and statistical analyses addressing your main concerns. Please let us know if any points would benefit from further clarification.

Reviewer_V3B6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RASO retrieves sections from an external skill corpus and rewrites their procedural advice for the target agent's tools and constraints. It uses this process both to initialize skills without environment rollouts and to revise them from execution feedback. Experiments cover four benchmarks and two models, with comparisons against retrieval-free initialization and existing skill optimizers.


Thank you for the detailed rebuttal and additional experiments. The self-generated lesson control and fixed-initialization comparisons substantially address my two main concerns, and the evaluation clarification resolves my remaining question. Although the optimization gains remain modest, I find the new evidence convincing and have raised my rating accordingly.

Critical strengths
  1. Direct skill reuse can hurt: on ALFWorld, GPT-5.6-Luna scores 55.97 with SkillRouter versus 64.43 without skills. Table 3 reports that adaptation improves its SpreadsheetBench initialization score from 41.43 to 49.17. RASI improves over retrieval-free generation in all eight model-benchmark settings without requiring environment rollouts.

  2. Starting from the same RASI skill, RASU achieves the highest mean performance on OfficeQA and SpreadsheetBench for both models. These experiments use disjoint training, validation, and test sets and report results over three seeds.

Critical weaknesses
  1. The retrieval-free control may be unnecessarily constrained. Appendix A.1 attributes the RASI-RFSI gap to retrieved and adapted knowledge rather than the prompt. Yet the shared synthesis prompt says "add nothing that is not in" the supplied material. RFSI receives the task and harness descriptions and concern decomposition, but no adaptation lessons. It therefore loses an intermediate reasoning stage while remaining restricted from adding knowledge during synthesis. The comparison does not separate the value of retrieved documents from the value of letting the model write procedural advice itself. Since the competing optimizers also start from RFSI, this limitation affects the Table 1 comparisons as well.

  2. On Qwen ALFWorld and WebShop, RASO's advantage over competing optimizers comes mostly from initialization. On Qwen, RASI alone exceeds every competing optimizer on ALFWorld (47.76 versus at most 43.53) and WebShop (23.27 versus at most 13.43). RASO adds 3.24 and 1.46 points after initialization, reaching 51.00 and 24.73, respectively. Fixed-initialization comparisons are limited to OfficeQA and SpreadsheetBench; corresponding comparisons on ALFWorld and WebShop are needed to support the update claim.

Questions for authors
  1. Can you retain the intermediate lesson-generation calls without retrieved sections, allowing the model to draw on its own knowledge? Also, what does the no-adaptation variant in Table 3 receive?

  2. Can you compare the update methods from the same RASI initialization on WebShop and ALFWorld, especially for Qwen? Do the update gains persist once the initial skill is matched?

  3. Please clarify the OfficeQA evaluator: Appendix A.3 describes numerical correctness within a relative tolerance, whereas the harness description in A.8 specifies normalized exact match. Which rule produced the reported results, and did the optimizer receive an accurate description?

2 comments

We thank the reviewer for the careful and thorough review.

[W1, Q1] Value of retrieved knowledge over self-generated lessons Retrieved documents provide value beyond letting the model write procedural lessons itself. We keep the intermediate lesson-generation calls but remove the retrieved sections, so the model writes lessons from its own knowledge. Since these lessons are passed to synthesis as supplied material, the synthesis prompt does not restrict their use. The table below shows that parametric lesson trails RASI by 5.89 points on OfficeQA (OQA) and 4.10 points on SpreadsheetBench (SSB), and it falls below RFSI on OQA. This also shows that RFSI is not an unduly weak initialization for the competing optimizers in Table 1, since adding self-generated lessons does not consistently improve it. The no-adaptation variant in Table 3 receives the raw retrieved sections directly, without Cross-Harness Adaptation.

OQASSB
RFSI40.11 ± 0.5844.40 ± 2.74
self-generated lesson39.85 ± 0.5145.07 ± 1.83
RASI45.74 ± 1.4049.17 ± 1.52

[W2, Q2] Additional optimization results under RASI initialization RASU achieves best mean performance when every method starts from the same RASI initialization on ALFWorld and WebShop (Table below). RASO is also the only method that improves over the RASI initialization in every setting, while each competing optimizer falls below RASI in at least one setting. Averaged over the four settings, RASU adds +1.93 points over RASI, while baselines show +0.48 (SkillOpt), +0.46 (WikiSkill), +0.10 (GEPA), and −3.98 (TextGrad).

GPT-5.6-Luna

ALFWorldWebshop
RASI72.64 ± 1.6345.06 ± 0.40
TextGrad71.64 ± 2.2836.09 ± 4.92
GEPA73.38 ± 1.1044.86 ± 0.41
SkillOpt71.64 ± 2.4045.90 ± 0.28
WikiSkill73.88 ± 1.1446.00 ± 0.16
RASO74.13 ± 1.0046.61 ± 0.11

Qwen-3.5-9b

ALFWorldWebshop
RASI47.76 ± 3.5323.27 ± 0.98
TextGrad49.50 ± 3.3515.57 ± 2.84
GEPA48.01 ± 2.5322.89 ± 1.08
SkillOpt49.75 ± 1.6323.37 ± 1.74
WikiSkill48.51 ± 3.8822.18 ± 1.19
RASO51.00 ± 2.4524.73 ± 0.41

[Q3] OfficeQA evaluation rule All reported OfficeQA results use exact match. We set the relative tolerance described in Appendix A.3 to 0, so the evaluation rule is identical to the normalized exact match in the harness description (Appendix A.8). The optimizer therefore received an accurate description of the evaluator. We will state this in the implementation details.

Dear Reviewer V3B6,

Thank you again for your review.

With the discussion deadline approaching, we would appreciate your feedback on our rebuttal, particularly the self-generated-lesson control and the fixed-initialization comparisons on ALFWorld and WebShop addressing your two main concerns. Please let us know if any points would benefit from further clarification.

Reviewer_AifWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes RASO, which uses a public skill corpus as prior knowledge in both stages of agent skill optimization:

  1. RASI (initialization): Uses no rollouts. It generates requirements and queries from the task and harness descriptions, retrieves sections with BM25, and synthesizes an initial skill.
  2. RASU (update): Generates text grad and queries from those failed trajectories, retrieves sections, and updates the skill.

Both stages share Cross-Harness Adaptation, which rewrites retrieved content into lessons that fit the target task and harness. The validation covers four benchmarks and two models swtting. The paper reports that RASO has the highest mean in every setting compared with the baselines.

Critical strengths
  1. RASI delivers real gains without spending rollouts. Its mean beats RFSI on all four benchmarks with both models, and some gaps far exceed the error. This matters because rollouts are the main cost of existing methods. Some other gaps are small, though, such as +1.17 on GPT WebShop.

  2. The method is simple and the evaluation setup is careful. BM25 plus one LLM adaptation step is easy to reproduce and deploy. The authors also exclude benchmark-targeting skills at the repository level, fix T and H across methods, share the same task decomposition with RFSI, and report costs. These choices make the results more credible and practical.

Critical weaknesses
  1. The contribution of the retrieved content itself is not isolated. RFSI simply omits the lessons, so the gap between RASI and RFSI combines the retrieved content with the effect of an extra per-requirement generation step. Using just 1% of the corpus already gives a clear gain, and the useful section in the qualitative example comes from the distant domain of satellite imagery. Both fit an alternative explanation: retrieved text may only act as a prompt, with the knowledge really coming from the LLM. I found no random-retrieval or retrieval-free-lesson control.

  2. The evidence for RASU’s incremental advantage is weak. In Table 1, RASO starts from RASI while the baselines start from RFSI. With initialization fixed, the gaps shrink considerably (GPT OfficeQA +1.94, Qwen Spreadsheet +0.48), and most fall within the error range. There are only three seeds, and I found no significance tests.

Questions for authors
  1. How do random-retrieval and retrieval-free-lesson controls perform? If they come close to RASI, does the central claim need to change?

  2. Were K=5 and the other hyperparameters selected on the validation set or the test set? Which data split do Figures 3 and 4 report?

  3. The contributions state that the gains do not depend on particular source documents. What systematic analysis supports this, beyond the corpus-size curve and the single qualitative example?

5 comments

We thank the reviewer for the careful and thorough review.

[W1, Q1] Contribution of retrieved content The gains of RASI come from the retrieved content itself, not from the extra generation step. We compare two controls that share RASI’s per-requirement generation step. Random section replaces retrieved sections with randomly sampled sections from the corpus. Retrieval-free (self-generated) lesson generates lessons from the LLM’s own knowledge without retrieval. The table below shows the results. Parametric lesson trails RASI by 5.89 points on OfficeQA (OQA) and 4.10 points on SpreadsheetBench (SSB), and it even falls below RFSI on OQA. The extra generation step alone therefore does not explain the gains. Random section improves over RFSI but still trails RASI by 5.10 and 4.39 points. This shows that external procedural knowledge provides information the LLM lacks, and that relevance-based retrieval adds further gains.

OQASSB
RFSI40.11 ± 0.5844.40 ± 2.74
random section40.64 ± 1.0344.78 ± 0.94
self-generated lesson39.85 ± 0.5145.07 ± 1.83
RASI45.74 ± 1.4049.17 ± 1.52

[W2] Statistical significance of RASU under fixed initialization Under fixed initialization, RASU outperforms the strongest baseline in all 8 settings (2 initial skills × 2 benchmarks × 2 backbones), with a mean gain of +1.88 points (paired t-test over settings, p = 0.003; Wilcoxon signed-rank test, p = 0.008). To isolate the effect of the update procedure, we compare RASU with the strongest competing update method in each setting under the same initial skill (Tables 4, 6, 7, and 8). The strongest competing baseline is selected separately for each setting, making the performance comparison conservative toward RASU.

[Q2] Data splits for hyperparameter selection and reporting All hyperparameters, including K = 5, were selected on the validation set. Figures 3 and 4 report performance on the test set.

[Q3] Robustness across disjoint corpus shards The gains of RASI do not depend on particular source documents. We partitioned the corpus into five disjoint shards and ran RASI on each shard independently. The table below shows that every shard outperforms RFSI on both benchmarks, by +3.11 to +4.65 points on OQA and +2.00 to +3.58 points on SSB. The variation across shards is small. Since no documents are shared between shards, the gains stem from the general availability of procedural knowledge rather than from specific documents.

BenchmarkShard 1Shard 2Shard 3Shard 4Shard 5RFSI
OQA44.09 ± 2.0543.22 ± 1.0843.60 ± 2.3344.76 ± 1.0843.41 ± 1.3840.11 ± 0.58
SSB46.40 ± 2.7447.98 ± 2.8647.62 ± 3.0147.10 ± 0.9546.69 ± 2.9844.40 ± 2.74

Thanks for the detailed response and the new experiments.

The random-section and self-generated-lesson controls answer my main concern.

A few small things for the revision:

  1. Random section and self-generated lesson vs RFSI are +0.4 to +0.7, well within the std, so I'd say "no clear difference from RFSI" rather than "improves".

  2. The 8 fixed-init settings share benchmarks and backbones, so the t-test isn't over independent samples. "Wins in 8/8" is fine, but individual gaps (e.g., Qwen Spreadsheet +0.48) are still within error.

  3. Please confirm K = 5 was chosen from the validation curve, not from Fig 3 on test, and add the val curve to the appendix. Also say how the shards were split (random or by repo).

Thanks again.

Thank you for the reply.

We are glad that our rebuttal addressed your main concern.

[R1] As suggested, we will state that the random-section and self-generated-lesson controls show no clear difference from RFSI. We will also revise our interpretation to clarify that these results support the contribution of relevant retrieved content beyond additional generation steps or arbitrary corpus text.

[R2] Thank you for the suggestion. In the revision, we will report that RASU achieves the highest mean performance in 8/8 settings under fixed initialization, while explicitly acknowledging that several individual improvements are modest and within the reported uncertainty.

[R3-1] As stated in [Q2] of our rebuttal, we selected K = 5 based on validation performance, not the test results in Figure 3. The validation results (mean over three seeds) for each K are shown below. We will add the corresponding curves to the appendix.

OQAK=1K=3K=5K=10
RASI38.8943.0651.3950.00
RASU45.8350.0058.3356.94
SSBK=1K=3K=5K=10
RASI40.0043.3351.6750.83
RASU59.1761.6770.8368.33

[R3-2] For the shard experiment, we randomly partitioned the corpus at the skill level into five disjoint shards with approximately equal numbers of skills. We chose skill-level partitioning to avoid large corpus-size imbalances, as repository-level partitioning could result in substantially different numbers of skills per shard due to variation in repository sizes. We will clarify this procedure in the revision.

Reviewer_rCN6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Retrieval-Augmented Skill Optimization, a framework that leverages the external GitSkills corpus as prior knowledge throughout skill optimization. It combines rollout-free Retrieval-Augmented Skill Initialization with trajectory-guided Retrieval-Augmented Skill Update, both sharing section-level BM25 retrieval and a Cross-Harness Adaptation operator that translates mismatched external procedures into the vocabulary and constraints of the target harness. Evaluated across four agent benchmarks and two models, the method consistently outperforms both direct skill retrieval and retrieval-free optimization baselines.

Critical strengths
  • The controlled comparisons under fixed initializations and matched rollout budgets are consequential because they confirm that retrieving external procedural knowledge during iterative updates resolves failure modes that trajectory-only reflection misses while achieving the lowest API cost.
  • Combining rollout-free section-level retrieval with Cross-Harness Adaptation is consequential because 92.9 percent of retrieved external skills originate from mismatched harnesses where direct reuse degrades accuracy below the no-skill baseline, whereas adapting them yields strong zero-rollout initial skills.
Critical weaknesses
  • Using lexical BM25 retrieval and benchmark-name blocklisting is consequential because keyword matching misses abstract cross-domain mechanisms and fails to exclude near-duplicate task tutorials that omit benchmark names.

  • Relying on source-code-extracted harness descriptions is consequential because Appendix A.8 already supplies domain solutions like openpyxl flags, driving a 28.7-point jump without retrieval on OfficeQA versus 5.6 additional points from retrieval.

Questions for authors
  • What proportion of retrieved sections come from distant domains versus standard openpyxl tutorials after keyword blocklisting, and how does dense retrieval compare to BM25?

  • Do update baselines in Table 1 also receive the solution-rich harness descriptions in Appendix A.8, and how does RASO perform using only minimal tool API signatures?

2 comments

We thank the reviewer for the careful and thorough review.

[W1, Q1] Retrieval source comparison between BM25 and dense retrieval BM25 retrieves a substantial share of cross-domain material, and its cross-domain shares and downstream scores closely match those of dense retrieval (SkillRouter-0.6B). Using an LLM judge, we labeled each section used in the RASO runs on SpreadsheetBench (SSB) and OfficeQA (OQA) as in-domain if its source skill and the target benchmark share an application domain, and cross-domain otherwise. Dense retrieval changes the cross-domain share by at most 2.2 percentage points and mean RASI and RASO scores by at most 0.78 points. In-domain sections include general spreadsheet tutorials such as openpyxl usage, but none reproduce test content: across all benchmarks, we found zero exact 8-gram or 12-gram matches between retrieved sections and test instructions or gold solutions.

BenchmarkRetrieverRASIRASOCross-domainIn-domain
SSBBM2549.17±1.5263.33±2.6933.8%66.2%
SSBDense retrieval48.77±2.4363.10±1.1535.5%64.5%
OQABM2545.74±1.4049.03±1.8558.6%41.4%
OQADense retrieval44.96±1.9849.42±1.6556.4%43.6%

[W2, Q2 - 1] Harness description usage in update baselines All update baselines in Table 1 start from the same initialization, which is generated by taking the harness description in Appendix A.8 as an input. Since the update procedures of these baselines are not designed to leverage a harness description, we applied their original algorithms without modification. Instead, Table 2 provides a controlled comparison, showing that the gain comes from retrieval rather than from access to the harness descriptions. RFSU receives the same task and harness descriptions as RASU but performs no retrieval. Starting from the same RFSI and RASI, RFSU still underperforms RASU (40.70 vs. 47.56 on OfficeQA, 51.67 vs. 61.07 on SpreadsheetBench).

[W2, Q2 - 2] RASO performance with minimal harness description The advantage of RASO does not depend on the content of the harness description. With a minimal harness description, the ordering RFSI < RASI < RASO holds on all benchmarks (table below) and still outperforms the strong baseline, SkillOpt. For OfficeQA (OQA), the original harness description in Appendix A.8 is already minimal. Our rollout analysis suggests that OQA’s 28.7-point gain from No skill to RFSI stems mainly from producing answers in the normalized format required by the exact-match scorer. On SpreadsheetBench, all three variants improved with the minimal description. Our rollout analysis suggests that some of the original guidance led generated skills to recommend actions that hindered execution.

BenchmarkSettingRFSIRASIRASOSkillOpt
OQAOriginal (= Minimal)40.11 ± 0.5845.74±1.4049.03±1.8545.54 ± 2.05
SSBOriginal44.40 ± 2.7449.17 ± 1.5263.33 ± 2.6957.02 ± 3.60
SSBMinimal58.21 ± 1.6463.45 ± 2.2367.62 ± 2.0762.74 ± 2.52
ALFOriginal69.40 ± 1.5572.64 ± 1.6374.13 ± 1.0072.64 ± 0.90
ALFMinimal69.90 ± 0.6671.89 ± 0.2573.88 ± 0.8671.90 ± 1.18
WSOriginal43.89 ± 1.1745.06 ± 0.4046.61 ± 0.1145.17 ± 0.23
WSMinimal42.21 ± 1.8944.72 ± 0.8246.38 ± 0.3245.12 ± 0.28

Dear Reviewer rCN6,

Thank you again for your review.

With the discussion deadline approaching, we would appreciate your feedback on our rebuttal, particularly the dense-retrieval comparison and the minimal-harness experiments addressing your two main concerns. Please let us know if any points would benefit from further clarification.

Comments on the paper

1 comment

We thank the reviewers for recognizing the strengths of RASO and for their constructive feedback. Below, we summarize the recognized strengths and the additional evidence provided during rebuttal.

Strengths recognized by the reviewers

  • [rCN6, AifW, V3B6, JKbq] Effective rollout-free initialization (RASI). RASI outperforms retrieval-free initialization in all eight model–benchmark settings without environment rollouts. With Qwen-3.5-9B, it also surpasses every evaluated retrieval-free rollout-based optimizer on all four benchmarks.
  • [rCN6, V3B6] Effective retrieval-augmented updates (RASU) beyond trajectory-only reflection. External knowledge helps address failures missed by trajectory-only reflection, with the lowest API cost among the compared methods. Under the same RASI initialization, RASU achieves the highest mean performance.
  • [rCN6, V3B6] Effective cross-harness adaptation. Direct skill reuse can perform worse than using no skill. Adapting retrieved procedural knowledge produces effective initial skills despite substantial source–target harness mismatch.
  • [rCN6, AifW, V3B6, JKbq] Controlled evaluation. The evaluation includes matched rollout budgets, fixed-initialization comparisons, component ablations, disjoint data splits, three-seed results, and cost reporting.
  • [AifW, JKbq] Simple and practical design. The BM25-plus-adaptation framework is simple, reproducible, and practical to deploy.
  • [JKbq] Strong gains where skill optimization struggles. With the 9B model, retrieval-free optimizers often barely improve or degrade the skill, while RASO delivers substantial gains on ALFWorld and WebShop.

Additional evidence provided during rebuttal

  • [AifW, V3B6, JKbq] Contribution of retrieved procedural knowledge. RASI’s gains come from relevant retrieved content rather than additional generation calls alone. Keeping the intermediate lesson-generation stage, we compared relevant retrieval with self-generated lessons and randomly sampled sections. Both controls show no clear difference from RFSI, while RASI achieves higher performance, supporting the value of relevance-based retrieval.
  • [rCN6, JKbq] Effectiveness with minimal harness descriptions. RASO’s advantage persists without detailed procedural guidance. OfficeQA’s original harness description is already minimal; for the other benchmarks, we repeated experiments using only minimal tool and I/O descriptions. RASI continues to outperform RFSI, and RASO further improves performance across all four benchmarks. Detailed harness guidance therefore does not explain the retrieval gains. Task and harness descriptions were generated from benchmark source code without rollouts and were not refined using train or validation rollouts or test performance.
  • [AifW, V3B6] Contribution of retrieval-augmented updates. RASU contributes beyond stronger initialization. We extended fixed-initialization comparisons to ALFWorld and WebShop, where RASU achieves the highest mean performance for both models and improves over the shared RASI initialization in every setting. Together with the original fixed-initialization comparisons, these results support the contribution of retrieval-augmented updating.
  • [rCN6] Cross-domain retrieval with BM25. BM25 retrieves substantial cross-domain material and achieves performance comparable to dense retrieval in our experiments. Comparing both retrievers showed similar cross-domain proportions and downstream RASI/RASO scores, supporting BM25’s effectiveness for retrieving useful procedural knowledge.
  • [JKbq] Resolution of harness-mismatch errors. Cross-Harness Adaptation eliminated references to unexecutable commands and unavailable tools in the analyzed sections across all four benchmarks in GPT. This directly supports its role in translating retrieved procedures into guidance compatible with the target harness.
  • [rCN6, JKbq] Overlap analysis of retrieved knowledge. Our checks found no exact 8-gram or 12-gram matches between retrieved sections and test instructions or gold solutions across all benchmarks, including SpreadsheetBench’s forum-derived tasks. These results provide no evidence of direct textual overlap at the tested n-gram lengths.
  • [JKbq] Statistical support for overall performance gains. Paired bootstrap comparisons over test instances support the advantage of the full RASO framework over the strongest competing baseline in most model–benchmark settings, with confidence intervals excluding zero. We acknowledge that some individual improvements remain within statistical uncertainty.

Reviewers AifW and JKbq explicitly confirmed that their main concerns were addressed and maintained their weak-accept recommendations. We will incorporate the remaining clarifications and qualifications discussed during rebuttal into the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.