acceptodds
Under review as a conference paper at ICLR 2027

StraTune: Adaptive Selection of Revision Operators for Self-Evolving LLM Skills

27 views

Abstract

Large language models (LLMs) can learn reusable textual skills from execution feedback without updating their parameters, but effectively deciding how to revise these skills remains a key challenge. Existing methods typically rely on a fixed revision operator, a search strategy and the revision forms applied under it. However, we observe that no single revision operator consistently performs best across tasks, and repeatedly applying an unsuitable operator can limit further improvement. We propose StraTune (strategy-guided skill tuning), which lets a frozen optimizer LLM choose the revision operator at every round from the optimization state, which is defined as the current execution feedback together with the recorded outcomes of earlier strategies and forms. Candidate skills from every revision operator pass one candidate evaluation, which screens for gains and regressions on a small sample set and validates them on a larger one, and every outcome is written back to the optimization state for later choices. Across four benchmarks and two LLM settings, StraTune outperforms all five baselines in most settings. Ablations attribute the gains to the adaptive choice of the revision operator, since fixed, random, scheduled, and bandit strategy choices all score lower, and skills learned with a small target LLM also improve a stronger one.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM prompt optimization, Self-evolving LLM agents, LLM skill learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Hoq3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents StraTune, a method for optimizing textual skills while keeping model parameters frozen. An optimizer LLM dynamically selects search strategies and revision forms based on execution feedback and revision histories, while a two-stage evaluation procedure filters candidate skills. Experiments cover four benchmarks, two model configurations, and transfer of learned skills to a stronger target model.

Critical strengths
  1. Clear motivation. Different tasks and optimization stages may benefit from different revision operators. Making search strategies and revision forms part of the per-round decision is a reasonable and practical design.
  2. Broad component analysis. The experiments examine fixed operators, random and cyclic schedules, histories, revision forms, and candidate evaluation. Table 4 further attempts to separate the contributions of candidate generation and evaluation.
  3. Practically useful results. StraTune achieves the best or joint-best performance in most evaluated settings, and skills optimized with a smaller target model can be reused directly with a stronger model.
Critical weaknesses
  • The substantial Math gains are difficult to interpret as improvements in mathematical reasoning. The released skills explicitly exploit an option stating that another option is correct but a stronger result can be proven; the Opus skill instructs the model to select this option whenever it appears. The provided samples also show a strong association between this wording and the correct answer. Learning this pattern from training data does not imply test leakage, but without subgroup results and a simple heuristic baseline, the source of the largest reported gains remains unclear.
  • The advantage of adaptive selection over simple schedules lacks sufficient statistical support. On Math, cyclic selection achieves 65.40, close to StraTune’s 65.88, while the best fixed operator achieves 63.51. Most baselines and ablations are evaluated once, whereas four-run statistics are provided only for StraTune, leaving the robustness of smaller differences uncertain.
  • Some experimental conclusions require narrower interpretation. The bandit selectors primarily receive numerical rewards, whereas the LLM selector also receives semantic failure information, confounding the selection mechanism with its inputs. Moreover, matching target-model executions does not establish equal total cost, and transfer is evaluated only within one model family on unchanged tasks.
Questions for authors
  • Could the authors report Math accuracy separately for questions with and without the “stronger result” option, for the initial skill, main baselines, and StraTune? How much of the gain is explained by a baseline that selects this option whenever present and otherwise uses the initial skill?
  • Could the authors provide repeated runs for the main baselines, random selection, cyclic selection, and the best fixed operator to establish whether adaptive selection provides a consistent advantage?
  • Could different selectors be compared under matched feedback information, with complete token or monetary costs reported across methods? Does skill transfer also hold across model families or under changes in the task distribution?
6 comments

Q1. LiveMath accuracy with and without the "stronger result" option.

We agree that this option is a pattern in LiveMath. Whenever it appears, it is the correct answer in 177 training questions and 86 test questions. Table R1 splits the Table 1 scores by whether a question contains this option. Following the reviewer's suggestion, it also includes the rule-based baseline the reviewer describes (Option rule). This hand-crafted rule selects this option whenever it appears and otherwise answers with the initial skill.

Table R1. LiveMath test accuracy by question type (Haiku 4.5 target LLM, Sonnet 4.6 optimizer LLM, Table 1 runs).

MethodAll (211)With the option (86)Without the option (125)
Initial skill29.381.248.8
GEPA43.6044.243.2
TextGrad31.752.352.0
SkillOpt45.0248.842.4
Trace2Skill33.188.150.4
Iter-CoT36.9714.052.8
StraTune65.8879.156.8
Option rule69.67100.048.8

StraTune also improves the questions without the option. On these 125 questions, it has the highest accuracy of all methods and is significantly above GEPA (p = 0.04) and SkillOpt (p = 0.005) in a paired sign-flip test. On the 86 questions with the option, StraTune's skill is also the best of all learned skills, and the target LLM chooses the option far more often with it than with the skills of GEPA and SkillOpt. The Option rule method is written with the pattern known in advance, whereas StraTune discovers it from execution feedback. This prior knowledge gives Option rule an unfair advantage on the 86 questions with the option, where it is always correct. On the 125 questions without the option, however, it does no better than the initial skill, while StraTune has the highest accuracy there. Option rule therefore scores higher overall only because of the questions with the option.

Q2. Repeated runs of the baselines and controls.

We agree that single runs of the controls cannot show whether the advantage is consistent. We therefore ran the three baselines that are strongest on at least one dataset in Table 1 (GEPA, SkillOpt, and Iter-CoT), Rot., Rand., and the best fixed search strategy three more times each, with new seeds and the same budget and data splits. Together with the run in the main manuscript, each setting now has four runs (Table R2).

Table R2. Test scores over four runs, mean ± standard deviation (Haiku 4.5 target LLM, Sonnet 4.6 optimizer LLM).

SettingDocVQALiveMathMind2WebSpreadsheetBench
GEPA91.08 ± 0.4547.75 ± 4.6345.97 ± 0.9748.21 ± 3.39
SkillOpt91.39 ± 0.3350.83 ± 5.9044.48 ± 2.7247.32 ± 6.63
Iter-CoT71.24 ± 4.6942.06 ± 5.6548.73 ± 1.1049.20 ± 4.56
Rot.91.65 ± 0.2356.87 ± 10.9846.42 ± 0.2453.48 ± 1.55
Rand.91.40 ± 0.4247.51 ± 15.5046.88 ± 0.6654.64 ± 1.96
Best fixed search strategy91.21 ± 0.7352.61 ± 15.0646.99 ± 0.5153.30 ± 2.03
StraTune (from Appendix D.3)92.52 ± 0.8065.40 ± 1.1647.52 ± 0.6156.16 ± 0.90

For the best fixed search strategy, we take the strategy that scored highest among I1 only, I2 only, and I3 only in Table 2 on each dataset. This is I1 on DocVQA, I3 on LiveMath and SpreadsheetBench, and I2 on Mind2Web. Appendix D.2 also reports an earlier Rot. run on SpreadsheetBench (45.71), which is not among these four runs. With it, the five runs average 51.93.

The single runs in the main manuscript in fact make the ablations look stronger than they are. On LiveMath, Rot. drops from 65.40 in Table 2 to a mean of 56.87, and I3 with F3 from 63.51 in Table 12 (Appendix D.1) to 44.79. The best fixed search strategy also averages lower than in Table 2 on every dataset. Over four runs, StraTune has the highest mean of all settings on every dataset except Mind2Web. On Mind2Web, Iter-CoT has the highest mean, just as it has the highest single-run score in Table 1. Adaptive selection therefore has a higher mean than random, cyclic, and fixed choices of search strategy on every dataset, with the largest margins on LiveMath.

Q3. Selectors with matched information, total cost, and transfer.

Thank you for these questions. We address them in turn.

Selectors with matched information. We agree that the bandits in Table 2 receive less information than StraTune's optimizer LLM. To separate the two, we added three variants that change only the choice of search strategy (Table R3). Bandit, dense rewards the bandit rules in every round with the screening gain of each candidate skill, rather than only when a candidate skill is accepted. LinUCB also sees five numbers that describe the current run, such as how often the current skill fails on the batch and how much budget remains. LLM, reward table gives the optimizer LLM only the reward table that the bandits see.

Table R3. Test scores with different choices of search strategy (Haiku 4.5 target LLM, Sonnet 4.6 optimizer LLM). The three numbers in a cell are ε-greedy / UCB1 / Thompson. All entries are single runs.

SettingInformation used to choose the search strategyLiveMathSpreadsheetBench
Bandit (Table 2)reward when a candidate skill is accepted or saved55.45 / 38.39 / 45.0249.64 / 53.21 / 53.57
Bandit, densescreening gain of each candidate skill, every round43.13 / 53.08 / 60.6651.79 / 54.29 / 50.71
LinUCBscreening gain and five numbers describing the current run50.2450.71
LLM, reward tablethe bandits' reward table only46.4555.71
StraTune (Table 2)execution feedback and strategy and form histories65.8856.43

A denser reward or context does not close the gap. Even the best variant stays well below StraTune on LiveMath. With only the information the bandits see, the optimizer LLM also falls short. It scores below StraTune on both datasets, and far below on LiveMath. These results suggest that StraTune's advantage comes from information that only an LLM can use. It reads which samples fail and why, together with the recorded outcome of each search strategy and revision form.

Total cost. We agree that an equal number of target-LLM executions does not guarantee equal cost. Table R4 therefore reports the total training cost of every method, including all target-LLM and optimizer calls.

Table R4. Total training cost per run in USD (Haiku 4.5 target LLM, Sonnet 4.6 optimizer LLM).

MethodDocVQALiveMathMind2WebSpreadsheetBench
GEPA7.224.2129.723.2
TextGrad8.649.577.839.7
SkillOpt13.037.054.457.1
Trace2Skill27.444.8146.9121.8
Iter-CoT7.011.752.520.8
StraTune8.720.248.033.6

StraTune costs less than SkillOpt and Trace2Skill on every dataset. It is also the cheapest method on Mind2Web and the second cheapest on LiveMath.

Transfer across model families. We applied the skills learned with Haiku 4.5, without further optimization, to six target LLMs from other families: Llama 4 Maverick, Qwen3-VL 235B, Mistral Large 3, Gemma 3 27B, DeepSeek V3.2, and Kimi K2.5. Each skill is thus tested on 23 combinations of target LLM and dataset, namely the six target LLMs on four datasets, except DocVQA for the text-only DeepSeek V3.2 (Table R5).

Table R5. Transfer of the Table 1 skills to target LLMs from other families. Improvement is computed over the initial skill on the same target LLM.

SkillMean improvementImproved over the initial skillBest skill
GEPA+7.0316 of 231
TextGrad+3.3117 of 230
SkillOpt+8.2418 of 231
Trace2Skill+4.1916 of 230
Iter-CoT+12.5620 of 236
StraTune+17.5022 of 2316

StraTune's skills transfer best across model families. They improve on the initial skill in all but one combination and are the best skill far more often than any other method.

Transfer to unseen document types, websites, and domains. We also applied the skills with Haiku 4.5, again without further optimization, to three test sets that the paper does not use. The DocVQA skills were run on InfographicVQA, and the Mind2Web skills on the two other official Mind2Web test splits, with unseen websites and unseen domains. StraTune improves on the initial skill on all three test sets, by 20.22, 0.95, and 1.53 points. Among the baselines, only TextGrad also does so.

We thank the reviewer again. The revised paper discusses the LiveMath option in Section 4.2 and the transfer to other model families in Section 4.3, and adds these results to Tables 4 and 6 and Appendices D.2, D.5, and D.7. All revisions are marked in red.

Dear Reviewer Hoq3,

Thank you again for your review. We have responded to each of your points above and revised the paper accordingly, with all revisions marked in red. As the discussion period ends on November 18, we would appreciate it if you could let us know whether our response addresses your concerns, or whether any points need further clarification.

The authors

Reviewer_cWRGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StraTune optimizes reusable textual skills without updating model parameters. A frozen optimizer LLM selects among three search strategies and four revision forms using execution feedback and optimization history. Candidates undergo regression-aware screening and validation, followed by final skill selection. Experiments cover four benchmarks and two target–optimizer model settings, with substantial improvements in some settings, particularly LiveMath. However, unclear methodological presentation and insufficiently controlled comparisons weaken the evidence that state-dependent operator selection drives these gains.

Critical strengths

S1. Adaptive selection of skill-revision procedures addresses a practical limitation of fixed optimization strategies. Combining current execution feedback with previous revision outcomes is a reasonable design, and the paper examines strategy selection, revision forms, and candidate evaluation through several ablations.

S2. Evaluation spans four distinct tasks and two model settings. Some gains are substantial: on Haiku-based LiveMath, StraTune achieves 65.88 compared with 45.02 for the strongest baseline. Transfer experiments also show that skills learned with a smaller target model can benefit a stronger one.

Critical weaknesses

W1. Confusing presentation obscures both the algorithm and its ablations. Table 2 groups random and cyclic selectors under “without the per-round decision,” although both still select a strategy each round. Fixed-strategy rows also retain adaptive revision forms and therefore do not represent fixed operators. Figure 2 omits intermediate-skill update and reset rules, making the procedure difficult to reconstruct.

W2. Statistical evidence does not adequately support the claimed advantage of adaptive selection. Table 1 and most ablations report single runs, while Appendix D.3 provides variability only for StraTune. On LiveMath, fixed rotation achieves 65.40 versus StraTune’s 65.88, a gap smaller than StraTune’s reported standard deviation of 1.16. Matched multi-seed comparisons are needed.

W3. Bandit comparisons confound selection mechanisms with available information. StraTune reads textual failures and detailed histories, whereas bandits receive coarse rewards based on candidate acceptance or saving. With only zero to two accepted updates per run, these rewards are sparse. Contextual or dense-reward controls are needed to isolate the benefit of LLM-based selection.

W4. Replay does not cleanly isolate responsiveness to the optimization state. Its training-batch order differs from the original StraTune run, introducing an additional source of variation. Appendix D.2 also reports SpreadsheetBench rotation runs scoring 45.71 and 52.14, demonstrating substantial run sensitivity. Adaptive and replayed selectors should be evaluated under matched batch orders and multiple seeds.

W5. Matching target executions does not establish equal computational cost. Execution length depends on the generated skill, and optimizer calls are excluded from the budget. Table 8 reports optimizer tokens reaching 22.7% of target tokens on SpreadsheetBench, whereas the highlighted overhead covers only strategy selection. Total-token and monetary-cost comparisons across methods are necessary to assess efficiency.

W6. The operator taxonomy and incremental contribution require stronger justification. F1–F3 describe content types, whereas F4 describes editing scope: a full rewrite can introduce rules, examples, or procedures. These categories are not cleanly separated. Although the introduction acknowledges prior adaptive prompt-operator selection methods, direct comparisons are absent, leaving the advantage over existing adaptive approaches unclear.

Questions for authors

See weaknesses

6 comments

W1. Presentation of the ablations and of I2.

We thank the reviewer for these points and will revise the presentation accordingly.

Table 2 labels. By "without the per-round decision", we meant that the optimizer LLM does not choose the search strategy from the current optimization state. Rand. draws it at random; Rot. cycles through I1, I2, and I3; Once keeps its first choice; and Replay copies a recorded sequence. We agree that the label is ambiguous, since Rand. and Rot. still choose a search strategy each round. Following the reviewer's suggestion, we renamed this group "Strategy chosen without the optimization state".

Fixed-strategy rows. These rows are designed to test the per-round choice of search strategy, the core of StraTune, so they fix only the search strategy and keep everything else as in StraTune. The reviewer asks for runs that fix both the search strategy and the revision form. These runs are already in Appendix D.1 (Table 12 of the revised paper), which we placed in the appendix only because of limited space of main content. Table 12 covers all twelve combinations of I1–I3 and F1–F4 on LiveMath. The best combination, I3 with F3, scores 63.51, and the median is 37.20, against 65.88 for StraTune. The best combination also cannot be known in advance, since finding it requires running all twelve. StraTune needs no such prior choice. To make this visible, we labeled the rows "Fixed search strategy, forms adaptive", and Table 2 now refers to Appendix D.1.

Update and reset rules of I2. We added panel (d) to Figure 2 and state both rules in Section 3.1. An evaluated I2 candidate replaces the intermediate skill if it performs at least as well as the intermediate skill, even if it is not yet good enough to replace the current skill. When an I2 candidate is accepted, the next round proceeds from the new current skill as for any accepted candidate, and the refinement progress is reset.

W2. Matched multi-seed comparisons.

Thank you for this suggestion. We agree that single runs of the controls cannot settle the comparison. We therefore ran the three baselines that are strongest on at least one dataset in Table 1 (GEPA, SkillOpt, and Iter-CoT), together with Rot., Rand., and the best fixed search strategy, three more times each, with new seeds and the same budget and data splits. Together with the run in the main manuscript, each setting now has four runs (Table R1).

Table R1. Test scores over four runs, mean ± standard deviation (Haiku 4.5 target LLM, Sonnet 4.6 optimizer LLM).

SettingDocVQALiveMathMind2WebSpreadsheetBench
GEPA91.08 ± 0.4547.75 ± 4.6345.97 ± 0.9748.21 ± 3.39
SkillOpt91.39 ± 0.3350.83 ± 5.9044.48 ± 2.7247.32 ± 6.63
Iter-CoT71.24 ± 4.6942.06 ± 5.6548.73 ± 1.1049.20 ± 4.56
Rot.91.65 ± 0.2356.87 ± 10.9846.42 ± 0.2453.48 ± 1.55
Rand.91.40 ± 0.4247.51 ± 15.5046.88 ± 0.6654.64 ± 1.96
Best fixed search strategy91.21 ± 0.7352.61 ± 15.0646.99 ± 0.5153.30 ± 2.03
StraTune (from Appendix D.3)92.52 ± 0.8065.40 ± 1.1647.52 ± 0.6156.16 ± 0.90

For the best fixed search strategy, we take the strategy that scored highest among I1 only, I2 only, and I3 only in Table 2 on each dataset. This is I1 on DocVQA, I3 on LiveMath and SpreadsheetBench, and I2 on Mind2Web. Appendix D.2 also reports an earlier Rot. run on SpreadsheetBench (45.71), which is not among these four runs. With it, the five runs average 51.93.

The single runs in the main manuscript in fact make the ablations look stronger than they are. The close Rot. run on LiveMath (65.40) is the best of its four runs, which average 56.87. Rand. and I3 only drop in the same way, and the best fixed search strategy averages lower than in Table 2 on every dataset. Over four runs, StraTune has the highest mean of all settings on every dataset except Mind2Web. On Mind2Web, Iter-CoT has the highest mean, just as it has the highest single-run score in Table 1.

W3. Bandit selectors with denser rewards, context, and the same information.

Thank you for this suggestion. The Bandit rows in Table 2 are StraTune with one change. A bandit rule replaces the optimizer LLM in the choice of search strategy, following the standard setup of bandit-based operator selection, as in OPTS (Ashizawa et al., 2025). A search strategy is rewarded when its candidate skill is accepted or saved. In StraTune, the optimizer LLM makes this choice from the execution feedback and the strategy and form histories. To isolate the role of this information, we added three variants that also change only the choice of search strategy.

  1. Bandit, dense. The bandit rules in Table 2 are rewarded only when a candidate skill is accepted or saved. A run accepts zero to two candidate skills, so this reward is almost always zero. We instead reward them in every round with the screening gain, the score of the candidate skill minus that of the current skill on the screening set (Fialho et al., 2010). Rejected candidate skills thus also show how well each search strategy did.
  2. LinUCB. A plain bandit learns one ranking of the search strategies for the whole run. This contextual bandit uses the screening gain and also sees five numbers that describe the current run, for example, how often the current skill fails on the batch and how much budget remains. It can therefore choose a different search strategy in a different situation.
  3. LLM, reward table. The optimizer LLM still chooses the search strategy, but it sees only the reward table that the bandits use, without the execution feedback and the strategy and form histories.

Table R2. Test scores with different choices of search strategy (Haiku 4.5 target LLM, Sonnet 4.6 optimizer). The three numbers in a cell are ε-greedy / UCB1 / Thompson. All entries are single runs.

SettingInformation used to choose the search strategyLiveMathSpreadsheetBench
Bandit (Table 2)reward when a candidate skill is accepted or saved55.45 / 38.39 / 45.0249.64 / 53.21 / 53.57
Bandit, densescreening gain of each candidate skill, every round43.13 / 53.08 / 60.6651.79 / 54.29 / 50.71
LinUCBscreening gain and five numbers describing the current run50.2450.71
LLM, reward tablethe bandits' reward table only46.4555.71
No hist. (Table 2)execution feedback, no strategy and form histories34.1252.14
StraTune (Table 2)execution feedback and strategy and form histories65.8856.43

A denser reward or context does not close the gap. Even the best of these variants stays well below StraTune on LiveMath. With only the information the bandits see, the optimizer LLM also falls short. Given just the bandits' reward table, it scores below StraTune on both datasets, and far below on LiveMath. These results suggest that StraTune's advantage comes from information that only an LLM can use. StraTune reads which samples fail and why, together with the recorded outcome of each search strategy and revision form, and no reward or fixed set of features can express this. Removing only the histories (No hist.) also lowers the score, so StraTune needs both the feedback and the histories.

W4. Replay under matched batches and seeds.

Thank you for raising this. The training batches are fixed by the dataset, and round r uses the same batch in every run. Replay therefore sees the same batches in the same order as the StraTune run it copies. It differs only in the sampling seed, which changes the sampled screening and validation subsets. The phrase "a different order of training batches" in Section 4.1 was an error, which we have corrected in the revision.

Replay underperforms StraTune across runs. Its LiveMath run in Table 2 is lower than StraTune's, and a second Replay run on DocVQA and SpreadsheetBench, like the first, is lower than StraTune's four-run mean (Appendix D.3) on both datasets.

Run sensitivity of Rot.. We agree that Rot. is sensitive to the run. On SpreadsheetBench, its five runs range from 45.71 to 55.00, and all five are below StraTune's four-run mean (Appendix D.3). This sensitivity belongs to the fixed rotation, not to StraTune. StraTune chooses the search strategy from the optimization state and varies much less, with a standard deviation of 0.90.

W5. Total training cost and comparison at equal cost.

Thank you for this comment. We first clarify two points. The budget is the same for every method. The 6|D_train| budget counts target-LLM executions, in the same way as the rollout budget of GEPA (Agrawal et al., 2026). The paper reports the full optimizer cost, not only strategy selection. Appendix C.4 counts every call of both LLMs, and the 22.7% on SpreadsheetBench is the ratio of all of StraTune's optimizer tokens to its target tokens. Section 4.2 highlights strategy selection because it is the only step StraTune adds to a method with a fixed revision operator. Every method already spends optimizer calls on generating candidates.

To compare total cost across methods, we report the training cost of every method, including all target-LLM and optimizer calls (Table R3).

Table R3. Total training cost per run in USD (Haiku 4.5 target LLM, Sonnet 4.6 optimizer).

MethodDocVQALiveMathMind2WebSpreadsheetBench
GEPA7.224.2129.723.2
TextGrad8.649.577.839.7
SkillOpt13.037.054.457.1
Trace2Skill27.444.8146.9121.8
Iter-CoT7.011.752.520.8
StraTune8.720.248.033.6

StraTune costs less than SkillOpt and Trace2Skill on every dataset. It is also the cheapest method on Mind2Web and the second cheapest on LiveMath. The optimizer accounts for a small share of StraTune's cost, 2–26% with every optimizer call included, against 25–56% for SkillOpt and 37–81% for Trace2Skill. StraTune's final skills are also short, so they add little to the length of each execution.

To compare the methods at equal cost, we then limited every method to StraTune's training cost on each dataset (Table R4). Methods that cost less than StraTune keep their final skill. Methods that cost more are stopped once they reach StraTune's cost.

Table R4. Test scores when every method is limited to StraTune's training cost on each dataset (best in bold).

DocVQALiveMathMind2WebSpreadsheetBench
GEPA91.2643.6046.1852.50
TextGrad90.6931.7545.9945.00
SkillOpt91.8736.9740.5550.71
Trace2Skill†90.6233.1846.1846.79
Iter-CoT73.4236.9747.99†48.57
StraTune92.1265.8847.4956.43
Cost limit (USD)8.720.248.033.6

† Final skill. Trace2Skill and Iter-CoT produce their skill in a single pass at the end of a run, so they cannot be stopped earlier. Trace2Skill on every dataset and Iter-CoT on Mind2Web cost more than StraTune.

At equal cost, StraTune keeps its rank from Table 1 on every dataset. It stays first on DocVQA, LiveMath, and SpreadsheetBench and second on Mind2Web.

W6. The revision forms, and comparison with adaptive operator selection.

Thank you for these comments.

A full rewrite overlaps with the other forms in content, but not in what it changes. F1–F3 each add one kind of content and keep the rest of the skill. F4 rewrites the whole skill and can reorganize existing instructions (Section 3.1). Each form also follows existing methods. SkillOpt and ExpeL add conditional rules (F1), Iter-CoT adds worked examples (F2), AWM writes workflows (F3), and TextGrad, GEPA, and OPRO rewrite the skill in full (F4).

A full rewrite cannot replace the other forms. Allowing only full rewrites (F4 only) lowers the score on all four datasets (Table 2). With a fixed search strategy, F4 also underperforms the other forms on LiveMath (Appendix D.1). It scores 58.29 against 63.51 for F3 under I3, and 34.12 against 46.92 for F1 under I1.

Comparison with adaptive operator selection. The introduction of our manuscript cites OPTS (Ashizawa et al., 2025) and SEE (Cui et al., 2025), which choose among editing operators within one fixed search procedure. We now compare with both by replacing StraTune's choice of revision operator with their selection rules. The revision operators, candidate evaluation, final skill selection, and budget stay unchanged (Table R5). Because the first SEE run on LiveMath came close to StraTune, we repeated the LiveMath runs and report their mean and standard deviation.

Table R5. StraTune against the selection rules of OPTS and SEE (Haiku 4.5 target LLM, Sonnet 4.6 optimizer). LiveMath entries are mean ± standard deviation over two (OPTS) and three (SEE) runs; SpreadsheetBench entries are single runs.

SelectorLiveMathSpreadsheetBench
OPTS rule43.13 ± 9.3849.64
SEE rule52.92 ± 16.5753.93
StraTune65.40 ± 1.1656.43

Both rules stay below StraTune and vary far more between runs, with standard deviations of 9.38 and 16.57 on LiveMath against 1.16 for StraTune.

Thank you! The revised paper adds these results to Section 4.2and Appendices C and D, with all revisions marked in red.

Reviewer_S2aoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies self-evolving skills: a frozen target LLM performs tasks under a reusable text skill, and a frozen optimizer LLM revises the skill from execution feedback. It organizes prior methods by search strategy (I1 direct revision, I2 iterative refinement of a separate working copy, I3 parallel sampling with ranking) and revision form (F1 rules, F2 worked examples, F3 procedures, F4 full rewrite), and argues that each existing method commits to one "revision operator" (strategy plus forms) and that no single operator is best across tasks. StraTune lets the optimizer LLM choose the strategy and forms at every round from an optimization state containing the current execution feedback and a history of earlier candidates' outcomes, and passes all candidates through a shared two-stage acceptance procedure (screening on a small set, then validation on a larger disjoint training subset), followed by a final selection among saved candidates. With Haiku 4.5 (target) / Sonnet 4.6 (optimizer) and Opus 4.8 / Opus 5 on DocVQA, LiveMathematicianBench, Mind2Web, and SpreadsheetBench, under a matched budget of 6|D_train| target executions, StraTune is best or tied in seven of eight settings (Table 1), with the largest gain on LiveMath with Haiku; ablations compare fixed, random, rotating, replayed, bandit, and history-free strategy choices (Table 2), evaluation components (Table 3), and baselines' generators under StraTune's evaluation (Table 4). Skills learned with Haiku also transfer to Opus.

Critical strengths
  1. A clear design space and a careful comparison protocol. Organizing skill-learning methods by search strategy and revision form (Figure 1) is a useful lens, and comparing five methods under a shared target, optimizer, initial skill, test set, and a budget defined in target-LLM executions (Section 2, Section 4.1) is more controlled than most prior comparisons. The data splits are clean (e.g., a temporal split for LiveMath; App. C.1).

  2. Thorough ablations of the selection mechanism. The paper compares fixed, random, rotating, replayed, bandit-based, and history-free strategy choices (Table 2), isolates the candidate evaluation from candidate generation by running GEPA and SkillOpt generators under StraTune's evaluation (Table 4), analyzes the per-round choices (Figure 4), and shows that skills learned with a cheaper target transfer to a stronger one (Figure 5). Code and learned skills are provided.

Critical weaknesses

1. It is unclear why a fixed menu of search strategies is needed with a strong optimizer. The revision forms (rules, examples, procedures, full rewrite) are kinds of text edits that a capable optimizer LLM can already choose among implicitly. The search strategies differ mainly in harness control flow — keeping a separate working draft (I2), or generating several candidates and ranking them with target executions (I3) — which are hand-designed procedures. The paper does not test the natural alternative: an optimizer given the same capabilities as general tools (keep drafts, propose several candidates, run them on a few training samples, decide when to submit) and left to decide how to use them, without a predefined menu. The existing ablations (F4 only, No hist., bandits) and baselines all retain a fixed operator structure, so the evidence does not show that the explicit menu is what matters, as opposed to giving the optimizer more freedom and information.

2. The benefit of per-round adaptivity is weakly supported. Most of the gain comes from avoiding a poor fixed strategy rather than from switching within a run. Against the best fixed strategy for each benchmark (Table 2), StraTune improves by +0.20 (DocVQA, I1), +2.85 (LiveMath, I3), +0.15 (Mind2Web, I2), and +1.43 (SpreadsheetBench, I3), and the DocVQA and Mind2Web margins are smaller than StraTune's own seed standard deviations on those benchmarks (0.80 and 0.61; Table 11). Random strategy choice is close on three benchmarks (91.10, 63.98, 46.58). Only one or two candidates are accepted per run (Section 4.3), so the many per-round choices mostly produce rejected candidates, which limits how much within-run adaptivity can contribute. A comparison with splitting the budget across the three fixed strategies and selecting the best on validation would clarify what per-round selection adds.

3. Limited statistical support for the main comparison. Baselines and all ablations are single runs; variance is reported only for StraTune (Table 11, ±0.40–1.16). Outside LiveMath with Haiku, the margins over the best baseline are mostly 0.25–1.4 points (Table 1), comparable to that variance, and the authors note that LiveMath scores vary between runs of every configuration (App. D). Ablations are run only in the Haiku setting.

Questions for authors

Questions that could change my assessment:

  1. How does StraTune compare with an optimizer LLM that is given the same capabilities (maintaining a working draft, proposing several candidates, evaluating them on a few training samples) as general tools, without a predefined menu of strategies? What does the explicit strategy menu add beyond this?
  2. How does StraTune compare with splitting the same budget across the three fixed strategies and selecting the best resulting skill on validation? Can you quantify how much of the gain comes from switching strategies within a run, given that only one or two candidates are accepted per run?
  3. Can you report multiple runs (mean ± std) for the baselines and the key ablations, so the margins in Tables 1 and 2 can be compared with run-to-run variance?

Clarifications: 4. Compute: the budget counts target-LLM executions and excludes optimizer calls. Can you report total optimizer and target tokens (or API cost) for every method, so the comparison can also be read at matched total cost? 5. Do the re-implemented baselines reproduce their published results in comparable settings (e.g., SkillOpt on SpreadsheetBench-Verified)?

8 comments

W1 / Q1. Are explicit revision operators needed when the optimizer is strong?

Thank you for this question. To test whether StraTune needs its explicit revision operators, the "menu" in the review, we implemented the ablation setting the reviewer proposes (General tools). The optimizer LLM receives no revision operators but four tools: (1) keep drafts across rounds, (2) propose any skill, (3) try a skill on up to 24 training samples, and (4) submit a skill to StraTune's candidate evaluation. Table R1 compares it with StraTune and with the ablations F4 only and No hist. from the paper. All three settings score below StraTune.

Table R1. StraTune against General tools and two ablations from the paper (Haiku 4.5 target LLM, Sonnet 4.6 optimizer, same budget and candidate evaluation). General tools is the mean ± standard deviation over three runs, and StraTune over four runs (Appendix D.3); the other rows are single runs.

SettingHow it worksLiveMathSpreadsheetBench
F4 onlyevery candidate skill is a full rewrite (F4) whose content the optimizer LLM chooses freely42.1851.79
General toolsthe four tools instead of the revision operators; a new skill must be submitted every round45.50 ± 3.4251.67 ± 1.44
No hist.revision operators kept, but the strategy and form histories are withheld34.1252.14
StraTuneexplicit revision operators, with the strategy and form histories65.40 ± 1.1656.16 ± 0.90

Without explicit revision operators, the optimizer LLM finds fewer effective changes. With general tools, only one candidate skill is accepted per run, and almost all changes are conditional rules or reasoning procedures. The optimizer LLM never makes a worked example its main change and never writes a full rewrite on LiveMath, yet StraTune's final skills on SpreadsheetBench and LiveMath rely on exactly these forms. Each revision operator works like an LLM harness. We agree with the reviewer that the search strategies are procedures rather than text edits, and this is exactly their role. Like the harness around a coding agent, a search strategy fixes how the calls of the optimizer LLM are combined. I3 generates several candidate skills and ranks them by running the target LLM. I2 keeps an intermediate skill and refines it over several rounds. General tools gives the optimizer LLM the same raw capabilities but leaves it to build such procedures itself, and the results above show that it does not. Even for a strong optimizer LLM, a well-designed harness therefore makes a difference, as it does for tool-calling agents.

W2 / Q2. Budget split and switching within a run.

Thank you for this suggestion. The reviewer proposes splitting the budget across the three fixed search strategies and keeping the best resulting skill. We ran this split (Budget split). Each strategy receives a third of the budget (2|D_train| target-LLM executions). StraTune needs no held-out validation set, so we did not hold one out for Budget split either. Instead, we pick the best of the three skills by test score, which favors Budget split, since StraTune never sees test scores. Table R2 compares StraTune with Budget split and with the ablations Once and Replay from the paper, which measure the contribution of switching within a run.

Table R2. StraTune against Budget split, Once, and Replay (Haiku 4.5 target LLM, Sonnet 4.6 optimizer).

SettingHow it worksDocVQALiveMathMind2WebSpreadsheetBench
Budget splitI1, I2, and I3 with 2|D_train| each; the best skill by test score91.9240.7646.0940.00*
Oncethe optimizer LLM chooses a search strategy once and keeps it91.5363.0346.1455.36
Replayfollows the strategy sequence recorded in a StraTune run instead of the optimization state92.0051.6646.1853.21
StraTunesearch strategy and revision forms chosen each round from the optimization state92.1265.8847.4956.43

* On SpreadsheetBench, none of the three search strategies has changed the initial skill after 2|D_train| target-LLM executions in our runs.

Budget split stays below StraTune on all four datasets, even with the best skill picked by test score. StraTune scores 0.59 to 2.85 points above Once, which never switches. Once lets the optimizer LLM choose a search strategy at the start and keep it, so this difference reflects the gain from switching. The switches must follow the optimization state. Replay switches as often as StraTune, but at the rounds recorded in an earlier run, and it scores lower on every dataset. Switching helps even though few candidate skills are accepted, because each rejected candidate skill is recorded in the strategy and form histories and guides the next choice.

W3 / Q3. Multiple runs.

Thank you for this request. We agree that single runs cannot show whether the margins exceed the run-to-run variance. We therefore ran the three baselines that are strongest on at least one dataset in Table 1 (GEPA, SkillOpt, and Iter-CoT) and the key ablations of Table 2 (Rot., Rand., and the best fixed search strategy) three more times each, with new seeds and the same budget and data splits. Together with the run in the main manuscript, each setting now has four runs (Table R3).

Table R3. Test scores over four runs, mean ± standard deviation (Haiku 4.5 target LLM, Sonnet 4.6 optimizer LLM).

SettingDocVQALiveMathMind2WebSpreadsheetBench
GEPA91.08 ± 0.4547.75 ± 4.6345.97 ± 0.9748.21 ± 3.39
SkillOpt91.39 ± 0.3350.83 ± 5.9044.48 ± 2.7247.32 ± 6.63
Iter-CoT71.24 ± 4.6942.06 ± 5.6548.73 ± 1.1049.20 ± 4.56
Rot.91.65 ± 0.2356.87 ± 10.9846.42 ± 0.2453.48 ± 1.55
Rand.91.40 ± 0.4247.51 ± 15.5046.88 ± 0.6654.64 ± 1.96
Best fixed search strategy91.21 ± 0.7352.61 ± 15.0646.99 ± 0.5153.30 ± 2.03
StraTune (from Appendix D.3)92.52 ± 0.8065.40 ± 1.1647.52 ± 0.6156.16 ± 0.90

For the best fixed search strategy, we take the strategy that scored highest among I1 only, I2 only, and I3 only in Table 2 on each dataset. This is I1 on DocVQA, I3 on LiveMath and SpreadsheetBench, and I2 on Mind2Web. Appendix D.2 also reports an earlier Rot. run on SpreadsheetBench (45.71), which is not among these four runs. With it, the five runs average 51.93.

The single runs in the main manuscript in fact make the ablations look stronger than they are. On LiveMath, Rand. drops from 63.98 in Table 2 to a mean of 47.51, and Rot. and I3 only drop in the same way. The best fixed search strategy also averages lower than in Table 2 on every dataset, so StraTune's lead over it is larger than in Table 2 on all four datasets. Over four runs, StraTune has the highest mean of all settings on every dataset except Mind2Web. On Mind2Web, Iter-CoT has the highest mean, just as it has the highest single-run score in Table 1.

All repeated runs use the Haiku setting. This is where the ablations of Table 2 were run and where the closest margins arise, so these runs cover the comparisons in question across all four datasets. Repeating every setting with Opus 4.8 and Opus 5 would cost roughly five times as much, over USD 10,000, so we could not complete it within the rebuttal period.

Q4. Total cost.

Thank you for this suggestion. We agree that an equal number of target-LLM executions does not guarantee equal cost. We therefore report the total training cost of every method, including all target-LLM and optimizer calls (Table R4).

Table R4. Total training cost per run in USD (Haiku 4.5 target LLM, Sonnet 4.6 optimizer).

MethodDocVQALiveMathMind2WebSpreadsheetBench
GEPA7.224.2129.723.2
TextGrad8.649.577.839.7
SkillOpt13.037.054.457.1
Trace2Skill27.444.8146.9121.8
Iter-CoT7.011.752.520.8
StraTune8.720.248.033.6

StraTune costs less than SkillOpt and Trace2Skill on every dataset. It is also the cheapest method on Mind2Web and the second cheapest on LiveMath.

To compare the methods at equal cost, we then limited every method to StraTune's training cost on each dataset (Table R5). Methods that cost less than StraTune keep their final skill. Methods that cost more are stopped once they reach StraTune's cost.

Table R5. Test scores when every method is limited to StraTune's training cost on each dataset (best in bold).

DocVQALiveMathMind2WebSpreadsheetBench
GEPA91.2643.6046.1852.50
TextGrad90.6931.7545.9945.00
SkillOpt91.8736.9740.5550.71
Trace2Skill†90.6233.1846.1846.79
Iter-CoT73.4236.9747.99†48.57
StraTune92.1265.8847.4956.43
Cost limit (USD)8.720.248.033.6

† Final skill. Trace2Skill and Iter-CoT produce their skill in a single pass at the end of a run, so they cannot be stopped earlier. Trace2Skill on every dataset and Iter-CoT on Mind2Web cost more than StraTune.

At equal cost, StraTune keeps its rank from Table 1 on every dataset. It stays first on DocVQA, LiveMath, and SpreadsheetBench and second on Mind2Web.

Q5. Do the baselines reproduce their published results?

Thank you for this question. GEPA, SkillOpt, Trace2Skill, and Iter-CoT run from their released code with their published defaults, and TextGrad follows its published update procedure (Appendix C.2). On SpreadsheetBench, our baselines improve over the initial skill as much as or more than in the SkillOpt paper. Our 280 test tasks are the SpreadsheetBench-Verified test split released with that paper. Table R6 compares how much each baseline improves over the initial skill. The target LLMs differ (GPT-5.4-mini in the SkillOpt paper, Haiku 4.5 here). The initial skills start at a similar level.

Table R6. Improvement over the initial skill on SpreadsheetBench-Verified, as published in the SkillOpt paper and in our runs.

SkillOpt paper (GPT-5.4-mini)Ours (Haiku 4.5)
Initial skill36.140.00
SkillOpt+11.4+10.71
GEPA+6.4+12.50
TextGrad+2.1+5.00
Trace2Skill+4.6+6.79

GEPA, TextGrad, and Trace2Skill improve more over the initial skill in our runs than in the SkillOpt paper, and SkillOpt improves about as much. On DocVQA and LiveMath, our splits are stricter than the random split of the SkillOpt paper, so the numbers there are not directly comparable. In our splits, DocVQA shares no document between training and test, and LiveMath is split by time. The baselines are therefore faithful and competitive.

We thank the reviewer again. The revised paper adds these results to Section 4.2 (Tables 4 and 6) and Appendices C.4 and D.4, and Section 4.1 states that the repeated runs use the Haiku setting. All revisions are marked in red.

Reviewer_oR9kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the optimization of reusable textual skills with frozen language models. StraTune prompts an optimizer LLM to select a search strategy and revision form at each round using execution feedback and recorded outcomes. Proposed skills then undergo a shared screening, validation, and final-selection procedure. The evaluation compares five baselines across four tasks and two model configurations, with additional fixed-operator, random, cyclic, bandit, and transfer experiments. StraTune achieves the best or near-best performance in most settings, with its largest improvement on LiveMath using the Haiku target model.

Critical strengths
  1. The problem is practically relevant and evaluated across diverse tasks. Different tasks plausibly benefit from different revision operators, making feedback-dependent operator selection a reasonable approach. The comparison includes five existing methods and two model configurations rather than relying on a single weak baseline.

  2. The ablations go beyond reporting final performance. Fixed complete operators, cyclic schedules, bandit selectors, and baseline proposal generators integrated with StraTune’s evaluation procedure help examine where improvements originate. The appendix also reports variability across four StraTune runs.

Critical weaknesses
  1. The evidence does not yet establish that per-round adaptive selection is the main source of improvement. On LiveMath, the cyclic strategy obtains 65.40 compared with StraTune’s 65.88, and the best fixed complete operator in Appendix D.1 reaches 63.51. StraTune’s reported standard deviation on this task is 1.16, whereas these key controls are primarily single runs. Several differences on DocVQA and Mind2Web are also small.

  2. The budget comparison and the scope of the evidence remain limited. Matching target-model task executions excludes optimizer calls and does not ensure comparable costs for different skill lengths and proposal procedures. Table 8 reports tokens only for StraTune, so the low overhead of its selection call does not establish the cost-effectiveness of the complete method relative to baselines.

Questions for authors
  1. Can the authors report paired repeated runs and confidence intervals for StraTune, cyclic selection, random selection, and the strongest fixed complete operator? These comparisons are more directly relevant to the central claim than StraTune’s variance alone.

  2. What are the total target-model and optimizer input/output tokens for each method? Do the main rankings persist when budgets are matched by total tokens or actual cost?

  3. Beyond the DocVQA formatting rule, can the authors provide representative final skills and accepted revisions showing which substantive changes account for the gains?

6 comments

W1 / Q1. Is per-round selection the source of the gain?

Thank you for this important question. We agree that single runs of the controls cannot settle it, especially on LiveMath, where scores vary between runs. Following the reviewer's request, we ran Rot., Rand., the best fixed search strategy, and the best fixed revision operator (I3 with F3) three more times each, with new seeds and the same budget and data splits. Together with the run reported in the main manuscript, each setting now has 3-4 runs (Table R1).

Table R1. Test scores over four runs, mean ± standard deviation (Haiku 4.5 target LLM, Sonnet 4.6 optimizer LLM).

SettingDocVQALiveMathMind2WebSpreadsheetBench
Rot.91.65 ± 0.2356.87 ± 10.9846.42 ± 0.2453.48 ± 1.55
Rand.91.40 ± 0.4247.51 ± 15.5046.88 ± 0.6654.64 ± 1.96
Best fixed search strategy91.21 ± 0.7352.61 ± 15.0646.99 ± 0.5153.30 ± 2.03
StraTune (from Appendix D.3)92.52 ± 0.8065.40 ± 1.1647.52 ± 0.6156.16 ± 0.90

For the best fixed search strategy, we take the strategy that scored highest among I1 only, I2 only, and I3 only in Table 2 on each dataset. This is I1 on DocVQA, I3 on LiveMath and SpreadsheetBench, and I2 on Mind2Web. Appendix D.2 also reports an earlier Rot. run on SpreadsheetBench (45.71), which is not among these four runs. With it, the five runs average 51.93.

The single runs in the main manuscript in fact make the controls look stronger than they are. The close runs on LiveMath are the best of their four runs. Rot. drops from 65.40 in Table 2 to a mean of 56.87, and I3 with F3 from 63.51 in Table 12 (Appendix D.1) to 44.79. The best fixed search strategy also averages lower than in Table 2 on every dataset. Over four runs, StraTune has a higher mean than every control on every dataset, with the largest margins on LiveMath.

W2 / Q2. Total cost and rankings at matched cost.

Thank you for this suggestion. We agree that an equal number of target-LLM executions does not guarantee equal cost. We therefore report the total training cost of every method, including all target-LLM and optimizer calls (Table R2).

Table R2. Total training cost per run in USD (Haiku 4.5 target LLM, Sonnet 4.6 optimizer LLM).

MethodDocVQALiveMathMind2WebSpreadsheetBench
GEPA7.224.2129.723.2
TextGrad8.649.577.839.7
SkillOpt13.037.054.457.1
Trace2Skill27.444.8146.9121.8
Iter-CoT7.011.752.520.8
StraTune8.720.248.033.6

StraTune costs less than SkillOpt and Trace2Skill on every dataset. It is also the cheapest method on Mind2Web and the second cheapest on LiveMath.

To compare the methods at equal cost, we then limited every method to StraTune's training cost on each dataset (Table R3). Methods that cost less than StraTune keep their final skill. Methods that cost more are stopped once they reach StraTune's cost.

Table R3. Test scores when every method is limited to StraTune's training cost on each dataset (best in bold).

DocVQALiveMathMind2WebSpreadsheetBench
GEPA91.2643.6046.1852.50
TextGrad90.6931.7545.9945.00
SkillOpt91.8736.9740.5550.71
Trace2Skill†90.6233.1846.1846.79
Iter-CoT73.4236.9747.99†48.57
StraTune92.1265.8847.4956.43
Cost limit (USD)8.720.248.033.6

† Final skill. Trace2Skill and Iter-CoT produce their skill in a single pass at the end of a run, so they cannot be stopped earlier. Trace2Skill on every dataset and Iter-CoT on Mind2Web cost more than StraTune.

At equal cost, StraTune keeps its rank from Table 1 on every dataset. It stays first on DocVQA, LiveMath, and SpreadsheetBench and second on Mind2Web.

Q3. Which changes account for the gains?

Thank you for this question. We conducted additional analysis which helped us understand the gains better. On LiveMath, SpreadsheetBench, and Mind2Web, most of StraTune's gain comes from one part of its final skill. Interestingly, a different revision operator learned this part on each dataset.

  • LiveMath. StraTune's final skill tells the target LLM to favor the option "…but a stronger result can be proven". It was learned as a reasoning procedure (F3) and later kept in a full rewrite (F4). The skills of GEPA, TextGrad, SkillOpt, and Trace2Skill mention this option only conditionally. With StraTune's skill, the target LLM picks this option far more often where it is correct (79%, against 44% and 49% with the skills of GEPA and SkillOpt).
  • SpreadsheetBench. StraTune's final skill tells the target LLM to write computed values rather than formula strings, which always score 0, and it includes a worked example. Parallel sampling (I3) learned this part. The skills of GEPA, SkillOpt, and Trace2Skill still allow formula strings when the instruction asks for a formula, and TextGrad's skill does not address them. As a result, the target LLM writes far fewer formula strings with StraTune's skill (5 of 280 tasks, against 10 and 40 with the skills of GEPA and SkillOpt).
  • Mind2Web. StraTune's final skill tells the target LLM to check its progress on the sub-goals before each action. Iterative refinement (I2) learned this part. The skills of GEPA, SkillOpt, and Trace2Skill contain no such check.

To test whether these parts really drive the gains, we removed each one from StraTune's final skill and added it to the final skills of the baselines, on the two datasets with the largest gains (Table R4).

Table R4. Test score without → with this part of StraTune's skill (Haiku 4.5 target LLM). Original final skills as in Table 1; edited skills evaluated separately.

SkillLiveMathSpreadsheetBench
StraTune36.49 → 65.8845.36 → 56.43
GEPA43.60 → 59.9552.50 → 56.25
SkillOpt45.02 → 47.6350.71 → 45.00

This part explains most of StraTune's lead. Without it, StraTune's skill loses its lead, and adding it to GEPA's skill raises GEPA's score on both datasets. For SkillOpt the effect is mixed, because its long skills contain sections that conflict with the added part.

We thank the reviewer again for the constructive comments, which have strengthened the paper. The revised paper adds these results to Section 4.2 (Tables 4 and 6) and Appendices C.4, D.1, and D.6, with all revisions marked in red.

Dear Reviewer oR9k,

Thank you again for your review. We have responded to each of your points above and revised the paper accordingly, with all revisions marked in red. As the discussion period ends on November 18, we would appreciate it if you could let us know whether our response addresses your concerns, or whether any points need further clarification.

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of new results and revisions

We thank all four reviewers for their careful reading and constructive comments. We ran the requested experiments and revised the paper. All revisions are marked in red. The main changes are as follows.

1. Repeated runs (oR9k, S2ao, cWRG, Hoq3). We ran the strongest baselines (GEPA, SkillOpt, and Iter-CoT) and the key ablations (Rot., Rand., and the best fixed search strategy) three more times with new seeds. Over four runs, StraTune has a higher mean than every ablation on every dataset, and the highest mean of all methods on every dataset except Mind2Web (Section 4.2, Table 4; Appendix D.3).

2. Total training cost (oR9k, S2ao, cWRG, Hoq3). We report the total cost of every method, including all target-LLM and optimizer calls (Section 4.2, Table 6). StraTune costs less than SkillOpt and Trace2Skill on every dataset. When every method is limited to StraTune's cost, StraTune keeps its rank on every dataset (Appendix C.4).

3. Replacing adaptive selection or the revision operators (oR9k, S2ao, cWRG, Hoq3). Each of the following underperforms StraTune:

  • fixed revision operators, repeated for the best one (Appendix D.1);
  • bandit selectors with denser rewards, context, or the same information given to the optimizer LLM, and the selection rules of OPTS and SEE (Appendix D.2);
  • an optimizer LLM with general tools instead of revision operators, and a budget split across the three search strategies (Appendix D.4).

4. Where the gains come from (oR9k, Hoq3).

  • On LiveMath, StraTune also has the highest accuracy on the questions without the recurring "stronger result" option (Section 4.2; Appendix D.5).
  • On LiveMath and SpreadsheetBench, one part of StraTune's final skill accounts for most of its lead (Appendix D.6).
  • StraTune's skills transfer best to six target LLMs from other model families, and they improve on the initial skill on all three shifted test sets (Section 4.3; Appendix D.7).

5. Clarifications (cWRG).

  • We renamed the ablation groups in Table 2.
  • Figure 2(d) and Section 3.1 now state the update and reset rules of I2.
  • We corrected the description of Replay in Section 4.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.