- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents StraTune, a method for optimizing textual skills while keeping model parameters frozen. An optimizer LLM dynamically selects search strategies and revision forms based on execution feedback and revision histories, while a two-stage evaluation procedure filters candidate skills. Experiments cover four benchmarks, two model configurations, and transfer of learned skills to a stronger target model.
Critical strengths
- Clear motivation. Different tasks and optimization stages may benefit from different revision operators. Making search strategies and revision forms part of the per-round decision is a reasonable and practical design.
- Broad component analysis. The experiments examine fixed operators, random and cyclic schedules, histories, revision forms, and candidate evaluation. Table 4 further attempts to separate the contributions of candidate generation and evaluation.
- Practically useful results. StraTune achieves the best or joint-best performance in most evaluated settings, and skills optimized with a smaller target model can be reused directly with a stronger model.
Critical weaknesses
- The substantial Math gains are difficult to interpret as improvements in mathematical reasoning. The released skills explicitly exploit an option stating that another option is correct but a stronger result can be proven; the Opus skill instructs the model to select this option whenever it appears. The provided samples also show a strong association between this wording and the correct answer. Learning this pattern from training data does not imply test leakage, but without subgroup results and a simple heuristic baseline, the source of the largest reported gains remains unclear.
- The advantage of adaptive selection over simple schedules lacks sufficient statistical support. On Math, cyclic selection achieves 65.40, close to StraTune’s 65.88, while the best fixed operator achieves 63.51. Most baselines and ablations are evaluated once, whereas four-run statistics are provided only for StraTune, leaving the robustness of smaller differences uncertain.
- Some experimental conclusions require narrower interpretation. The bandit selectors primarily receive numerical rewards, whereas the LLM selector also receives semantic failure information, confounding the selection mechanism with its inputs. Moreover, matching target-model executions does not establish equal total cost, and transfer is evaluated only within one model family on unchanged tasks.
Questions for authors
- Could the authors report Math accuracy separately for questions with and without the “stronger result” option, for the initial skill, main baselines, and StraTune? How much of the gain is explained by a baseline that selects this option whenever present and otherwise uses the initial skill?
- Could the authors provide repeated runs for the main baselines, random selection, cyclic selection, and the best fixed operator to establish whether adaptive selection provides a consistent advantage?
- Could different selectors be compared under matched feedback information, with complete token or monetary costs reported across methods? Does skill transfer also hold across model families or under changes in the task distribution?
6 comments
Q3. Selectors with matched information, total cost, and transfer.
Thank you for these questions. We address them in turn.
Selectors with matched information. We agree that the bandits in Table 2 receive less information than StraTune's optimizer LLM. To separate the two, we added three variants that change only the choice of search strategy (Table R3). Bandit, dense rewards the bandit rules in every round with the screening gain of each candidate skill, rather than only when a candidate skill is accepted. LinUCB also sees five numbers that describe the current run, such as how often the current skill fails on the batch and how much budget remains. LLM, reward table gives the optimizer LLM only the reward table that the bandits see.
Table R3. Test scores with different choices of search strategy (Haiku 4.5 target LLM, Sonnet 4.6 optimizer LLM). The three numbers in a cell are ε-greedy / UCB1 / Thompson. All entries are single runs.
| Setting | Information used to choose the search strategy | LiveMath | SpreadsheetBench |
|---|---|---|---|
| Bandit (Table 2) | reward when a candidate skill is accepted or saved | 55.45 / 38.39 / 45.02 | 49.64 / 53.21 / 53.57 |
| Bandit, dense | screening gain of each candidate skill, every round | 43.13 / 53.08 / 60.66 | 51.79 / 54.29 / 50.71 |
| LinUCB | screening gain and five numbers describing the current run | 50.24 | 50.71 |
| LLM, reward table | the bandits' reward table only | 46.45 | 55.71 |
| StraTune (Table 2) | execution feedback and strategy and form histories | 65.88 | 56.43 |
A denser reward or context does not close the gap. Even the best variant stays well below StraTune on LiveMath. With only the information the bandits see, the optimizer LLM also falls short. It scores below StraTune on both datasets, and far below on LiveMath. These results suggest that StraTune's advantage comes from information that only an LLM can use. It reads which samples fail and why, together with the recorded outcome of each search strategy and revision form.
Total cost. We agree that an equal number of target-LLM executions does not guarantee equal cost. Table R4 therefore reports the total training cost of every method, including all target-LLM and optimizer calls.
Table R4. Total training cost per run in USD (Haiku 4.5 target LLM, Sonnet 4.6 optimizer LLM).
| Method | DocVQA | LiveMath | Mind2Web | SpreadsheetBench |
|---|---|---|---|---|
| GEPA | 7.2 | 24.2 | 129.7 | 23.2 |
| TextGrad | 8.6 | 49.5 | 77.8 | 39.7 |
| SkillOpt | 13.0 | 37.0 | 54.4 | 57.1 |
| Trace2Skill | 27.4 | 44.8 | 146.9 | 121.8 |
| Iter-CoT | 7.0 | 11.7 | 52.5 | 20.8 |
| StraTune | 8.7 | 20.2 | 48.0 | 33.6 |
StraTune costs less than SkillOpt and Trace2Skill on every dataset. It is also the cheapest method on Mind2Web and the second cheapest on LiveMath.
Transfer across model families. We applied the skills learned with Haiku 4.5, without further optimization, to six target LLMs from other families: Llama 4 Maverick, Qwen3-VL 235B, Mistral Large 3, Gemma 3 27B, DeepSeek V3.2, and Kimi K2.5. Each skill is thus tested on 23 combinations of target LLM and dataset, namely the six target LLMs on four datasets, except DocVQA for the text-only DeepSeek V3.2 (Table R5).
Table R5. Transfer of the Table 1 skills to target LLMs from other families. Improvement is computed over the initial skill on the same target LLM.
| Skill | Mean improvement | Improved over the initial skill | Best skill |
|---|---|---|---|
| GEPA | +7.03 | 16 of 23 | 1 |
| TextGrad | +3.31 | 17 of 23 | 0 |
| SkillOpt | +8.24 | 18 of 23 | 1 |
| Trace2Skill | +4.19 | 16 of 23 | 0 |
| Iter-CoT | +12.56 | 20 of 23 | 6 |
| StraTune | +17.50 | 22 of 23 | 16 |
StraTune's skills transfer best across model families. They improve on the initial skill in all but one combination and are the best skill far more often than any other method.
Transfer to unseen document types, websites, and domains. We also applied the skills with Haiku 4.5, again without further optimization, to three test sets that the paper does not use. The DocVQA skills were run on InfographicVQA, and the Mind2Web skills on the two other official Mind2Web test splits, with unseen websites and unseen domains. StraTune improves on the initial skill on all three test sets, by 20.22, 0.95, and 1.53 points. Among the baselines, only TextGrad also does so.
We thank the reviewer again. The revised paper discusses the LiveMath option in Section 4.2 and the transfer to other model families in Section 4.3, and adds these results to Tables 4 and 6 and Appendices D.2, D.5, and D.7. All revisions are marked in red.
Dear Reviewer Hoq3,
Thank you again for your review. We have responded to each of your points above and revised the paper accordingly, with all revisions marked in red. As the discussion period ends on November 18, we would appreciate it if you could let us know whether our response addresses your concerns, or whether any points need further clarification.
The authors
Q1. LiveMath accuracy with and without the "stronger result" option.
We agree that this option is a pattern in LiveMath. Whenever it appears, it is the correct answer in 177 training questions and 86 test questions. Table R1 splits the Table 1 scores by whether a question contains this option. Following the reviewer's suggestion, it also includes the rule-based baseline the reviewer describes (Option rule). This hand-crafted rule selects this option whenever it appears and otherwise answers with the initial skill.
Table R1. LiveMath test accuracy by question type (Haiku 4.5 target LLM, Sonnet 4.6 optimizer LLM, Table 1 runs).
StraTune also improves the questions without the option. On these 125 questions, it has the highest accuracy of all methods and is significantly above GEPA (p = 0.04) and SkillOpt (p = 0.005) in a paired sign-flip test. On the 86 questions with the option, StraTune's skill is also the best of all learned skills, and the target LLM chooses the option far more often with it than with the skills of GEPA and SkillOpt. The Option rule method is written with the pattern known in advance, whereas StraTune discovers it from execution feedback. This prior knowledge gives Option rule an unfair advantage on the 86 questions with the option, where it is always correct. On the 125 questions without the option, however, it does no better than the initial skill, while StraTune has the highest accuracy there. Option rule therefore scores higher overall only because of the questions with the option.
Q2. Repeated runs of the baselines and controls.
We agree that single runs of the controls cannot show whether the advantage is consistent. We therefore ran the three baselines that are strongest on at least one dataset in Table 1 (GEPA, SkillOpt, and Iter-CoT), Rot., Rand., and the best fixed search strategy three more times each, with new seeds and the same budget and data splits. Together with the run in the main manuscript, each setting now has four runs (Table R2).
Table R2. Test scores over four runs, mean ± standard deviation (Haiku 4.5 target LLM, Sonnet 4.6 optimizer LLM).
For the best fixed search strategy, we take the strategy that scored highest among I1 only, I2 only, and I3 only in Table 2 on each dataset. This is I1 on DocVQA, I3 on LiveMath and SpreadsheetBench, and I2 on Mind2Web. Appendix D.2 also reports an earlier Rot. run on SpreadsheetBench (45.71), which is not among these four runs. With it, the five runs average 51.93.
The single runs in the main manuscript in fact make the ablations look stronger than they are. On LiveMath, Rot. drops from 65.40 in Table 2 to a mean of 56.87, and I3 with F3 from 63.51 in Table 12 (Appendix D.1) to 44.79. The best fixed search strategy also averages lower than in Table 2 on every dataset. Over four runs, StraTune has the highest mean of all settings on every dataset except Mind2Web. On Mind2Web, Iter-CoT has the highest mean, just as it has the highest single-run score in Table 1. Adaptive selection therefore has a higher mean than random, cyclic, and fixed choices of search strategy on every dataset, with the largest margins on LiveMath.