acceptodds
Under review as a conference paper at ICLR 2027

SkillTrainBench: Can LLM Agents Author Skills That Improve Other LLM Agents?

Abstract

Skills are compact packages of instructions and reference material supplied to an LLM agent's harness, allowing its behaviour to change without updating model weights. We ask whether LLM agents can write such skills for other LLM agents. SkillTrainBench gives a curator agent a fixed interaction budget with one or more learner agents whose model weights remain fixed. The curator iteratively writes and revises a skill, chooses which learner-task pairs to probe, and uses the resulting performance to guide further revisions. The final skill is evaluated on private held-out tasks. We study four domains drawn from HealthBench, Humanity's Last Exam, quantitative-finance evaluations, and τ³ Banking, using multiple frontier curators and learners spanning a range of capabilities. We find that curator-written skills improve every learner on HealthBench and transfer to models that never saw them, but rarely help on the other three domains. Skill quality is also non-monotonic: performance frequently peaks before the final iteration, and curators rarely submit their best skill because the development scores they see cannot rank their own drafts. SkillTrainBench therefore provides a controlled setting for studying how LLM agents can improve other LLM agents through externally authored skills, and where this process currently fails.

Primary area:
datasets and benchmarks
Keywords:
LLM Agent, Skill Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_58eYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SkillTrainBench, which aims to test the capabilities of LLM agents authoring skills for other LLM agents. With tasks on HealthBench, \tau^3-bench banking, HLEBench and QFBench, this paper provides some preliminary empirical results on this specific setting.

Critical strengths
  • The investigated setting seems interesting.
Critical weaknesses
  • This submission seems to be largely written by generative AI. The overall logic and presentation are not smooth for human readers. The findings are also not presented in a precise manner.
  • The overall novelty of this paper is quite limited. This paper only introduces an interesting setting (while this setting is not novel and investigated by many prior papers) without constructing novel tasks (or providing enough justification on why we should choose these four tasks) or novel agent framework. Thus, I do not think this paper meets the basic bar of ICLR.
Questions for authors

None

2 comments

Dear Reviewer 58eY,

We thank you for your time and thoughtful comments. We appreciate the opportunity to clarify the contribution and improve the presentation. We have expanded the benchmark rationale and added a human–AI curator ablation.

The contribution is a benchmark protocol and empirical study of skill authoring. A curator chooses development tasks, spends a metered learner-interaction budget, revises a package and declares its final submission. Private held-out evaluation measures the submitted package, saved revisions and transfer to other learners. The study covers 36 curator–learner–domain configurations. We do not claim that generating reusable instructions is itself new; Related Work now distinguishes our evaluation focus from SkillsBench, SkillLearnBench, SkillGenBench and Metacognitive Reuse.

The four benchmarks were chosen to cover distinct interaction and verification settings: clinical consultation with rubric grading, interactive banking, expert science answers, and quantitative-finance checks. This makes it possible to test whether an authoring method's gains extend beyond one type of task.

In the new §5.3, we report a human–AI curator ablation. We organized a hackathon in which human–AI teams replaced autonomous curators while performing the same skill-authoring task. We allocated approximately matched learner-token budgets and $50 in OpenAI credits per participant, with two participants per team. Teams could also use additional coding assistants; learner probes and evaluation used GLM 5.3 Flash through OpenHands. On 159 shared HealthBench tasks with GLM 5.3 Flash, 5, 7 and 9 of 18 teams exceed the respective Opus-, Fable- and GPT-curated packages. The cohort has 18 HealthBench entries and 16 per other domain. Human HealthBench evaluation allowed four learner iterations versus twelve for autonomous packages, and total curator resources differed. We therefore present a descriptive comparison, not a controlled ranking of authors.

We take responsibility for the writing and presentation. We corrected duplicated and miscopied appendix tables, repaired error counts, and made the domain and checkpoint-selection conclusions more precise. We hope the clarified contribution, additional evidence and presentation changes address your concerns. If so, we would be grateful if you would consider updating your assessment. We welcome any further questions and thank you again for your comments.

Thanks for the response. I would like to maintain my score as a clear reject.

Reviewer_8iTVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SkillTrainBench, a benchmark for evaluating whether LLM agents can author skills that improve frozen learner agents. Under a fixed interaction budget, the curator selects development tasks, reads learner traces, and iteratively revises the skill. The submitted skill and intermediate versions are evaluated on private held-out tasks. Experiments across four domains with multiple curators and learners show consistent gains on HealthBench and transfer to other learners, but no consistent gains in the other three domains. The authors also find that curators cannot reliably identify which of their skills performs best on held-out tasks.

Critical strengths
  • The evaluation setup is meaningful and well-motivated. It separates the curator from the frozen learner, keeps held-out tasks private, and evaluates intermediate skill versions. This supports studying both skill effectiveness and how curators use feedback and select skills.
  • The experiments cover four domains, three curators, and three learners, with additional transfer evaluations. This variety helps reveal where written skills are useful and how their benefits vary across domains and models.
Critical weaknesses
  • The held-out sets are small, particularly QFBench with 14 tasks and τ3\tau^3 banking with 30 tasks. Many confidence intervals include both no improvement and substantial gains. The lack of consistent gains may therefore reflect insufficient evidence rather than limited skill effectiveness. This makes broad conclusions about differences across domains less convincing.
  • Evaluation variability makes the reported gains and skill rankings difficult to interpret. Table 11 shows that evaluating the same unchanged skill twice gives scores of 0.2095 and 0.1429. Table 20 shows substantial variation in the no-skill baseline scores for the same learner and domain, including Opus 5 on τ3\tau^3 banking and GLM 5.3 Flash on QFBench. For Opus 5 on τ3\tau^3 banking, the no-skill baselines are 0.433 and 0.300 in the Opus 5 and Fable 5 curator runs, respectively. Thus, despite identical final scores of 0.367, the baseline differences alone produce opposite gross lifts of −0.067 and +0.067. More repeated evaluations are needed to separate genuine skill differences from evaluation noise, especially when concluding that curators fail to select their best skill.
Questions for authors
  • Could you address the concerns raised in the Weaknesses section?
1 comment

Dear Reviewer 8iTV,

We thank you for your time and thoughtful comments. We appreciate your careful examination of the evaluation variability and test-set sizes. We agree: wide intervals do not establish ineffectiveness, and single evaluations cannot establish reliable checkpoint rankings. We have revised both claims and added repeated evaluations and a larger QF test split (Appendix M).

On the original 14 QF tasks, three executions per condition with Sonnet 4.6 gave a mean skill effect of −0.20 percentage points, 95% CI [−10.71,+12.62], averaging repetitions within each task. The unchanged package scored 48.33, 54.01 and 67.03 across executions, confirming substantial variability. Separately, fresh Opus curation on 34 development / 34 held-out tasks gave −5.25 points [−20.94,+10.13]. We retained the medium/hard task pool and ran fresh curation, avoiding evaluation of an old package on its former development tasks. Both effects remain inconclusive; repetitions do not increase the number of independent tasks.

We verified your banking example: both final packages score 11/30, and the different baselines alone reverse their gross lifts. Their direct paired score difference is 0.00 points [−16.67,+16.67]. The archived GLM/QF baselines also reproduce as 0.605, 0.592 and 0.330; a timeout explains only part of that variation. Our Sonnet repeats are not GLM replications, and banking still has 30 test tasks.

We replicated one HealthBench package's benefit on 160 tasks: +28.20 points [22.62,33.80]. We therefore retain the positive HealthBench finding while describing effects elsewhere as unresolved. The unchanged HLE package's discrepant evaluations also caution against interpreting hindsight checkpoint maxima as reliably better skills. We have not rerun the cited HLE or banking comparisons.

We hope these additional results and clarifications address your concerns. If so, we would be grateful if you would consider updating your assessment. We welcome any further questions and thank you again for your helpful comments.

Reviewer_X24JOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SkillTrainBench, a benchmark for evaluating whether an LLM agent can develop resuable artifacts (skills) that improve the performance of another LLM agent without modifying it's model weights.

The LLM agent (curator) iteratively writes skills, test them on certain tasks, observers learner performance and revises it's instructions for skill writing. The resulting skills are evaluated on private held-out tasks across 4 domains: HealthBench, HLEBench, QFBench and τ³ Banking.

The paper reports 2 central findings:

  • Effectiveness of skill writing is domain dependent: Curator generated skills improves performance on HealthBench, including when transferred to previously unseen learner models. Improvements on the other 3 benchmarks are less clear.
  • Credit assignment across generated skills: Curator struggle to identify their best skills i.e., sometimes intermediate versions frequently outperform the submitted version.

The paper's main contribution is an evaluation framework that measures the effectiveness of generated skills.

Critical strengths

The paper address an interesting question: Can improvements in agent capability be transferred through resuable artifacts (skills) rather than model training ?

The distinction between generating a useful skill and recognising which generated skill is best particularly valuable. The paper shows that these are separate capabilities, and that progress in the former does not imply progress in the latter.

The paper's significance relies on establishing what actually is responsible for the improvements. Hence, it's important to distinguish the transfer of useful task knowledge from changes in the learner's tool use behaviour or leaner's response style.

Critical weaknesses
  1. The paper's strongest positive result on HealthBench , but section 6.3 indicates that a good part of the improvement comes from learners answering questions they previously deferred to clinicians. The authors investigate answer length and report that it explains less than 20% of the improvement, but it still does not fully establish which aspects of the generated skills are responsible for the improvement.

  2. One of the paper's interesting finding is that curator is not able to select the best performing skill. However the current experiments do not fully separate limitations of the curations's decision making from limitations of the feedback available to it.

It's not weakness but just related work. Didolkar et al. (2025), Metacognitive Reuse: Turning Recurring LLM Reasoning Into Concise Behaviors (arXiv:2509.13237) extracts reusable procedural instructions from LLM reasoning traces and demonstrates improvements through in-context reuse, including across different models. This work It is closely related to the motivation for SkillTrainBench, particularly its investigation of whether one model can generate instructions that benefit another.

Questions for authors
  1. Suggested ablation: I encourage the authors to conduct a targeted ablation of the HealthBench skills, separating domain specific knowledge, response completeness and safety or referral guidance. Doing these ablations will help determine whether the improvements are because of transferring domain knowledge or from eliciting behaviours that the learner could already perform.

  2. To address the second weakness (see critical weaknesses section): An ablation comparing selection using the existing probe scores against selection using repeated evaluations of a fixed development subset would be valuable. The author could measure whether better feedback improves the curator's ability to select strong skills under a comparable evaluation budget. It will also be helpful to see what happens if we give more thinking budget to the curator.

1 comment

Dear Reviewer X24J,

We thank you for your time and thoughtful comments. We agree that identifying the source of the HealthBench gains and separating feedback quality from curator judgment are important. We added a targeted content ablation and a rubric-level analysis, and narrowed the claims these results do not establish.

The content ablation is inconclusive. On the same 160 HealthBench tasks, the intact skill scored 40.51 versus 38.85 after deleting five prespecified clinical-fact/example passages: a difference of +1.66 percentage points, 95% CI [−2.23,+5.63]. Thus, this experiment does not establish whether those passages contribute to the gain. We retained general response and safety guidance; the deletion also changes package length and leaves some clinical content. This is a partial test of your proposed decomposition, not separate ablations of domain knowledge, completeness, and referral guidance.

The rubric analysis locates most of the observed gain in completeness and context awareness. Together, these two axes contribute 22.04 of the 28.20-point improvement in the repeated intact-skill versus no-skill comparison. We used all five original HealthBench rubric axes and retained their contributions to the original score (Appendix M). This finding supports a narrower account of what improved, but does not identify which instructions caused it or distinguish transferred knowledge from elicited behavior. We have revised the mechanism claims accordingly.

We agree that the observed checkpoint rankings do not distinguish limitations in feedback from limitations in curator judgment. Repeated evaluations of unchanged skills produce substantial score variation, so a higher observed checkpoint score does not by itself establish a better skill. The revised paper therefore treats these rankings as descriptive and explicitly states that the hindsight comparisons do not establish that curators rejected reliably better packages. Our evidence supports this qualification, rather than an explanation based on deficient curator reasoning.

We also added the requested comparison with Didolkar et al., Metacognitive Reuse, in Related Work.

We hope the additional experiments, analysis, and revised claims address your concerns. We would be grateful if you would consider these revisions in your assessment, and we remain happy to answer further questions. We thank you again for your useful comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.