acceptodds
Under review as a conference paper at ICLR 2027

Mining Agent Skills from Production Traces

Abstract

Agent skills that record procedural instructions are increasingly mined from execution traces rather than curated by hand. Skill-mining pipelines often use known task outcomes or feedback to guide skill construction. In production, reliable information on whether a run has succeeded may be unavailable. We study how the choice of execution traces, access to success or failure information, and the form of the mined skills affect downstream task performance. Holding the mining pipeline fixed, we compare six combinations of mining evidence and skill forms. Mining evidence has three levels: successful trajectories only, successes and failures with their outcome labels, or the same mix with labels withheld. Skill form has two types: an ordered workflow plan, or a declarative ontology of entities, states, and policies. We evaluate the mined skills on two enterprise benchmarks, ThinkingBox-Bench and APEX-Agents. Analysis of task-level paired differences shows that the benefits of different configurations of mining evidence and skill forms depend on the enterprise domain. On ThinkingBox-Bench, paired differences show that workflows score better than ontology by pp, Goldilocks beats success-only evidence type by pp and Goldilocks blind simulating skills learnt without outcomes is worse by pp. APEX-Agents shows a moderate preference for ontologies and no clear preference between evidence regimes. Within each domain, task structure related constraints drive uneven performance with mined skills. These findings motivate tailoring meta-skills to the demands of the target tasks rather than adopting a one-size-fits-all approach.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, agentic AI, agent skills, skill mining, skill learning, workflows, ontologies, agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3y8pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to extract useful skills from the execution records of intelligent agents. The author compared using only the success records, using both success and failure records with result labels, and the same batch of records without labels, as well as comparing step-by-step workflows and declarative operational ontologies. Experiments checked the effectiveness of these configurations on tasks not involved in skill construction and analyzed whether the skills were read and why they still failed after reading.

Critical strengths

S1. The label comparison is persuasive. Fixing the same batch of execution records and changing the result labels and accompanying instructions makes it easier to see the effect of this configuration and avoid confusing it with a different batch of records.

S2. The evaluation separates skill construction from testing tasks, and also uses repeated runs and paired comparisons for the same task. This is more capable of demonstrating whether the skills are useful than only showing a few successful cases.

S3. The author checked whether the skills were read and whether they still failed after reading. These analyses can help distinguish problems with the skill content and actual usage issues, and also make negative results more interpretable.

Critical weaknesses

W1. I think this paper is more about comparing various approaches to generating skills. The experiment has some value, but there is not much novelty in the method. I still don't quite understand how these findings can help us improve the skills.

W2. Workflow is a global skill, but the ontology is generated separately for each task group, and the organizational methods on both sides are different. Although the paper mentioned this difference, now seeing the different effects, I'm not sure whether it's the representation form that matters, or the organizational method.

W3. Regarding the differences from existing work, I think it's necessary to explain clearly. "AWM" has already summarized reusable processes from the execution records, and "From Raw Experience to Skill Consumption" and "Demystifying Agent Skills" have also studied how to use success and failure records. This paper adds the comparison of workflow and ontology on enterprise tasks, but it's not clear yet what new key discoveries these experiments have brought.

W4. I looked at the prompt in the appendix and found that these prompts for generating skills actually provide a very specific operational framework, and it leans more towards business process tasks. Although the author also tested other types of tasks, I will still be concerned about how much the results depend on how well this framework matches the tasks. However, I haven't seen a control where skills are generated based solely on domain and tool information without considering the execution records. Therefore, it's still difficult to determine exactly how much additional improvement has been achieved from the experiences within the trajectory.

Questions for authors

Q1. Can you please specifically explain what the most crucial new discovery of this paper is compared to the works mentioned above, and how it can help us improve the skills?

Q2. Can you use the same set of records to generate skills in both forms, using the same task grouping, and then compare the effects using the same loading method?

Q3. Can you add a control group that generates skills without looking at the execution records, only based on domain descriptions and tool information, and test it on both business process tasks and analytical tasks? This way, I can see how much additional improvement the experience in the trajectory brings.

2 comments
AuthorsOpenReview ↗

Response to 3y8p: Practical Takeaways, Relation to Prior Work, and Prompt Framework

Thank you for the review. We are glad you found the label comparison persuasive, the separation of skill construction from testing with repeated paired runs more informative than showing a few successful cases, and the read-versus-failure analyses useful for interpreting negative results.

[W1, Q1] "How do the findings help us improve skills?" The findings translate into concrete choices:

  • If outcomes can be graded: mix failures with passes and keep the labels. Goldilocks beats Success-only by 2.4 pp in ThinkingBox-Bench, and keeping the labels adds 3.1 pp.
  • If labels are unavailable: expect smaller gains, and do not assume the same result on other task types. In APEX-Agents the label effect is about zero (+0.3 pp).
  • Check routing first: unread skills cannot help, and in some APEX-Agents domains only 28–47% of tasks read the workflow skill in every repetition.
  • Evaluate the mining configuration on representative tasks, because the best form and evidence differ by benchmark and domain.

[W3, Q1] Relation to AWM, Raw Experience to Skill Consumption, and Demystifying Agent Skills. Those works cover mining routines and some evidence and label effects, mostly with the evidence pool varying across conditions. Here the trajectories are held fixed while label visibility changes, and the setting is exact-state enterprise tasks. We also report routing-versus-execution failures for each condition. The paper adds these measurements to the existing picture and does not claim a new mining method.

[W2, Q2] Form vs. organization. Both forms are mined from the same fold's training pool and loaded as native skills with the same routing. Per skill, the miner sees at most ten tasks with five trajectories each. An ontology is organized by task group by nature, so having several skills is part of the form. We thank you for suggesting a scoped workflow on identical task groups, which would separate form from organization; we regard it as a valuable next step.

[W4, Q3] Dependence on the prompt framework; a no-trajectory control. The prompts were authored against ThinkingBox-Bench and reused unchanged for APEX-Agents, which has analytical deliverables, and gains there are smaller. So we agree that framework fit matters, and it is one reason the paper recommends tailoring meta-skills. The framework is fixed across arms, yet changing only the trajectories produces large differences, for example Success vs. Goldilocks in retail and hotel (4.5 and 4.9 pp) and labeled vs. blind in ThinkingBox-Bench (3.1 pp). The framework alone cannot explain these. A control that generates skills from domain and tool descriptions without trajectories would need a different meta-skill. We thank you for suggesting it; it is an interesting direction, but we expect it to add little production insight, since a deployment would compare the mined configurations that are actually available to it. The no-skill baseline gives the total lift, but we have not run the no-trajectory control, so we cannot yet quantify the added value of trajectory experience.

Summary: We clarified the practical implications, how the work relates to prior studies, and what the prompt-framework concern does and does not affect. We hope this addresses your concerns, and we would be glad to answer follow-up questions.

Thank you for the response. The paper still lacks comparisons using the same task grouping and generating skills without execution records. These are important for understanding where the gains come from, so I will keep my score.

Reviewer_Aj2HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how mining evidence and skill representations affect the downstream performance of LLM agents. Specifically, it compares three evidence configurations (Success, Goldilocks, Goldilocks Blind) and two skill representations (Workflow Plans, Operational Ontologies). Experiments on ThinkingBox-Bench show that workflow plans and Goldilocks evidence perform better on average, with Goldilocks outperforming both Success and Goldilocks Blind. On APEX-Agents, ontologies show a slight preference, while no evidence configuration demonstrates a clear advantage. Overall, the results suggest that the effectiveness of skill configurations varies across task domains, with no universally optimal choice.

Critical strengths
  1. The paper addresses a clear and practically relevant question: how skill representations and outcome-label availability affect downstream performance during skill mining. The experimental design is well organized, including a matched-trajectory comparison between Goldilocks and Goldilocks Blind to examine the contribution of outcome labels.
  2. The paper is clearly structured, with detailed descriptions of the methodology and experimental setup. Beyond reporting overall performance and statistical results, the authors provide domain-level analyses and concrete examples to explain the benefits and regressions of different skill configurations, offering useful empirical insights into their applicability across different tasks.
Critical weaknesses
  1. The paper mainly evaluates combinations of existing trajectory selection strategies and skill representations, with limited methodological novelty. Its main conclusion that different configurations perform differently across domains is relatively intuitive, without systematically validating which task characteristics predict the best configuration. As a result, its contributions beyond prior work remain limited.
  2. The comparison between Workflow Plans and Operational Ontologies simultaneously changes the number of skills, mining scope, and evidence allocation, making it difficult to isolate the effect of skill representation. Similarly, Success and Goldilocks differ in both task difficulty distributions and the composition of successful and failed trajectories. While this allows a comparison of the two evidence selection strategies, it does not isolate the contribution of including failure trajectories.
Questions for authors
  1. Prior studies have already investigated the effects of outcome labels on skill generation and the task-dependent benefits of agent skills. Beyond comparing different configurations on enterprise benchmarks, what are the key new insights of this work? Which findings provide new understanding beyond existing studies?
  2. In §3.2, Workflow Plans use a single global skill, whereas Operational Ontologies generate separate skills for each task group, simultaneously changing the number of skills, mining scope, and evidence allocation. How do the authors distinguish the effects of skill representation from these factors? Have they considered controlled comparisons with matched skill counts, mining scopes, and evidence budgets?
  3. On Page 2, the authors state that "Goldilocks-blind selection yields lower point estimates than Goldilocks across all reported settings." However, Table 2 shows that Goldilocks Blind outperforms Goldilocks in some settings, such as Car Insurance with Workflow Plans (+7.0 pp vs. +3.7 pp). Could the authors clarify or revise this statement?
  4. In §3.3, Success and Goldilocks differ in both the baseline success rates of selected tasks and the composition of successful and failed trajectories. Additionally, Goldilocks and Goldilocks Blind differ in their mining instructions as well as label visibility. Could the authors provide more controlled ablations, such as matched-task comparisons and experiments that vary only label visibility, to better isolate the contributions of evidence composition and outcome labels?
  5. Both skill mining and downstream execution use GPT-5.4, and each skill is generated through a single mining call. Could the authors further investigate whether the relative performance of different skill configurations remains consistent across other models or independent mining runs with different random seeds?
  6. The paper contains several typos and formatting issues, such as "constrative labels help resolving branches" (Page 2) and the duplicated parentheses in "((Zhang et al., 2026b; Xiong et al., 2026))" (Page 2, Line 95). A careful proofreading of the manuscript is recommended.
1 comment
AuthorsOpenReview ↗

Response to Aj2H: Key Findings, Matched Comparisons, and Corrections

Thank you for the review. We are glad you found the question practically relevant, the matched-trajectory label comparison well designed, and the structure, domain-level analyses, and concrete examples useful.

[W1, Q1] "Limited novelty; conclusions intuitive; key new insights?" The findings we think are new are specific and not obvious from prior work:

  • Label access on fixed trajectories. Withholding labels from the same trajectories costs 3.1 pp in ThinkingBox-Bench [1.4, 4.7] and about 0.3 pp in APEX-Agents. This targets a production constraint.
  • Evidence that includes failures. Goldilocks beats Success-only evidence in retail and hotel by 4.5 and 4.9 pp (intervals exclude zero). This is not seen in APEX-Agents.
  • Routing vs. execution. Skills are often not read: only 28–47% of investment-banking and law tasks read the workflow skill in every repetition. This separates a routing problem from a content problem, and tells practitioners where to intervene.
  • Direction depends on the benchmark. Workflow plans lead in ThinkingBox-Bench and ontologies slightly lead in APEX-Agents. We state this as an empirical finding, and the paper does not claim a rule for predicting it.

[W2, Q2] Skill count, scope, and evidence allocation. Per skill, the miner sees at most ten tasks with five trajectories each, for both forms. Ontology arms run one such call per task group. Loading is identical (native skills). We thank you for suggesting a comparison with matched task groups (a scoped workflow); we regard it as a valuable next step.

[Q3] "Lower across all reported settings." Thank you for catching this. The sentence is too broad. In the domain-level contrast (Goldilocks − Blind, averaged over both forms), the pooled estimate is positive in all five ThinkingBox-Bench domains and in two of three APEX-Agents domains (law is −0.7 pp). At the level of individual domain-by-form cells there are exceptions, such as car insurance with workflow plans (+3.7 vs. +7.0 pp). This is the accurate statement.

[Q4] Matched-task and label-visibility-only ablations. Goldilocks vs. Goldilocks Blind is already that comparison: the exact same task and trajectory cohort, with only the labels hidden. The evidence description changes because it cannot refer to labels that are hidden (the workflow meta-skill text changes by one word, "labeled" to "unlabeled"). Success vs. Goldilocks is not matched by design, because Success uses passing trajectories only, from tasks with 8–10 baseline passes. A failure-share sweep on a fixed task pool would separate failures from difficulty. We thank you for the suggestion; it is an interesting direction, but we expect it to add little signal for a production decision between the full configurations.

[Q5] Other models and mining seeds. We use a single model, GPT-5.4, for mining and execution; this is a limitation of the current study. Repeating the study with other models is an interesting direction, but a deployment would re-run the comparison on its own model, so we expect it to add little to production guidance. Each arm's skills come from five separate mining calls, one per fold, so the results are not from a single draw.

[Q6] Typos. Thank you for the careful proofreading. We acknowledge the typos and the duplicated parentheses you identified.

Summary: We gave the accurate Goldilocks-vs.-Blind statement and clarified what each comparison isolates. We hope this addresses your concerns about isolating the effects of form and evidence, and we would be glad to answer further questions.

Reviewer_vz9EOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This empirical study examines how evidence selection, outcome-label access, and skill form affect skills mined from agent trajectories. With a fixed mining pipeline, it compares Success, Goldilocks, and Goldilocks Blind evidence with workflow and ontology representations, using five-fold held-out evaluation and repeated runs on ThinkingBox and APEX-Agents. Benefits depend on the domain: ThinkingBox more clearly favors workflows and labeled Goldilocks evidence, while APEX differences are smaller and less certain. Supplementary studies include Random Blind mining, skill-use audits, and task-level case analyses.

Critical strengths
  1. The questions are practically relevant, and the matched-trajectory comparison is informative. Reliable outcome labels may be unavailable in production. Goldilocks and Goldilocks Blind retain the same trajectories and ordering, making the effect of label access and its associated mining instructions easier to interpret than comparisons that replace the evidence pool. Random Blind additionally tests selection without consulting outcomes. Together these conditions examine meaningful constraints on skill mining (Section 3.3; Appendix A.7).

The evaluation has useful scale and reporting granularity. Both benchmarks use five-fold held-out evaluation, five skill-enabled runs per task and condition, and task-level paired analyses. The paper reports domain differences, gains and regressions, failures to read skills, and failures after reading them, with multiple-comparison corrections in the supplement. It also treats case analyses as exploratory and avoids asserting a universally optimal configuration. This is useful evidence about the limits of deployed skills (Section 4; Tables 1–4; Appendices A.5–A.8).

Critical weaknesses
  • Confounded experimental factors limit interpretation of the central comparisons. Workflows produce one global skill per fold, whereas ontologies produce multiple task-group-specific skills, changing mining scope, skill count, and available evidence quantity together. Their difference therefore cannot be attributed specifically to representation. Success and Goldilocks also select tasks with different baseline pass frequencies while changing the successful/failed trajectory mix. The paper acknowledges these limitations in Sections 3 and 5.2, but they leave the central questions answered at the level of bundled configurations rather than provide well-isolated evidence for transferable design choices.

  • Unresolved measurement issues weaken confidence in the results. Appendix A.5 states that APEX results precede corrections for judge-service errors and retain inconsistencies between scores and grading rationales. Historical no-skill controls were also not randomized alongside skill conditions, leaving service drift possible. Since aggregate APEX gains are small and none of its six baseline comparisons meets the conventional 0.05 threshold after Holm correction, corrected grading and contemporaneous controls could materially affect the conclusions. Domain-level explanations should remain exploratory; selected cases cannot replace robust aggregate evidence (Tables 1–3; Appendices A.4–A.6).

Questions for authors
  1. Can workflow and ontology be compared with the same skill count, task grouping, evidence trajectories, and approximately matched context budget? Separately, does varying the inclusion of failed trajectories within a fixed task pool preserve the main ThinkingBox findings?

  2. How many APEX runs are affected by judge-service errors or score/rationale inconsistencies, and are they balanced across conditions? Do corrected or repeated grading results change the main gains, configuration ordering, or confidence intervals?

  3. Are contemporaneous no-skill controls available, or are there other measurements quantifying drift relative to historical controls? Which main conclusions remain when the analysis is restricted to these comparable runs?

The study provides substantial enterprise-agent experiments and useful observations about heterogeneous skill effects; negative findings are valuable in their own right. My concerns are the insufficient isolation of the main factors and unresolved grading problems, rather than the small effect sizes alone. The evidence currently supports exploratory observations about specific configurations more strongly than clear and reliable general conclusions, leading me to weak rejection.

1 comment
AuthorsOpenReview ↗

Response to vz9E: Confounds, APEX Grading Sensitivity, and Controls

Thank you for the careful, constructive review. We are glad you found the matched Goldilocks/Goldilocks Blind comparison informative, the scale and reporting granularity useful (gains, regressions, read failures, multiplicity corrections), and the exploratory framing appropriate.

[W1] "Confounded factors; conclusions are about bundled configurations." We agree the form comparison is about packaging strategies, but we disagree that scope or evidence quantity is a free confound. Per skill, the miner sees at most ten tasks with five trajectories each, for workflow and ontology alike. Ontology arms run one such call per task group. That a scoped ontology has several skills is intrinsic to the representation. The evidence-selection and label comparisons are within one form, with scope and skill count fixed, so they do not depend on this point. For Success vs. Goldilocks, we agree that the pools differ in difficulty: Success tasks have 8–10 passes in ten baseline runs and Goldilocks tasks have 3–7. The comparison is best read as one between selection strategies.

[W2] "Grading issues and non-contemporaneous controls." The grading issues are minor and do not affect any conclusion. They touch only about 1% of APEX-Agents grades, they are spread evenly across arms, and correcting for them changes no ordering or sign of the main estimates (details under Q2).

  • Judge-service errors affected 48 of 19,200 grades, as in the paper: 16 of 4,800 control runs and 32 of 14,400 skill runs.
  • Score/verdict conflicts, where the judge's explicit verdict contradicts the recorded criterion score, affected 190 further grades: 40 control and 150 skill.
  • Balance. Overall, about 1.2% of runs are flagged, and the share per arm is similar (1.1–1.4%).

[Q1] Matched skill count/grouping/budget, and a fixed-pool failed-trajectory ablation. Per-skill budgets are already matched, as above. We thank you for these suggestions. A scoped workflow on the ontology's task groups would separate form from organization, and we regard it as a valuable next step. A failure-share sweep on a fixed pool is also interesting, but we expect it to add little signal for a production system, which must still choose between the full configurations compared here.

[Q2] How many APEX runs are affected, and do corrections change the results? The counts are above, and the effect of the issues is small. We re-ran every estimate with the explicit verdict substituted for conflicting scores; this is a sensitivity check, not a validated regrade.

  • All six baseline lifts rise by 0.1–0.35 pp, and their ordering is unchanged.
  • Direct contrasts move by at most 0.12 pp. Workflow − Ontology goes from −0.71 to −0.60 pp, Success − Goldilocks from +0.40 to +0.29, and Goldilocks − Blind from +0.31 to +0.33. All intervals include zero in both versions.
  • Bounding the 48 judge-service errors changes the sign of no pooled estimate. We agree that no APEX baseline comparison survives Holm correction. The paper treats APEX gains as modest and does not draw conclusions from them.

[Q3] Contemporaneous controls and which conclusions survive. The main contrasts (form, evidence selection, label access) use no control, so control drift cannot affect them. Retail and hotel use fresh, contemporaneous five-run controls collected with the skill runs. In both domains the control-free contrasts hold: Goldilocks over Success (+4.5 and +4.9 pp) and labeled Goldilocks over Goldilocks Blind (+4.8 and +5.1 pp). The other domains use earlier controls, and the paper reports a pilot in which a fresh consulting control passed 25.8% against 33.8% for earlier runs, so baseline lifts against earlier controls are best read as exploratory.

Summary: The label-access result, the within-form selection results, and the ThinkingBox-Bench conclusions do not depend on the controls or on the flagged APEX grades. We hope this resolves your concerns about measurement and isolation. We would be glad to answer follow-ups.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response: Strengths, Isolation of Factors, and Scope of the Claims

We thank all three reviewers for their careful reading. We are glad they found the question practically relevant: how skills should be mined when outcome labels may be unavailable in production. They also singled out several strengths of the design:

  • The matched-trajectory Goldilocks vs. Goldilocks Blind comparison, which isolates the effect of label access (all three reviewers).
  • Held-out five-fold evaluation with repeated runs, paired analyses, and multiple-comparison correction (vz9E, 3y8p).
  • The skill-read and post-read failure analyses, which separate routing problems from content problems and make negative results interpretable (vz9E, 3y8p).
  • The clear structure and the task-level case analyses (Aj2H).

Isolating factors (raised by all three): We want to state precisely what the design does and does not isolate.

  • Label access is isolated. Goldilocks and Goldilocks Blind use the same trajectories in the same order. Only the labels and the minimal wording that describes them differ.
  • Success vs. Goldilocks are compared as two realistic evidence-selection strategies. Success evidence is drawn from tasks with 8–10 passes in ten baseline runs, and Goldilocks evidence from tasks with 3–7 passes. We report them as bundled strategies.
  • Workflow vs. ontology compares the two canonical packaging choices. A workflow is one procedure for a deployment. An ontology is scoped by task group, so having several skills is part of what an ontology is. The factors one might suspect are matched. Every mining call, whichever form, sees at most ten tasks with five trajectories each. Both forms are loaded as native skills with the same routing.

What the paper shows and does not show: Within a fixed pipeline and model, it shows which configurations perform better on held-out tasks in eight domains, with label access isolated. It does not separate the contribution of skill count from that of form, the contribution of failure share from that of task difficulty, or generalization across models. We thank the reviewers for suggesting further ablations. A scoped workflow on identical task groups, which would separate skill form from skill organization, is a valuable next step. Others, such as varying the failure share on a fixed task pool, generating skills from domain and tool descriptions without trajectories, or repeating the study with additional models, are interesting directions, but we expect them to add little to what a production deployment can act on.

APEX-Agents measurement (vz9E). The judge-service and grading issues are minor. They affect about 1% of APEX-Agents grades, are balanced across arms, and change no ordering or sign of the main estimates. Details are in our reply to vz9E.

Text accuracy: We will correct one over-broad sentence (Goldilocks vs. Blind "across all settings") and the typos Aj2H identified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.