acceptodds
Under review as a conference paper at ICLR 2027

EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?

Abstract

Modern LLM-based agents rely on an evolving harness of tools, reusable skills, and specialist agents, yet existing evaluations largely assume this harness is fixed. We introduce EvoHarnessBench, a benchmark for systematically evaluating agents under controlled harness evolution along three axes: tools, skills, and agents. Unlike existing continual-learning benchmarks for agents, which typically place non-stationarity in the task stream while keeping the harness fixed, EvoHarnessBench places non-stationarity in the externally supplied harness itself. The benchmark contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks, comprising 802 tasks, 520 tools, 42 skills, and 62 agents. We evaluate two complementary settings that capture the central challenges of harness evolution: deployment evaluation, which measures whether agents retain previously accessible competence as the harness expands, and self-evolving adap- tation evaluation, which tests whether accumulated experience remains useful as new capabilities are introduced. Our experiments reveal three persistent limitations in state-of-the-art agents. First, harness expansion alone can degrade performance on previously solved tasks, leading to harness-induced forgetting. Second, gains from self-evolving adaptation are inconsistent across stages, capability axes, and environments. Third, retention and adaptation can be in tension: preserving earlier competence does not necessarily improve adaptation to newly introduced capa- bilities, and vice versa. Together, these results establish harness evolution as a distinct and important challenge for building agents that can continuously exploit new capabilities while preserving previously effective behavior.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Agent Benchmarks, Harness Evolution, Continual Learning, Self-Evolving Agents, Tools, Skills, Multi-Agent Systems, Harness-Induced Forgetting

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_isCrOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces EvoHarnessBench, a benchmark measuring agents' adaptability to an evolving harness along 3 capability axes: tools, skills and specialist agents, along with its scalable construction pipeline. The benchmark evaluates both deployment robustness and self-evolving adaptation, measuring task performance, retention of previously acquired capabilities, and adaptation to newly introduced capabilities. Experiments across multiple agent systems and adaptation methods reveal harness-induced forgetting, inconsistent adaptation gains, and potential trade-offs between retention and adaptation.

Critical strengths
  1. The paper introduces a diagnostic framework for evaluating agents under evolving harnesses, going beyond conventional static benchmarks to characterize how capability expansion affects task performance, retention, and adaptation. This provides a valuable diagnostic perspective on agent robustness and self-evolving capabilities.

  2. The paper proposes an automated method to construct evolving harness benchmarks from existing agent benchmarks completely without LLM. By leveraging existing task annotations and deterministically deriving capability requirements from procedural and tool structures, the pipeline enables cost-effective, scalable and reproducible benchmark construction.

  3. The paper presents extensive experiments across 3 harness evolution axes, multiple agent architectures, and diverse self-evolving adaptation methods. The evaluation includes detailed transfer analyses, ablation studies, and investigations of failure mechanisms, providing a thorough empirical characterization of agent behavior under harness evolution.

Critical weaknesses
  1. Limited validation of capability annotations. Although the deterministic construction pipeline is a strength, the reliability of the derived capability annotations is not sufficiently validated. In particular, skill annotations rely on heuristic matching between procedural content and task requirements, while specialist agents are constructed by grouping tools based on their associated entities. These annotations may not accurately reflect the capabilities actually required to solve each task. This raises concerns about the validity of the task-specific references and the interpretation of capability-related analyses.
Questions for authors
  1. Have the authors independently validated the accuracy of the automatically derived skill and agent annotations? How might annotation errors affect the reliability of the benchmark?
  2. How sensitive are the main findings to the frequency-based capability release order? Would alternative release schedules lead to similar forgetting and adaptation patterns?
3 comments
AuthorsOpenReview ↗

Response to Reviewer isCr: Part 1

Thank you for your constructive review. We appreciate your recognition of our "valuable diagnostic perspective on agent robustness and self-evolving capabilities", the "cost-effective, scalable and reproducible benchmark construction" without LLM involvement, and our "extensive experiments" across three harness evolution axes, agent architectures, and adaptation methods, providing a "thorough empirical characterization of agent behavior under harness evolution." We address your comments below.

W1. “Limited validation of capability annotations. Although the deterministic construction pipeline is a strength, the reliability of the derived capability annotations is not sufficiently validated. In particular, skill annotations rely on heuristic matching between procedural content and task requirements, while specialist agents are constructed by grouping tools based on their associated entities. These annotations may not accurately reflect the capabilities actually required to solve each task. This raises concerns about the validity of the task-specific references and the interpretation of capability-related analyses.”

Thank you for raising this comment. We address annotation validity at three levels.

1. The annotations are grounded in source-benchmark information. Our deterministic pipeline preserves the original tasks, environments, and executable verifiers (Sec. 3.2). Tool annotations come directly from the source benchmarks. Skills are extracted from procedural descriptions and matched to verifier-checked outcomes for EOG, or task specifications and software requirements for ALE (Sec. 3.3, Appendix E). Specialist agents are derived from tool-ownership mappings, with coverage and disjointness checked in Appendix F.

For example, EOG-CSM task task_20251210_183114_382_99ba2325_071a72ab asks the agent to close a case and create a knowledge article documenting its resolution:

  • Verifier checks: The article is published, internally visible, assigned to the specified owner, and linked to the case as a resolution.
  • Matched skills: Linking Knowledge Articles matches the ownership and visibility constraints; Customer (Portal User) matches the visibility constraint.
  • Observed behavior: GPT-5.5 Codex read both skills in all three runs and passed both verifiers each time (3/3 successes; verifier score: 100/100).

This example shows a traceable link between annotations and task outcomes. As stated in Section 3.3, the annotations identify relevant skills without claiming necessity, sufficiency, or uniqueness.

2. Direct injection provides evidence that the task-specific matched skills are effective. We inject the matched skill content directly into the task prompt, removing the need for retrieval. The shuffled control supplies the same number of non-matching skills with approximately the same context length.

Table A. Verifier score (0–100) across SAS skill-access conditions, for EvoHarnessBench-EoG, with Codex-GPT-5.

ConditionCSMHRITSMOverall
Empty skills73.9 ± 3.662.6 ± 4.547.1 ± 4.861.6 ± 2.5
Retrievable stage-level oracle skills76.7 ± 2.765.5 ± 1.949.6 ± 6.164.3 ± 2.6
Shuffled direct injection74.3 ± 2.261.9 ± 2.248.0 ± 4.861.5 ± 1.7
Matched direct injection78.5 ± 1.864.7 ± 2.954.3 ± 5.665.6 ± 1.1

Matched injection improves verifier score over shuffled injection in all three domains, by 4.04 points overall. This supports the relevance of the extracted skills and their task associations.

3. Low invocation does not imply invalid annotations. Our benchmark provides skills for on-demand retrieval, leaving the agent to decide whether to read them. An unread skill therefore provides no direct evidence about its usefulness. Invocation also varies across models and adaptation methods: Fig. 5 and Appendix J.1 report 82% for task-specific GPT-5.5 Codex and 31% for task-specific GEPA, under their respective library denominators. Direct injection complements this analysis by removing the retrieval decision.

Finally, we clarify that task-specific annotations define controlled references, not performance upper bounds. We will also report a sampled manual audit of skill associations and specialist-agent mappings in the camera-ready version.

AuthorsOpenReview ↗

Response to Reviewer isCr: Part 2

Q1. “Have the authors independently validated the accuracy of the automatically derived skill and agent annotations? How might annotation errors affect the reliability of the benchmark?”

Thank you for this important question. Our deterministic construction is grounded in source-benchmark signals (Sec. 3.3): tool annotations are inherited directly, skills are matched to verifier-checked outcomes, and specialist agents are derived from tool-ownership mappings. Appendix F further validates specialist tool coverage and disjointness.

We acknowledge that annotation errors could be an influence factor, and we have clarified these limitations in Sections 3.1, 3.3, and 4 (Evaluation Setup). Importantly, final task success remains determined by the original executable verifiers, independently of the capability annotations. We will further assess annotation quality through the manual audit described in W1. Below we empirically demonstrate the quality of our constructed dataset.

Additional experiment on validity of our constructed skills. We evaluate SAS under four conditions: empty skills, retrievable stage-level oracle skills, shuffled direct injection, and matched task-specific oracle skill injection (Appendix J.1, Table J.2). Matched injection achieves the highest overall verifier score (65.6), outperforming empty skills (61.6), retrievable skills (64.3), and shuffled injection (61.5). The improvement is particularly notable on ITSM (54.3 vs. 47.1 with empty skills). These results suggest that directly providing task-matched skills can improve verifier-level performance, highlighting the importance of effective skill selection and utilization.

Table A. Verifier score (0–100) across SAS skill-access conditions, for EvoHarnessBench-EoG, with Codex-GPT-5.

ConditionCSMHRITSMOverall
Empty skills73.9 ± 3.662.6 ± 4.547.1 ± 4.861.6 ± 2.5
Retrievable stage-level oracle skills76.7 ± 2.765.5 ± 1.949.6 ± 6.164.3 ± 2.6
Shuffled direct injection74.3 ± 2.261.9 ± 2.248.0 ± 4.861.5 ± 1.7
Matched direct injection78.5 ± 1.864.7 ± 2.954.3 ± 5.665.6 ± 1.1

Q2. “How sensitive are the main findings to the frequency-based capability release order? Would alternative release schedules lead to similar forgetting and adaptation patterns?”

Thank you for this suggestion. Our frequency-based core-to-tail release schedule (Sec. 3.2) provides a controlled approximation of real-world capability growth (Fig. 2). We intentionally adopt additive expansion, keeping previously released capabilities and task requirements unchanged to isolate harness-induced forgetting. We have explicitly acknowledged the limitations of the frequency-based release ordering in Section 3.2, including its coupling of stage, catalog size, and capability specialization. As discussed in Section 5 (Conclusion) in the revision, we leave alternative release schedules and non-monotonic capability changes for future work.

Appreciate your response. I prefer to maintain my rating.

Reviewer_2mHvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether fixed-model agents retain access to existing competence and adapt to new tasks as external tool, skill, and specialist catalogs expand. EvoHarnessBench organizes 17 multistage streams across two environment families, covering 802 unique tasks, and separates reset deployment from cumulative memory-, prompt-, or code-based adaptation. Main results and catalog-width, skill-invocation, memory-content, and delegation-failure analyses show that expansion need not help and adaptation need not preserve previous performance. The contribution is an executable-verifier benchmark and failure analysis rather than a new best-performing agent metho

Critical strengths
  1. Fixed models and reset deployment make interference from an expanding external capability catalog separately measurable. This exposes long-running deployment risks that static agent benchmarks miss. That separation is a publishable contribution even without a new SOTA method.
  2. This benchmark construction is a useful methodological contribution. It converts existing agent benchmarks into staged capability-release streams while retaining their tasks, execution environments, and verifiers. Stage-wise adaptation/evaluation splits and reevaluation of earlier task cohorts allow adaptation methods to be compared on both new-task performance and retention under a common protocol. This is more informative for evaluating long-term adaptation than reporting final success rates alone, and the construction provides a concrete template for extending other static benchmarks.
Critical weaknesses
  1. Keyword-matched skill labels do not establish necessity, sufficiency, or uniqueness, and some models rarely read skills. Annotation-level new-capability pressure may therefore not represent behavioral pressure to use new skills, limiting cross-axis interpretation and conclusions about skill evolution. Frequency-based release under one ordering further limits scope, without negating the value of the other axes.
  2. Frequency ordering couples stage, catalog size, and task specialization. Old-cohort reevaluation and Figure 4 help, but one core-to-tail ordering and addition-only evolution do not establish robustness to arbitrary real-world changes. Oracle tool annotations may also omit alternative successful routes, which could partly explain why the full catalog outperforms the oracle reference.
Questions for authors
  1. Can you report a representative task-level label audit, actual skill-reading/invocation rates, and results for models or task subsets that genuinely invoke skills?
  2. For fixed old-task cohorts, are results available under alternative release orders with paired uncertainty estimates? Persistence of the retention conclusions after separating catalog size from release order would materially strengthen benchmark validity.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 2mHv: Part 1

Thank you for your constructive review. We appreciate your recognition that "fixed models and reset deployment make interference from an expanding external capability catalog separately measurable", exposing "long-running deployment risks that static agent benchmarks miss." We also thank you for highlighting our "useful methodological contribution", particularly the benchmark's ability to evaluate "both new-task performance and retention under a common protocol." We address your questions below.

W1. “Keyword-matched skill labels do not establish necessity, sufficiency, or uniqueness, and some models rarely read skills. Annotation-level new-capability pressure may therefore not represent behavioral pressure to use new skills, limiting cross-axis interpretation and conclusions about skill evolution. Frequency-based release under one ordering further limits scope, without negating the value of the other axes.”

Thank you for raising this comment. We have clarified in Sections 3.1 and 3.3 that capability annotations are reference-based rather than guarantees of necessity, sufficiency, or uniqueness. Specifically, Section 3.3 of the original manuscript describes skill annotations as heuristic associations grounded in verifier-checked outcomes and we have revised it to use the term “reference-associated skill annotations.” Skill invocation is intentionally optional, as selecting and utilizing relevant skills is part of the agent behavior being evaluated. We have clarified in Section 4.1 (Figure 5) and Appendix J.1 that invocation rates depend on the underlying model and skill-access configuration, with different denominators across conditions. Consequently, our skill-axis results measure end-to-end skill discovery and utilization under library expansion, rather than isolating the effects of library growth alone.

Additional experiment on validity of our constructed skills. We evaluate SAS under four conditions: empty skills, retrievable stage-level oracle skills, shuffled direct injection, and matched task-specific oracle skill injection (Appendix J.1, Table J.2). Matched injection achieves the highest overall verifier score (65.6), outperforming empty skills (61.6), retrievable skills (64.3), and shuffled injection (61.5). The improvement is particularly notable on ITSM (54.3 vs. 47.1 with empty skills). These results suggest that directly providing task-matched skills can improve verifier-level performance, highlighting the importance of effective skill selection and utilization.

Table A. Verifier score (0–100) across SAS skill-access conditions, for EvoHarnessBench-EoG, with Codex-GPT-5

ConditionCSMHRITSMOverall
Empty skills73.9 ± 3.662.6 ± 4.547.1 ± 4.861.6 ± 2.5
Retrievable stage-level oracle skills76.7 ± 2.765.5 ± 1.949.6 ± 6.164.3 ± 2.6
Shuffled direct injection74.3 ± 2.261.9 ± 2.248.0 ± 4.861.5 ± 1.7
Matched direct injection78.5 ± 1.864.7 ± 2.954.3 ± 5.665.6 ± 1.1

W2. “Frequency ordering couples stage, catalog size, and task specialization. Old-cohort reevaluation and Figure 4 help, but one core-to-tail ordering and addition-only evolution do not establish robustness to arbitrary real-world changes. Oracle tool annotations may also omit alternative successful routes, which could partly explain why the full catalog outperforms the oracle reference.”

Thank you for the suggestion. We intentionally restrict EvoHarnessBench to additive expansion, keeping previously released capabilities and task requirements unchanged to isolate harness-induced forgetting from failures caused by capability modification or removal. Our frequency-based core-to-tail release schedule (Sec. 3.2) provides one controlled approximation of real-world capability growth, rather than establishing robustness to arbitrary orderings. We acknowledge this scope and identify non-monotonic evolution as future work (Sec. 5).

We further clarify that "oracle" refers to the task-specific tool annotations from the source benchmark, not an upper bound on agent performance. Alternative valid solution paths may therefore explain why broader tool exposure sometimes improves performance (Table 2). We have made this distinction explicit in Section 4 (Evaluation Setup), clarifying that the task-specific reference is an annotation-based capability-exposure control, not an oracle policy or a performance upper bound.

AuthorsOpenReview ↗

Response to Reviewer 2mHv: Part 2

Q1. “Can you report a representative task-level label audit, actual skill-reading/invocation rates, and results for models or task subsets that genuinely invoke skills?”

Thank you for this suggestion. Our skill annotations are deterministically matched to verifier-checked task outcomes (Sec. 3.3, Appendix E), identifying relevant procedural skills rather than guaranteeing necessity or uniqueness.

We reported actual skill invocation rates, measured through SKILL.md reads, in Fig. 5 and Appendix J.1, including 82% for GPT-5.5 Codex and 31% for task-specific GEPA. Appendix J.1 further reports invocation precision and recall against task-specific annotations. We have clarified the skill-invocation measurement and its interpretation in Section 4.1 and Appendix J.1, including the different denominators across skill-access conditions. We also report a controlled skill-injection experiment in Appendix J.1 (Table J.2), comparing empty skills, retrievable skills, shuffled injection, and matched task-specific injection.

To better illustrate, we provide a representative task-level annotation example below:

(EOG-CSM, Stage 3) Task task_20251210_183114_382_99ba2325_071a72ab asks the assistant to close case CS-0000944 and create a knowledge article documenting its resolution for future support use. The task is associated with two reference skill annotations: Linking Knowledge Articles and Customer (Portal User).

  • Verifier checks: The article is published, internally visible, assigned to the specified owner, and linked to case 944 as a resolution.
  • Matched skills: Linking Knowledge Articles is associated with knowledge.owner_id and knowledge.visibility='internal'; Customer (Portal User) is associated with the visibility constraint.
  • Observed behavior: GPT-5.5 Codex read both skills in all three runs and passed both verifiers each time (3/3 successes; verifier score: 100/100).

This example illustrates how reference skill annotations correspond to verifier-checked task conditions and confirms that the agent accessed the associated skills during successful executions. It does not establish that either skill was strictly necessary for success.

We have also incorporated this representative example into Appendix E.2 of the revised manuscript. We will assess annotation quality through the manual audit described in our response to Reviewer isCr.

Q2. “For fixed old-task cohorts, are results available under alternative release orders with paired uncertainty estimates? Persistence of the retention conclusions after separating catalog size from release order would materially strengthen benchmark validity.”

Thank you for this suggestion. Our benchmark uses a frequency-based core-to-tail release schedule (Sec. 3.2), while reevaluating fixed old-task cohorts under expanding harnesses to isolate harness-induced forgetting (Sec. 3.4). We intentionally adopt this controlled setting to keep previously released capabilities and task requirements unchanged. Our current results establish forgetting under the studied release schedule, and we have explicitly acknowledged this limitation in Section 3.2. As discussed in Section 5, We leave sensitivity to alternative release orders for future work.

Appreciate your response. I prefer to maintain my score.

Reviewer_68hgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EvoHarnessBench, a benchmark for evaluating LLM-based agents while their external harness (the tools, reusable skills, and specialist agents they are given) grows over time. It is built from EnterpriseOps-Gym and Agents' Last Exam and contains 17 multi-stage harness streams with 802 distinct tasks (1,510 axis-specific instances). The authors propose a controlled protocol for producing cumulative harness versions and evaluate agents in two settings. Deployment evaluation measures the effect of harness expansion alone. Self-evolving adaptation evaluation asks whether experience carries over from one stage to the next. Experiments cover single- and multi-agent systems and several adaptation methods. With model weights fixed, a larger harness can still break tasks the agent used to solve. Adaptation helps to different degrees depending on the environment and the capability type, and sometimes it hurts retention.

Critical strengths

1. Harness evolution is framed as a distinct evaluation problem. The paper separates two things that are usually mixed together: a shift in the task distribution, and a change in the tools, skills, or agents available to the system. The paper also identifies harness-induced forgetting: performance drops even though neither the model weights nor the task requirements have changed. This situation is common in deployed agent systems.

2. Benchmark design and analysis. Each multi-stage stream keeps the original verifier-based tasks and outcome checks. Because deployment evaluation and adaptation evaluation are run separately, one can tell how much of a change comes from the larger harness and how much from accumulated experience. The forward and backward transfer analyses show behavior that aggregate success rates would hide.

Critical weaknesses

1. The skill-evolution protocol does not say clearly what agents can actually access.
The paper mentions a retrievable skill pool, a provided skill library, hidden reference skills, and libraries produced by adaptation, in different places. It is not clear how the externally supplied skill harness changes across stages, or how it is separated from the skills the agent learns. Skill invocation rates are also low, so it is unclear whether the results measure adaptation to a growing library or just whether the agent uses skills at all.

2. Only additive harness expansion is studied.
Once a tool, skill, or agent is added, it is never removed or changed. This is a reasonable controlled setting. In practice, however, APIs change and tools get replaced or retired, and any of these can break procedures the agent learned earlier. The findings are really about additive expansion, and the paper should say more about how far they generalize.

Questions for authors

Q1. How do the hidden reference skills relate to (a) the retrievable library exposed to agents and (b) the skills produced by adaptation? A short table would help: the initial library, what is added at each stage, and where the main baselines get their accessible skills from.

Q2. Some configurations rarely or never call the skills they are given. To what extent do the results reflect robustness to a growing skill library, rather than whether the agent can find and use skills in the first place? Could the authors report skill invocation rates per harness stage, or explain how the current evaluation separates these two factors?

Q3. The benchmark uses monotonic capability expansion with a frequency-based release schedule. How sensitive are the main findings to this release strategy? And how far do the authors expect their conclusions to hold when harness components are modified, replaced, or removed?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 68hg: Part 1

We would like to thank Reviewer 68hg for the constructive review. We appreciate your recognition of "harness evolution as a distinct evaluation problem", the importance of "harness-induced forgetting", and the strengths of our "benchmark design and analysis", particularly how separating deployment and adaptation evaluations reveals behaviors that "aggregate success rates would hide." We address your questions below.

W1. “The skill-evolution protocol does not say clearly what agents can actually access. The paper mentions a retrievable skill pool, a provided skill library, hidden reference skills, and libraries produced by adaptation, in different places. It is not clear how the externally supplied skill harness changes across stages, or how it is separated from the skills the agent learns. Skill invocation rates are also low, so it is unclear whether the results measure adaptation to a growing library or just whether the agent uses skills at all.”

Thank you for raising this comment. We clarify that our protocol distinguishes three components:

  1. Hidden reference skills: Deterministically mined from source prompts to construct task annotations and the stage-wise release schedule (Sec. 3.3, Appendix E).
  2. Externally supplied skill harness: A cumulative library of retrievable skills that expands across stages, rather than being directly injected into agent prompts (Sec. 3.3). At each stage tt, agents can retrieve all skills released up to that stage as SKILL.md files, while unreleased skills and task-specific annotations remain inaccessible. Skills are provided for on-demand use, and invocation is intentionally not enforced.
  3. Self-evolving adaptation: Agent-learned artifacts (e.g., memories, generated skills, or prompts) maintained separately as adaptive state (ztz_t), which persists across stages only during adaptation (Sec. 3.4).

Importantly, we distinguish two separate skill-evolution settings:

  • Externally supplied skill evolution (Sec. 4, Evolving Skills, Table 3): Agents receive an expanding library of pre-defined, retrievable skills. This evaluates their ability to utilize a growing external skill harness.
  • Agent-driven skill learning (Sec. 4, Evolving skills, Observation 3; Appendix J.2): No pre-existing ground-truth skill library is provided. Instead, skills are learned from past execution trajectories and feedback, then incorporated into a retrievable skill library that evolves across stages. This evaluates agents' ability to acquire and accumulate reusable skills, rather than utilize externally supplied ones.

Regarding low invocation rates, we analyze this in Section 4.1 (Figure 5) and Appendix J.1. Task-specific GEPA invokes 31% of its offered stage-level skill library, while GPT-5.5 Codex invokes 82% of its offered task-specific skills; these percentages have different denominators and are not directly comparable. These findings highlight that the systems' skill retrieval and utilization are important bottlenecks, dependent on the underlying model and adaptation strategy. Section 4.1 (Figure 4) further examines the impact of skill-harness exposure during adaptation while holding the evaluation harness fixed. We have clarified the skill-access protocol in Section 3.3, the distinction between the external harness and persistent adaptive state in Section 3.4, and the interpretation of skill-invocation measurements in Section 4.1 and Appendix J.1.

W2 & Q3. “Only additive harness expansion is studied. Once a tool, skill, or agent is added, it is never removed or changed. This is a reasonable controlled setting. In practice, however, APIs change and tools get replaced or retired, and any of these can break procedures the agent learned earlier. The findings are really about additive expansion, and the paper should say more about how far they generalize.” “The benchmark uses monotonic capability expansion with a frequency-based release schedule. How sensitive are the main findings to this release strategy? And how far do the authors expect their conclusions to hold when harness components are modified, replaced, or removed?”

Thank you for raising this point. We focus on monotonic harness expansion to isolate the effects of adding new capabilities without confounding modifications or removals (Sec. 3.2). Our frequency-based release schedule provides a controlled approximation of core-to-long-tail capability growth (Fig. 2). While sensitivity to alternative schedules remains unexplored, our findings establish that additive expansion can induce forgetting and inconsistent adaptation. We have clarified the scope of our frequency-based release schedule in Section 3.2 and explicitly discuss capability modification, replacement, and retirement as future directions in Section 5 (Conclusion).

AuthorsOpenReview ↗

Response to Reviewer 68hg: Part 2

Q1. “How do the hidden reference skills relate to (a) the retrievable library exposed to agents and (b) the skills produced by adaptation? A short table would help: the initial library, what is added at each stage, and where the main baselines get their accessible skills from.”

Thank you for this suggestion. Sec. 3.3 and Appendix E describe reference-skill construction, while Sec. 3.4 distinguishes the external harness from the adaptive state (ztz_t). We clarify the access protocol below:

Component / SettingAccessible Skills
Reference skill annotationsDerived from mined procedural skills for task matching and release scheduling; task-to-skill labels remain hidden, while skill content becomes retrievable upon release.
Evolving skills (Sec. 4)Initial library (H1H_1) contains first-stage skills; subsequent stages cumulatively add newly released reference skills
Main adaptation baselinesAccess the same cumulative external library (HtH_t), plus method-specific persistent artifacts (ztz_t)
Skill learning (Appendix J.2)No pre-supplied ground-truth skill library; generated skills from trajectories and feedback populate the retrievable library

Importantly, external skill evolution and agent-driven skill learning are distinct settings: the former evaluates adaptation to an expanding predefined library, while the latter evaluates learning and accumulating skills from experience. Task-specific reference conditions expose only the annotated skills for each task.

We have clarified the distinction between hidden reference annotations, the externally supplied cumulative skill library, and agent-learned artifacts in Section 3.3, with the separation between the external harness and adaptive state formalized in Section 3.4. Stage-wise skill additions and cumulative library sizes are documented in Appendix C.2.

Q2. “Some configurations rarely or never call the skills they are given. To what extent do the results reflect robustness to a growing skill library, rather than whether the agent can find and use skills in the first place? Could the authors report skill invocation rates per harness stage, or explain how the current evaluation separates these two factors?”

Thank you for raising this distinction. Skill retrieval and invocation are supported but intentionally not enforced, as deciding when and which skills to use is part of the agent behavior being evaluated. As reported in Section 4.1 (Figure 5) and Appendix J.1, GPT-5.5 Codex invokes 82% of its offered task-specific skills, while task-specific GEPA invokes 31% of its offered stage-level skill library. Because these conditions have different denominators, their invocation percentages are not directly comparable. Thus, low invocation is itself an important performance indicator for the systems, rather than an interface limitation of our benchmark.

Our task-specific versus cumulative comparisons in Section 4 (Table 3) examine library expansion under the same retrieval mechanism, while Section 4.1 (Figure 4) examines the effect of skill-harness exposure during adaptation. We additionally report a controlled skill-injection experiment in Appendix J.1 (Table J.2) to distinguish the availability of task-relevant skill instructions from the agent's ability to retrieve them.

Additional experiment on validity of our constructed skills. We evaluate SAS under four conditions: empty skills, retrievable stage-level oracle skills, shuffled direct injection, and matched task-specific oracle skill injection (Appendix J.1, Table J.2). Matched injection achieves the highest overall verifier score (65.6), outperforming empty skills (61.6), retrievable skills (64.3), and shuffled injection (61.5). The improvement is particularly notable on ITSM (54.3 vs. 47.1 with empty skills). These results suggest that directly providing task-matched skills can improve verifier-level performance, highlighting the importance of effective skill selection and utilization.

Table A. Verifier score (0–100) across SAS skill-access conditions for EvoHarnessBench-EoG with Codex-GPT5.

ConditionCSMHRITSMOverall
Empty skills73.9 ± 3.662.6 ± 4.547.1 ± 4.861.6 ± 2.5
Retrievable stage-level oracle skills76.7 ± 2.765.5 ± 1.949.6 ± 6.164.3 ± 2.6
Shuffled direct injection74.3 ± 2.261.9 ± 2.248.0 ± 4.861.5 ± 1.7
Matched direct injection78.5 ± 1.864.7 ± 2.954.3 ± 5.665.6 ± 1.1

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.