acceptodds
Under review as a conference paper at ICLR 2027

SkillHorizon: Constructing Long-Horizon Agent Tasks through Skill Composition

Abstract

Long-horizon agents must complete workflows in which later activities depend on earlier outputs and changes to shared state. A key challenge for evaluation is to expand task requirements and their dependencies while keeping tasks executable and outcomes verifiable. We introduce SkillHorizon, a benchmark that constructs long-horizon tasks through skill composition in seven business environments. Its generators use domain-specific rules to turn reusable skill procedures into concrete business activities. These activities are connected through input–output dependencies, shared resources, and state updates, allowing workload and coordination demands to grow together. Successful reference executions validate task feasibility and measure interaction workload, while programmatic evaluators assess local completion and whole-task success. We evaluate five language models on 1,050 tasks with reference interaction workloads ranging from 64K to 1M tokens. Across these scales, mean local-goal completion falls from 90.4% to 46.8%, as completed work fails to keep pace with growing requirements. At larger workloads, agents more often submit unsuccessful final answers with substantial work unfinished and tool-call allowance remaining. Trace analysis also identifies cases where scripts execute successfully but their outputs fail downstream business checks. SkillHorizon provides a configurable testbed for studying sustained progress, dependency coordination, and complete task delivery in growing workflows.

Primary area:
datasets and benchmarks
Keywords:
Long-horizon, Agent skills, Benchmark, Tool use, Agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_87r5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SkillHorizon is an evaluation benchmark that constructs long-term tasks by combining skills, covering seven business environments. Its generator utilizes domain-specific rules to instantiate reusable skill processes into specific business activities, and connects these activities through input-output dependencies, shared resources, and state updates, enabling the task workload and coordination requirements to grow together. Subsequently, the system verifies the feasibility of the task through successful reference execution and measures the interaction workload, and then uses a programmatic evaluator to check the completion of local goals and the overall success of the task, thereby supporting the evaluation of the continuous execution, coordination dependence, and complete delivery capabilities in different-scale workflows.

Critical strengths
  1. This paper provides a testing platform by using a configurable task generator to construct workflows of different scales, thereby enabling the study of performance changes in long-term tasks.

  2. This standard covers seven business environments and clearly incorporates input-output dependencies, shared resources, version validity, and status updates into the requirements of the work. These mechanisms ensure that agents can correctly reuse previous results, adjust resources, and modify affected tasks according to changes, thereby enhancing the comprehensiveness of the assessment of complex business processes.

  3. The experimental scale of this paper is large, evaluating the performance of five models on 1050 tasks, and conducting additional analyses of various intermediate scenarios, including early termination, sub-business check failures, error recovery, and the use of cross-context dependencies.

Critical weaknesses
  1. The functional boundaries between construction components are insufficiently specified. It remains unclear which module requirements, dependencies, and completion conditions are inherited from skill specifications and which are separately defined by generators. This obscures how reusable skills are concretely translated into executable tasks.

  2. Workflow diversity may be limited by the predefined task families.The 1,050 instances only span 20 families across seven environments, and several families represent constraint variants or combinations within the same underlying business workflow. Therefore, the family count does not directly reflect the number of distinct workflows covered, leaving the breadth of task diversity less clear than the benchmark’s instance count suggests.

  3. There is a lack of comparison with the direct-related skill-driven task construction methods.

    SkillNet-Gym: https://arxiv.org/abs/2603.04448v2. This method combines skill construction dependency graphs and synthesizes companion reference solutions and validators for the tasks.

    SkillFlow: https://arxiv.org/abs/2604.17308v1 Constructing benchmarks by combining skills and workflow structures. The authors need to clarify what new designs SkillHorizon has and how these designs support the long-term business processes it emphasizes.

Questions for authors
  1. How are skill specifications translated into task modules and dependencies, and which requirements are separately defined by domain-specific generators?

  2. What guided the selection of the 20 task families, and what breadth of distinct business workflows are they intended to cover, given that some families are constraint variants of the same workflow?

  3. What are the key methodological differences between SkillHorizon and existing skill-driven benchmark construction approaches, particularly SkillNet-Gym and SkillFlow?

  4. All models are evaluated under a single framework with a fixed text-JSON action protocol and summary-based context management. Since models may differ in how well they adapt to these choices, how sensitive are the reported performance gaps and model rankings to the execution framework?

4 comments
AuthorsOpenReview ↗

Response(1/2)

We thank the reviewer for the careful reading and helpful comments. We respond to each point below.

Q1 & W1. How are skill specifications translated into task modules and dependencies, and which requirements are separately defined by domain-specific generators?

A skill defines a reusable procedure. The domain generator specifies where that procedure applies in a particular task: which objects are involved, what outputs are required, and which later activities need those outputs. A task module is therefore a concrete piece of required work, such as obtaining approval for a particular document. The generator defines the dependencies; it does not infer them from the skill text. During execution, the environment checks whether the artifacts the agent produces meet the specified conditions.

In DocOps, for example, the Coordinate Document Review skill tells the agent to submit the current draft to the designated reviewers, address their objections, and request a fresh review after an edit. The generator supplies the document, its designated reviewers, and its publication requirements. What connects review to publication is a version-specific approval: the environment permits publication only if the approval applies to the version being published. If the agent edits an approved draft, the earlier review still took place, but its approval no longer authorizes publication of the revised document.

Q2 & W2. What guided the selection of the 20 task families, and what breadth of distinct business workflows are they intended to cover, given that some families are constraint variants of the same workflow?

We chose the families to cover different decisions agents face when one activity depends on the outcome of another. The seven environments span financial reconciliation, field service commissioning, campaign delivery, passenger and baggage coordination, research evidence maintenance, document publication, and production dispatch. Within these domains, we included both distinct business operations and variants of the same workflow. The 20 families comprise three each in SocialOps, LogisticsOps, ResearchOps, DocOps, and FactoryOps, three in FinanceOps, and two in FieldOps.

FactoryOps illustrates why we included variants. The delivery objective stays the same, but equipment availability determines when an operation can run, while selective rework requires the agent to decide which quarantined outputs need rework. The combined family requires scheduling that rework within the available equipment windows. In FinanceOps, by contrast, matching incoming payments to invoices and updating a financial release after a source correction are different business operations. The 1,050 evaluated instances exercise these 20 designs across five workload bands. This lets us compare agents across business workflows and, within a given workflow, examine how they respond when the decisions required for delivery change.

AuthorsOpenReview ↗

Response(2/2)

Q3 & W3. How does SkillHorizon differ from SkillNet-Gym and SkillFlow?

SkillHorizon studies whether an agent can sustain correct skill use as a single task grows. The agent receives a skill library; the generator expands the work it must deliver and the dependencies among those activities. An agent may execute a procedure correctly at one stage but use its output incorrectly at the next, or fail to revisit an earlier result after its inputs change. We evaluate these consequences in the business environment: which goals remain satisfied, and does the final delivery pass acceptance?

SkillNet-Gym varies which skills a task brings together and how they connect. It constructs a graph by matching the output scenario of one skill to the input scenario of another, then samples skill combinations from that graph to generate executable tasks. SkillHorizon can keep those procedures fixed and instead vary the work that depends on their outputs within a single task. In DocOps, for example, the review and publication procedures stay the same as the portfolio grows. When an upstream contract changes, the agent must determine which published pages relied on it, revise those pages, obtain approval for the new versions, and leave unaffected pages intact. Knowing that review precedes publication does not tell the agent which earlier approvals are still valid. By checking the affected pages and their versions, we can measure how well the agent maintains prior work as new obligations accumulate.

SkillFlow studies how agents discover, refine, and reuse skills across successive tasks in a workflow family. What carries forward is the agent’s evolving skill library. In SkillHorizon, the library stays fixed while work accumulates within one task. In FinanceOps, for example, each new stage adds records to reconcile, and its closing record must incorporate the preceding stage’s result. SkillFlow thus tests whether a learned procedure transfers to the next task; SkillHorizon tests whether an agent using familiar procedures carries the right business result into the next stage.

Q4. All models are evaluated under a single framework with a fixed text-JSON action protocol and summary-based context management. Since models may differ in how well they adapt to these choices, how sensitive are the reported performance gaps and model rankings to the execution framework?

GPT-5.6 Sol’s task success rose from 18% to 84% when we changed the execution configuration. We evaluated it on the same 50 FinanceOps tasks, ten per workload band. Both configurations requested medium reasoning and used the same per-task tool allowances and business acceptance checks.

Execution configurationPassedFailed final submissionsTool allowance exhausted
Harbor + Inspect9383
Codex with direct MCP4280

The largest change was in how runs ended: unsuccessful final submissions fell from 38 to 8, while runs exhausting their tool allowance fell from three to zero. The 66-point gain shows that the execution configuration matters substantially for GPT on these tasks. The action interface, provider route, and context management changed together, so the comparison does not isolate any one of them. Only GPT was evaluated under both configurations; whether the five-model gaps and ranking persist across harnesses remains untested.

Early stopping was directly observable in the main evaluation. At 1M, 183 unsuccessful GPT final submissions had records of both progress and remaining allowance. In 88 of them, GPT submitted after completing less than a quarter of the required goals while still having more than half its tool allowance left. Terminal-Bench, Figure 8 also identifies premature termination in approximately 14% of its sampled failed GPT-5.2 runs under Terminus-2. Independent evidence for harness sensitivity comes from Baselines Before Architecture, Table V: GPT-5 solved 70 of 104 tasks in each of two Codex runs, compared with 57 and 54 in the corresponding OpenCode runs.

Thank you for the clarifications and additional experiments. I appreciate the new evidence, though I still have reservations about how strongly it supports the paper’s central claims, like the 20 task families still appear to cover a relatively narrow range of distinct workflows. I will maintain my original score.

Thank you for considering the additional evidence. We would like to clarify what the 20 task families represent. They are the families we selected for the 1,050-task evaluation, not a limit on what the generators can construct. We designed them around real-world business procedures and the constraints that arise together in practice. Some families require different deliverables: in FinanceOps, reconciling payments against invoices is a different job from updating a financial release after a source correction. Others deliberately retain the same deliverable so that we can study how the decisions required to produce it change.

FactoryOps illustrates why the latter families matter. In selective rework, inspection determines which units need another production step and which can proceed. In equipment scheduling, maintenance and occupancy determine when a step can run. In the combined family, the agent only learns which additional operations to schedule after seeing the inspection results, and it must fit that work into the existing equipment calendar. The combination introduces a decision that neither condition poses on its own: how should the production plan change once the defective units are known? This is why we evaluate the mechanisms both separately and together.

The released generators also implement scenarios outside the experimental selection. For example, beyond the two FieldOps scenarios evaluated in the paper, the generator supports a tight-window scenario in which the agent must find both a qualified crew and a maintenance window long enough for the repair and its test. We will revise the manuscript to distinguish the business workflows covered, the mechanisms we compose within them, and the families on which we report results.

Reviewer_ccuSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes constructing long-horizon tasks through skill composition and evaluates rollouts from five different models to verify whether the synthesized tasks are indeed long-horizon. However, this idea has already been explored in general domains, and the paper does not introduce a new methodological contribution beyond applying it to business scenarios. Moreover, the paper lacks training-based validation, making it unclear whether the long rollouts reflect inherently long-horizon task requirements or problems with the synthesized tasks. Without training-based validation, the quality of these long-horizon tasks cannot be guaranteed.

Critical strengths
  • The writing is clear.
  • The paper evaluates multiple models and includes extensive experiments.
  • The figures provide effective visualizations.
Critical weaknesses
  1. Lack of novelty. The idea of constructing long-horizon tasks through skill composition has already been explored in several earlier studies, yet the paper cites none of the following relevant works:

    [1] Toward Scalable Terminal Task Synthesis via Skill Graphs, Zhiyuan Fan et al.

    [2] Terminal-World: Scaling Terminal-Agent Environments via Agent Skills, Zihao Cheng et al.

    [3] FACET: Preserving Source Intent and Executable State in Terminal Task Synthesis, Kou Shi et al.

    [4] Skill2Env: Capability-Oriented Environment Synthesis from Skills for General Agents, Weiyi Xu et al.

    [5] Environment Evolution for Terminal Agents, Zhiyuan Fan et al.

    [6] Recursive Synthesis for Long-Horizon Terminal Tasks, Zhongzhi Li et al.

  2. Lack of training-based validation. The experiments are inference-only and do not include training. Consequently, it remains unclear whether the observed long rollouts arise from genuinely long-horizon task requirements or from quality issues in the synthesized tasks. Contemporary papers on long-horizon task synthesis include training-based validation.

Questions for authors

At a minimum, could the authors provide the necessary training experiments to demonstrate the effectiveness of the synthesized long-horizon tasks?

7 comments
AuthorsOpenReview ↗

Response(1/2)

We thank the reviewer for the careful reading and the references. We first clarify the contribution of SkillHorizon and then address the cited work.

W1. Lack of novelty.

Constructing seven executable business environments is itself a contribution of SkillHorizon. We built persistent worlds grounded in real-world resources and business procedures. From the same set of skills, their generators can create tasks that differ in the amount of work required, the dependencies among activities, and the updates an agent must handle. Each generated instance is checked by executing a reference solution against its business acceptance criteria.

These environments let us ask a question about long-horizon skill use: when a skill produces the expected output, can the agent still use that output correctly at a later stage? A downstream activity may need to combine several earlier results or distinguish a current artifact from one superseded by an update. The environment checks the artifact the agent actually uses against the requirements at that point. We can therefore identify cases where a local computation succeeded but its output was rejected downstream, and measure how much of the growing workflow the agent ultimately delivered.

We thank the reviewer for pointing us to Skill2Env. Its first arXiv submission was on September 27, 2026, after the September 25 ICLR full-paper deadline. Its absence from the original manuscript therefore does not support the claim that we overlooked prior work. The two studies also use generated environments for different purposes. Skill2Env starts from a skill and a target capability, synthesizes tasks and environments to exercise that capability, and evaluates whether training on the resulting trajectories improves agents on external benchmarks. SkillHorizon keeps the skill library fixed and expands the work required within a single task. We evaluate whether agents can continue to use familiar skills correctly as more activities depend on the results of their earlier work.

Environment Evolution for Terminal Agents was first submitted to arXiv on September 3, 2026, just 22 days before the ICLR full-paper deadline. It evolves a seed terminal environment into a succession of harder environments for reinforcement learning. SkillHorizon expands the workflow within a persistent business environment. We build seven domain environments with reusable tools, skills, and business rules. Within each environment, the generator can keep those procedures fixed while adding activities, making later work depend on earlier outputs, and introducing updates that require completed work to be revisited. This lets us ask whether an agent that can apply a skill to one assignment can continue to use it correctly as more work and dependencies accumulate under the same rules.

Recursive Synthesis was first submitted to arXiv on August 5, 2026. It addresses a different problem: how to turn a verified terminal task into a harder training task. Its generator rewrites the previous task and updates its reference solution and tests. For example, a task that initially asks an agent to produce JSON-diff reports is extended to require configuration repair and regression testing. The unit of synthesis is an existing task; the object of our evaluation is the agent’s use of supplied skills. In SkillHorizon, we keep the skill procedures fixed and expand the business activities that require them. This lets us test whether an agent can repeatedly apply familiar procedures, combine their outputs, and revisit earlier work when those outputs become stale. Recursive task rewriting is a precedent for extending task horizons, but it does not establish that this question about sustained skill use has already been studied.

FACET, first submitted on August 19, uses skills as source material for generating terminal training tasks. It reconstructs a scenario from related skills, preserves their intended goals and dependencies, and grounds the task instructions, reference solution, and verifier in the same executable environment. FACET asks whether a synthesized task preserves the dependencies specified by its source skills. SkillHorizon asks whether the agent can maintain those dependencies in practice as more activities rely on the outputs it produces.

AuthorsOpenReview ↗

Response(2/2)

W1 (continued). Comparison with SkillSynth and Terminal-World.

SkillSynth addresses a training-data problem: how can a generator control which combinations of command-line skills appear in training trajectories? It samples a path through a skill graph and builds an executable task around that path. SkillHorizon addresses an evaluation problem: given the procedures for doing the work, can an agent continue to apply them correctly as a single assignment grows? We give the skill library to the evaluated agent. The generator specifies the required business outcomes and the dependencies among them; success is judged by the state the agent leaves behind, not by whether it follows a sampled skill path. Keeping the procedures fixed lets us test what happens when the agent must use them repeatedly, carry its own outputs across stages, and revise earlier work after an upstream change. SkillSynth establishes a precedent for generating tasks from skill combinations. The distinct contribution of SkillHorizon is making sustained use of those skills the object of evaluation.

Terminal-World builds an executable environment for each generated task; SkillHorizon generates different tasks within the same business environment. Terminal-World uses skills to synthesize terminal tasks and collect teacher trajectories, then evaluates how well agents trained on those trajectories generalize to other benchmarks. In SkillHorizon, the evaluated agent itself uses the skill library. We can hold the available skills and business rules fixed while increasing both the work to be delivered and the dependencies among activities. This lets us measure how much of the required work the agent completes as the workflow grows, and whether the outputs it produces with one skill remain valid when a later activity needs them.

W2. Does a longer trajectory reflect more required work or less efficient execution?

The 1M label measures the workload of a fixed reference policy; it is not a lower bound on the tokens needed to solve the task. We assign each task to a band before running the evaluated agents, using the token count of public tool calls and their results in a successful reference execution. Repeated queries by an evaluated agent therefore cannot move a task into a longer band. The task specifications independently show what additional work is required. In DocOps, 64K tasks require 6–10 pages at configured dependency depths of 2–4, whereas 1M tasks require 52–105 pages at depths of 18–35. An agent may find a shorter execution, but it must still deliver those pages and satisfy the dependencies among them.

Q1. Have you evaluated whether training on the generated tasks improves agent performance?

We have not trained agents on SkillHorizon tasks in this study. One potential contribution of the benchmark is that it lets us test whether training improves an agent’s ability to use familiar skills over longer workflows. We plan to train an agent on generated workflows, then compare the trained and original models on the same held-out tasks. By keeping the skill library and business rules fixed while extending the dependencies among activities, we can test whether training helps the agent both complete more local goals and carry those gains through to successful delivery of the whole task.

I thank the authors for their detailed response. However, I maintain my Strong Reject recommendation for the following reasons:

  1. The authors have not directly addressed my concern that the paper’s core idea—constructing long-horizon tasks through skill composition—has already been extensively studied. Although this is the paper’s central claim, the response sidesteps a direct comparison on this point and instead emphasizes other differences from prior work. The authors reframe the contributions and claims of prior papers to avoid directly discussing the overlap in the core idea.

  2. The authors have still not addressed my concern that longer execution trajectories may simply reflect poor task quality rather than an intrinsic need for long-horizon execution. Their response provides no additional evidence to clarify this distinction and instead reiterates what was already presented.

  3. Related to point 2, my central concern is that, without training experiments, the effectiveness of the synthesis method remains unclear: does it produce tasks that genuinely require long-horizon execution, or merely low-quality tasks that induce longer trajectories? Training can help reveal this distinction. Many tasks may require numerous execution steps during evaluation simply because of poor task quality, while training on such superficially long-horizon tasks provides little benefit on tasks that genuinely require long-horizon execution.

Reviewer_MJ6kOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SkillHorizon, a benchmark for evaluating long-horizon agents on compositional business workflows across seven environments. The benchmark supports varying task scale and dependency structure, with reference executions for validation and programmatic checks for task completion. Experiments on five models show that performance drops substantially as workload increases. The paper also examines premature termination and failures in downstream use of intermediate results.

Critical strengths

The paper addresses an important problem, and the benchmark is reasonably comprehensive. The executable environments and programmatic evaluation are useful, and the trace-level analyses provide insights beyond aggregate success rates. The released benchmark could be valuable for future work on long-horizon agents.

Critical weaknesses

My main concern is that workload and dependency complexity are not sufficiently disentangled. As task scale increases, several factors change simultaneously, so it is difficult to tell whether the observed degradation is due to more work, more complex dependencies, or both.

I am also not fully convinced by the claimed gap between local completion and global delivery. Appendix F.3 reports that 2,424 out of 2,435 executions with all local goals completed also achieve task success. This suggests that genuine global integration failures are relatively uncommon in the current evaluation. The paper would benefit from a clearer distinction between unfinished local work and failures that occur despite all local goals being satisfied.

Questions for authors

Can dependency complexity be varied while keeping workload approximately constant?

What causes the 11 failures where all local goals are completed but the overall task still fails?

How does skill-based construction compare with directly composing task templates or business operations?

Could simple progress tracking or final-state verification mitigate the observed premature termination?

7 comments
AuthorsOpenReview ↗

Response(1/2)

We thank the reviewer for highlighting the value of executable environments and trace-level analysis, and for the questions below.

Q1 & W1. Can dependency complexity be varied while keeping workload approximately constant?

Yes. We generated two FinanceOps tasks from the same family and seed. Both require reconciling 64 work packages. One distributes them across two stages of 32 packages; the other distributes them across eight stages of eight. In both tasks, we verified that each stage after the first uses the preceding stage’s closing record. The first stage’s result therefore reaches the final close through one handoff in the first task and seven successive handoffs in the second. Both reference executions passed final acceptance, and their lengths differed by only 0.8%: 58,728 and 59,202 tokens, respectively.

We paired tasks within the same environment, task family, and workload band, without using model scores to select pairs. Within each pair, reference business-call counts differed by at most 10%. We weighted environments and models equally in both comparisons. Across 107 pairs in five environments, average task success was lower on tasks with deeper dependency graphs. Across 130 pairs in four environments, average task success was likewise lower on tasks where a larger share of activities required multiple upstream outputs.

Q2 & W2. What causes the 11 failures where all local goals are completed but the overall task still fails?

Of the 2,435 executions that completed every local goal, 2,424 passed final acceptance (99.55%). We examined the remaining 11. In FieldOps, the local goals count restored devices; final acceptance also requires operational services and a final acceptance record. In FinanceOps, the local goals count reconciled work packages; the published release must also use valid closing records.

EnvironmentRunsLocal goals completedWhy final acceptance failed
FieldOps, 1M7All devices restoredAll required services were operational, but the final acceptance record was missing.
FieldOps, 1M2All devices restoredIn each run, only 383 of 384 required services were operational, and the final acceptance record was missing.
FinanceOps, 64K (Grok)1All 24 work packages reconciledThe agent exhausted its tool allowance before publishing the release.
FinanceOps, 64K (Qwen)1All 64 work packages reconciledThe agent published the release, but its stage 4 closing record referenced a superseded stage 3 record.

The paper’s 80% completion threshold includes a different, more common case: the agent made substantial progress but still had local work left to do. Among runs that reached this threshold and failed final acceptance, 66 of 68 at 64K and 113 of 122 at 1M still had unfinished local goals.

AuthorsOpenReview ↗

Response(2/2)

Q3. How does skill-based construction compare with directly composing task templates or business operations?

We make skills explicit because long-horizon work often requires an agent to combine several reusable procedures to complete a single assignment. A sufficiently detailed template could encode the same assignment. The template specifies what must be delivered, and business operations let the agent inspect or change records. Skills provide the procedures the agent can consult and apply across assignments. In DocOps, for example, an upstream amendment may require several published pages to be revised. The agent can use the same procedures for tracing dependencies, coordinating review, and publishing approved versions across many such amendments. What changes each time is which pages are affected and which versions need fresh approval. The agent must determine both from the current records and leave unaffected pages unchanged.

Making the procedures explicit lets us vary the task and the guidance independently. We can keep the skills fixed and change which pages depend on the amended document, testing whether the agent can apply familiar procedures to a new set of dependencies. Conversely, we can keep the documents, tools, and acceptance checks fixed and revise one shared tracing instruction for every task that uses it. This lets us study a practical question in long-horizon work: can an agent use several familiar procedures together to deliver the right result when the records and dependencies change?

Q4. Could simple progress tracking or final-state verification mitigate the observed premature termination?

Our harness already includes a second confirmation, as in Terminus-2. Agents are instructed to check the task requirements against public results before submitting. After the first final answer, they can resume work or confirm submission. In one 1M DocOps task, GPT had completed 11 of 81 required pages and used 172 of its 1,715 permitted tool calls. In its own final answer, it explicitly acknowledged that most of the 81 documents still needed work. When prompted to confirm, it submitted the same answer without another tool call. No context compaction had occurred.

In this case, the agent knew the task was unfinished and still chose to stop. An explicit progress check would ask a different question: which pages still lack an approved publication or a corresponding tracker update? A checklist derived from public records could identify those pages and give the agent a concrete next action. We have not tested whether such a check reduces early stopping.

Reviewer_MJ6kOpenReview ↗

Reply from Reviewer

Thank you for the detailed and thoughtful responses. I appreciate the additional matched-pair analysis, which helps address my concern about disentangling workload from dependency complexity. The clarification of the 11 global failures is also helpful.

However, I still have some reservations. For Q1, it would be useful to report the actual performance differences and uncertainty estimates from the matched comparisons. For Q2, the results seem to confirm that failures despite complete local-goal satisfaction are relatively rare, and I would encourage the authors to make this distinction clearer.

My main remaining concern is the novelty of skill-based task construction relative to conventional DAG-based workflow composition. While the explanation of reusable procedures is reasonable, the response does not yet demonstrate a clear empirical advantage or a distinctive evaluation capability.

Overall, the rebuttal has improved my understanding of the work and addressed some of my concerns. However, I would prefer to keep my current score unchanged.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Dear Area Chair and Reviewers,

Thank you for your careful reading and constructive feedback. SkillHorizon evaluates whether agents can keep using familiar skills correctly as a business workflow grows. We built seven executable business environments grounded in real-world domain materials and procedures, with nearly 90,000 database records describing business objects, relationships, and events. Within each environment, we can hold the skill procedures fixed while adding work and changing which later activities depend on earlier outputs. The environment checks the results the agent actually delivers, both at each local goal and across the workflow as a whole.

The reviewers’ questions led us to clarify how tasks are constructed and to examine three aspects of execution more closely.

  • What stays fixed as a task grows. Reviewers 87r5 and ccuS asked how skills enter the benchmark and how our construction differs from prior work. In SkillHorizon, the evaluated agent receives the skill library. In DocOps, the same review procedure applies whether an amendment affects one page or many. The generator determines which pages depend on the amended document; the agent must identify those pages, revise them, and obtain approval for the versions it publishes. An approval for an unchanged page remains valid. This lets us vary the work that depends on a skill while holding the procedure itself fixed.

  • Dependency depth at comparable workloads. Following Reviewer MJ6k’s question, we generated two FinanceOps tasks from the same family and seed. Both require reconciling 64 work packages. Arranging them in two stages of 32 requires one handoff between stages; arranging them in eight stages of eight requires seven successive handoffs. Both reference executions passed, using 58,728 and 59,202 tokens, respectively. We also paired tasks within the same environment, family, and workload band, matching reference business-call counts to within 10%. Average task success was lower for the deeper graphs across 107 pairs and for tasks with a higher share of multi-input activities across 130 pairs.

  • Local completion and final delivery. Of the 2,435 executions that completed every local goal, 2,424 passed final acceptance (Appendix E.2). Of the remaining 11, nine were FieldOps runs in which every device was restored but the required final acceptance record was missing; in two of those nine, one required service was also still offline. In the two FinanceOps runs, the agents reconciled every work package but did not publish a valid release.

  • Early stopping and the execution harness. Like Terminus-2, our harness asks agents to confirm their first final answer or resume work. In one 1M DocOps run, GPT acknowledged that it had completed only 11 of 81 pages, yet confirmed submission without another tool call. It had used just 172 of its 1,715 permitted calls. To examine harness sensitivity, we compared GPT-5.6 Sol on the same 50 FinanceOps tasks with identical tool allowances and acceptance checks. GPT passed 9 tasks under Harbor + Inspect and 42 under Codex with direct MCP; final submissions that failed acceptance fell from 38 to 8 (Appendix E.5).

We thank the reviewers for prompting us to compare tasks at similar workloads and to trace individual failures back to the business requirements that remained unmet.

Best regards,
The Authors of SkillHorizon

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.