- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SkillHorizon is an evaluation benchmark that constructs long-term tasks by combining skills, covering seven business environments. Its generator utilizes domain-specific rules to instantiate reusable skill processes into specific business activities, and connects these activities through input-output dependencies, shared resources, and state updates, enabling the task workload and coordination requirements to grow together. Subsequently, the system verifies the feasibility of the task through successful reference execution and measures the interaction workload, and then uses a programmatic evaluator to check the completion of local goals and the overall success of the task, thereby supporting the evaluation of the continuous execution, coordination dependence, and complete delivery capabilities in different-scale workflows.
Critical strengths
-
This paper provides a testing platform by using a configurable task generator to construct workflows of different scales, thereby enabling the study of performance changes in long-term tasks.
-
This standard covers seven business environments and clearly incorporates input-output dependencies, shared resources, version validity, and status updates into the requirements of the work. These mechanisms ensure that agents can correctly reuse previous results, adjust resources, and modify affected tasks according to changes, thereby enhancing the comprehensiveness of the assessment of complex business processes.
-
The experimental scale of this paper is large, evaluating the performance of five models on 1050 tasks, and conducting additional analyses of various intermediate scenarios, including early termination, sub-business check failures, error recovery, and the use of cross-context dependencies.
Critical weaknesses
-
The functional boundaries between construction components are insufficiently specified. It remains unclear which module requirements, dependencies, and completion conditions are inherited from skill specifications and which are separately defined by generators. This obscures how reusable skills are concretely translated into executable tasks.
-
Workflow diversity may be limited by the predefined task families.The 1,050 instances only span 20 families across seven environments, and several families represent constraint variants or combinations within the same underlying business workflow. Therefore, the family count does not directly reflect the number of distinct workflows covered, leaving the breadth of task diversity less clear than the benchmark’s instance count suggests.
-
There is a lack of comparison with the direct-related skill-driven task construction methods.
SkillNet-Gym: https://arxiv.org/abs/2603.04448v2. This method combines skill construction dependency graphs and synthesizes companion reference solutions and validators for the tasks.
SkillFlow: https://arxiv.org/abs/2604.17308v1 Constructing benchmarks by combining skills and workflow structures. The authors need to clarify what new designs SkillHorizon has and how these designs support the long-term business processes it emphasizes.
Questions for authors
-
How are skill specifications translated into task modules and dependencies, and which requirements are separately defined by domain-specific generators?
-
What guided the selection of the 20 task families, and what breadth of distinct business workflows are they intended to cover, given that some families are constraint variants of the same workflow?
-
What are the key methodological differences between SkillHorizon and existing skill-driven benchmark construction approaches, particularly SkillNet-Gym and SkillFlow?
-
All models are evaluated under a single framework with a fixed text-JSON action protocol and summary-based context management. Since models may differ in how well they adapt to these choices, how sensitive are the reported performance gaps and model rankings to the execution framework?
4 comments
Response(2/2)
Q3 & W3. How does SkillHorizon differ from SkillNet-Gym and SkillFlow?
SkillHorizon studies whether an agent can sustain correct skill use as a single task grows. The agent receives a skill library; the generator expands the work it must deliver and the dependencies among those activities. An agent may execute a procedure correctly at one stage but use its output incorrectly at the next, or fail to revisit an earlier result after its inputs change. We evaluate these consequences in the business environment: which goals remain satisfied, and does the final delivery pass acceptance?
SkillNet-Gym varies which skills a task brings together and how they connect. It constructs a graph by matching the output scenario of one skill to the input scenario of another, then samples skill combinations from that graph to generate executable tasks. SkillHorizon can keep those procedures fixed and instead vary the work that depends on their outputs within a single task. In DocOps, for example, the review and publication procedures stay the same as the portfolio grows. When an upstream contract changes, the agent must determine which published pages relied on it, revise those pages, obtain approval for the new versions, and leave unaffected pages intact. Knowing that review precedes publication does not tell the agent which earlier approvals are still valid. By checking the affected pages and their versions, we can measure how well the agent maintains prior work as new obligations accumulate.
SkillFlow studies how agents discover, refine, and reuse skills across successive tasks in a workflow family. What carries forward is the agent’s evolving skill library. In SkillHorizon, the library stays fixed while work accumulates within one task. In FinanceOps, for example, each new stage adds records to reconcile, and its closing record must incorporate the preceding stage’s result. SkillFlow thus tests whether a learned procedure transfers to the next task; SkillHorizon tests whether an agent using familiar procedures carries the right business result into the next stage.
Q4. All models are evaluated under a single framework with a fixed text-JSON action protocol and summary-based context management. Since models may differ in how well they adapt to these choices, how sensitive are the reported performance gaps and model rankings to the execution framework?
GPT-5.6 Sol’s task success rose from 18% to 84% when we changed the execution configuration. We evaluated it on the same 50 FinanceOps tasks, ten per workload band. Both configurations requested medium reasoning and used the same per-task tool allowances and business acceptance checks.
| Execution configuration | Passed | Failed final submissions | Tool allowance exhausted |
|---|---|---|---|
| Harbor + Inspect | 9 | 38 | 3 |
| Codex with direct MCP | 42 | 8 | 0 |
The largest change was in how runs ended: unsuccessful final submissions fell from 38 to 8, while runs exhausting their tool allowance fell from three to zero. The 66-point gain shows that the execution configuration matters substantially for GPT on these tasks. The action interface, provider route, and context management changed together, so the comparison does not isolate any one of them. Only GPT was evaluated under both configurations; whether the five-model gaps and ranking persist across harnesses remains untested.
Early stopping was directly observable in the main evaluation. At 1M, 183 unsuccessful GPT final submissions had records of both progress and remaining allowance. In 88 of them, GPT submitted after completing less than a quarter of the required goals while still having more than half its tool allowance left. Terminal-Bench, Figure 8 also identifies premature termination in approximately 14% of its sampled failed GPT-5.2 runs under Terminus-2. Independent evidence for harness sensitivity comes from Baselines Before Architecture, Table V: GPT-5 solved 70 of 104 tasks in each of two Codex runs, compared with 57 and 54 in the corresponding OpenCode runs.
Thank you for the clarifications and additional experiments. I appreciate the new evidence, though I still have reservations about how strongly it supports the paper’s central claims, like the 20 task families still appear to cover a relatively narrow range of distinct workflows. I will maintain my original score.
Thank you for considering the additional evidence. We would like to clarify what the 20 task families represent. They are the families we selected for the 1,050-task evaluation, not a limit on what the generators can construct. We designed them around real-world business procedures and the constraints that arise together in practice. Some families require different deliverables: in FinanceOps, reconciling payments against invoices is a different job from updating a financial release after a source correction. Others deliberately retain the same deliverable so that we can study how the decisions required to produce it change.
FactoryOps illustrates why the latter families matter. In selective rework, inspection determines which units need another production step and which can proceed. In equipment scheduling, maintenance and occupancy determine when a step can run. In the combined family, the agent only learns which additional operations to schedule after seeing the inspection results, and it must fit that work into the existing equipment calendar. The combination introduces a decision that neither condition poses on its own: how should the production plan change once the defective units are known? This is why we evaluate the mechanisms both separately and together.
The released generators also implement scenarios outside the experimental selection. For example, beyond the two FieldOps scenarios evaluated in the paper, the generator supports a tight-window scenario in which the agent must find both a qualified crew and a maintenance window long enough for the repair and its test. We will revise the manuscript to distinguish the business workflows covered, the mechanisms we compose within them, and the families on which we report results.
Response(1/2)
We thank the reviewer for the careful reading and helpful comments. We respond to each point below.
A skill defines a reusable procedure. The domain generator specifies where that procedure applies in a particular task: which objects are involved, what outputs are required, and which later activities need those outputs. A task module is therefore a concrete piece of required work, such as obtaining approval for a particular document. The generator defines the dependencies; it does not infer them from the skill text. During execution, the environment checks whether the artifacts the agent produces meet the specified conditions.
In DocOps, for example, the Coordinate Document Review skill tells the agent to submit the current draft to the designated reviewers, address their objections, and request a fresh review after an edit. The generator supplies the document, its designated reviewers, and its publication requirements. What connects review to publication is a version-specific approval: the environment permits publication only if the approval applies to the version being published. If the agent edits an approved draft, the earlier review still took place, but its approval no longer authorizes publication of the revised document.
We chose the families to cover different decisions agents face when one activity depends on the outcome of another. The seven environments span financial reconciliation, field service commissioning, campaign delivery, passenger and baggage coordination, research evidence maintenance, document publication, and production dispatch. Within these domains, we included both distinct business operations and variants of the same workflow. The 20 families comprise three each in SocialOps, LogisticsOps, ResearchOps, DocOps, and FactoryOps, three in FinanceOps, and two in FieldOps.
FactoryOps illustrates why we included variants. The delivery objective stays the same, but equipment availability determines when an operation can run, while selective rework requires the agent to decide which quarantined outputs need rework. The combined family requires scheduling that rework within the available equipment windows. In FinanceOps, by contrast, matching incoming payments to invoices and updating a financial release after a source correction are different business operations. The 1,050 evaluated instances exercise these 20 designs across five workload bands. This lets us compare agents across business workflows and, within a given workflow, examine how they respond when the decisions required for delivery change.