acceptodds
Under review as a conference paper at ICLR 2027

WorkThrough-Bench: Benchmarking Agents on Long-Horizon Productivity Work

Abstract

Long-horizon productivity work is becoming the proving ground for AI agents. Evaluating agents on such work remains challenging, as existing benchmarks often focus on minute-scale tasks, narrow work settings, or one-shot generation. We present WorkThrough-Bench, 86 tasks from professional requests across dozens of industries, spanning deep research, deliverable creation, and application building. Multi-artifact deliverables are independently graded through rule-based checks, LLM judgments, and behavioral probes. Across six frontier models and two agent harnesses, execution averages 122.8 minutes per run. Agents satisfy an average of 68.0% of the evaluation criteria per task, yet task pass rates remain below 11% in every model–harness group, and 54 of the 86 tasks are never completed in full by any group. Examining this gap between partial progress and reliable delivery, trajectory analysis reveals self-contented completion, where agents conclude that a task is complete based on checks of their own implementation and remembered requirements, without verifying the original task contract. We introduce feedback that redirects verification toward the original requirements and prompts corrections to overlooked discrepancies. Using fixed feedback, we improve the mean task pass rate by 26.7% relative to the no-feedback baseline, with trajectories showing renewed inspection and targeted repairs. These findings highlight the potential of further feedback design to improve agents' completion judgments and final delivery.

Primary area:
datasets and benchmarks
Keywords:
Long-horizon Benchmark, Agent Evaluation, LLM Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cnD9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WorkThrough-Bench evaluates agents on 86 long-horizon productivity tasks derived from professional requests. The tasks cover deep research, deliverable creation, and application building. An isolated grading package combines executable checks, LLM-based judgments, and interactive behavioral tests. The benchmark reports both mean task completion and full-task pass rate. The authors evaluate six models with two agent harnesses and report a large gap between partial progress and complete delivery. They analyze cases in which agents verify their own implementation rather than the original requirements. A fixed feedback prompt asks agents to reread those requirements and inspect their deliverables. In the reported feedback experiment, mean pass rate rises from 5.63% to 7.13%.

Critical strengths
  1. A practical benchmark of usable deliverables and interdependent work.

The benchmark addresses a real need. It evaluates whether agents can complete research, produce files, and build working applications, rather than only answer questions. Tasks include dependencies between steps and require calculations, business rules, and multiple artifacts to remain consistent. This makes the task design closer to practical work than isolated generation tasks.

  1. A useful distinction between partial progress and complete delivery, with an actionable direction for improvement.

Reporting both Completion and Pass rate makes an important failure mode visible: an agent can satisfy many checks while still failing to produce an acceptable final deliverable. This distinction helps avoid overstating capability from partial credit alone. The trajectory examples also connect failures to a concrete behavior: agents check their own implementation instead of the original requirements. The fixed reminder to revisit those requirements offers a simple, practical direction for improving verification.

Critical weaknesses
  1. The overall aggregation and effective evaluation coverage are not sufficiently transparent.

Table 3 lists 31 research tasks, 26 deliverable-creation tasks, and 29 application-building tasks, with equal weight per task. For Qwen-3.8-Flash with Codex, the displayed category Completion scores give (31 × 59.7 + 26 × 68.9 + 29 × 78.9) / 86 = 68.96%, rather than the reported 69.3%. Rounding the category scores to one decimal place cannot explain this difference. Section 3.2 states that grading infrastructure failures produce no valid score, and Section 4.2 mentions missing coverage and shared-task comparisons. However, the paper does not provide enough per-group task counts and exclusion details to reconstruct the aggregate. Missing coverage could bias the results if it is related to task difficulty. Please report the effective task counts by category, excluded task IDs and reasons, repeat counts, and the exact aggregation procedure. This is a concern about transparency and comparability, not evidence of deliberate task selection.

  1. The feedback gain is reported without sufficient uncertainty analysis.

The 26.7% relative gain is arithmetically consistent with the reported increase in mean Pass rate from 5.63% to 7.13%. However, the absolute gain is 1.50 percentage points. Figure 4 reports comparisons on shared tasks, but does not provide confidence intervals, task-level paired outcomes, or sufficient repeat-level detail to judge the stability of the improvement. Please report absolute and relative gains together, the number of tasks and repeats in each comparison, and uncertainty estimates that respect the paired task structure. Without this information, it is difficult to assess whether the gain is robust or concentrated in a small number of successful runs.

  1. The feedback experiment does not isolate the benefit of the specific reminder from additional verification effort.

Section 4.3 allows up to three submissions within the original 24-hour limit. This keeps the maximum time allowance fixed, but does not establish matched realized compute between conditions. Feedback may improve results simply by prompting more inspection and revision. A generic reminder such as "check your work again" would be an important control. Please compare it with the proposed requirement-focused prompt under comparable continuation opportunities and resource budgets, and report actual token usage, tool calls, and runtime. Otherwise, the improvement cannot be attributed specifically to the proposed feedback content.

  1. The reliability of the learned graders is not sufficiently quantified.

The paper describes expert review, independent review, and agent-assisted audits during benchmark construction. These are useful safeguards. However, it does not provide a quantitative validation of the final LLM-based judgments and agent-based behavioral checks against human judgments on evaluated outputs. Agreement rates, false-positive and false-negative rates, and judge repeatability are not sufficiently reported. This matters because full-task Pass rate requires every scored check to pass, and a failed mandatory gate sets Completion to zero. A false rejection on one check can therefore change the task-level result. A human-adjudicated audit, including borderline cases and diverse valid solutions, would help establish whether the low pass rates reflect agent failures rather than grading errors.

Questions for authors
  1. What are the effective task counts and aggregation weights for each model–harness group in Table 3? For Qwen-3.8-Flash with Codex, weighting the displayed category Completion scores by 31, 26, and 29 tasks gives 68.96%, rather than 69.3%. Could you provide the included task IDs, exclusions and reasons, repeat counts, and aggregation code? Does evaluating all groups on a common task set change the overall scores or rankings?

  2. How stable is the feedback improvement from 5.63% to 7.13% mean Pass rate? Please report the paired task-level outcomes, task and repeat counts for each model–harness comparison, and confidence intervals for the absolute gain of 1.50 percentage points. How many tasks improve, regress, or remain unchanged, and is the gain concentrated in a small subset of tasks?

  3. How much of the feedback gain comes from its requirement-focused wording rather than additional inspection and revision? Both conditions share a 24-hour limit, but feedback allows up to three submissions. Could you compare the proposed prompt with a generic "check your work again" prompt under the same continuation opportunities and comparable resource budgets? Please report actual token usage, tool calls, and runtime for these conditions.

  4. How accurately and consistently do the LLM-based judges and behavioral evaluators assess final outputs? Could you report agreement with human judgments, false-positive and false-negative rates, and repeatability on a human-adjudicated sample that includes borderline cases and valid alternative solutions? Since a single failed check prevents full-task success, how often do grading errors change a task's pass/fail result or the measured feedback gain?

6 comments

We thank the reviewer for careful examination. We apologize for the computation error for Qwen-3.8-Flash with the Codex harness and will correct Table 3 in the future. We confirm that no task is excluded on the basis of difficulty or model outcome. Runs interrupted by infrastructure errors, such as API instability, are not counted for the final scores. We re-run these failed runs to ensure the results are complete. The sentence in §3.2 refers to this re-run policy, and we will revise it to state this explicitly.

Difficulty likewise plays no role in task selection. During benchmark construction (§3.1), candidate tasks are retained or removed solely based on realism, solvability, genuine dependencies, and verifiable grading, resulting in 182 candidates being filtered to the final 86 tasks, and all valid tasks are retained regardless of trial-agent success (Figure 2). Consequently, task coverage does not depend on how difficult a task is for any given model.

For score aggregation, we average three validly graded runs for each task, assign a score of 0 to any run that fails a prerequisite check, like no file output, and weight all tasks equally within each family and overall. We will document this detailed aggregation procedure in the Appendix and release both the per-run grading records and the aggregation script so that every cell in Table 3 can be independently reconstructed.

Here, we report absolute and relative performance gains alongside paired uncertainty estimates. Each model–harness cell evaluates tasks paired across both conditions with up to three trials per task, and 95% confidence intervals are estimated via a paired bootstrap over tasks. Here, A and B denote the no-feedback and fixed-feedback conditions, respectively, with Pass and Completion reported as percentages. Following the aggregation scheme, the confidence interval is constructed by resampling tasks within each cell. The results are shown below:

ModelHarnessPass A → BΔ Pass (95% CI)Rel. ΔCompletion A → BΔ Completion (95% CI)
GLM-5.3Hermes8.0 → 10.8+2.7 [−2.1, +7.4]+34%75.3 → 75.2−0.1 [−3.2, +2.9]
GLM-5.3Codex4.0 → 4.0+0.0 [−4.4, +4.0]0%67.0 → 68.5+1.5 [−1.4, +4.4]
Kimi-K3Hermes9.6 → 10.8+1.2 [−1.6, +4.4]+12%76.5 → 77.5+1.0 [−2.1, +3.5]
Kimi-K3Codex10.6 → 14.6+4.1 [−0.8, +8.9]+38%79.8 → 81.5+1.7 [−0.8, +4.3]
MiniMax-M3Hermes0.4 → 1.8+1.4 [+0.0, +3.2]+350%50.4 → 52.0+1.6 [−1.6, +4.7]
MiniMax-M3Codex1.2 → 0.8−0.4 [−2.0, +0.8]−33%47.7 → 48.7+1.0 [−1.8, +3.8]
Average5.63 → 7.13+1.50 [+0.05, +2.98]+26.7%66.1 → 67.2+1.1 [−0.0, +2.3]

The macro-averaged Pass gain is reliably positive, with a 95% confidence interval strictly above zero at +1.50 percentage points, corresponding to a +26.7% relative improvement. By contrast, overall Completion rises by only +1.1 percentage points. This divergence reflects the underlying mechanism: rather than boosting partial credit uniformly across all tasks, fixed feedback primarily helps near-complete deliverables resolve residual omissions, thereby converting borderline attempts into full passes.

To evaluate the generic reminder "check your work again", we conducted experiments with Kimi-K3 across both harnesses using the prompt: "Please check your work again before finishing. You may take more time if you need it. If you find anything that should be fixed, fix it. Then, submit again once you are finished." Aside from the prompt itself, all experimental parameters matched our feedback setting, including the tasks, grader, 24-hour time limit, three-submission budget, and three trials per task. The table below reports realized runtime, token usage, and tool calls alongside Completion and Pass:

HarnessExperimentCompletionPassRuntime (min)Input tokens (M)Output tokens (k)Tool callsSubmissions
HermesNo feedback76.79.6109.420.51731681
HermesGeneric reminder71.98.892.123.11321882.77
HermesFixed-feedback77.610.8132.325.81942012.26
CodexNo feedback82.09.0153.617.71851801
CodexGeneric reminder81.311.8136.222.61842092.94
CodexFixed-feedback84.718.1180.622.22032132.31

The generic reminder fails to reproduce the performance gains, and this shortfall cannot be attributed to differences in effort. On Codex, it consumes comparable input tokens and tool calls to our feedback while requiring more submissions, averaging 2.94 versus 2.31, yet achieves a Pass rate of only 11.8% compared with 18.1%; on Hermes, it yields no improvement over the no-feedback baseline. A generic "continue working" prompt exhibits similar behavior, as detailed in our response to Reviewer sAjU.

Trajectory analysis indicates that the divergence lies not in whether agents verify their work, but in the reference standard they verify it against. Following the generic reminder, agents announce a re-check in 74–82% of initial responses, compared with 86% under our feedback, yet they evaluate against their own prior output: only 10–28% re-read the task specification and 22–35% of initial responses reference the task requirements, compared with 32–37% and 61–71% under our feedback, respectively. Representative responses focus on "the remaining details I flagged" or "fields I set hastily" rather than the task itself. This pattern exemplifies the self-contented completion discussed in the paper, which our feedback redirects toward the original requirements.

The following table shows the average of all six model–harness cells of Figure 4 against our proposed fixed-feedback costs:

No feedbackFixed-feedbackChange
Completion (%)66.167.2+1.1 pp
Pass (%)5.637.13+26.7%
Runtime (min)102.3147.4+44.0%
Input tokens (M)22.427.6+23.0%
Output tokens (k)176192+9.3%
Tool calls189222+17.4%
Submissions12.20—
Reviewer_sAjUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WorkThrough-Bench, 86 long-horizon office tasks built with LLM assistance from expert-contributed professional requests, each with inputs, an execution environment and an isolated grader. Tasks span deep research (31), deliverable creation (26) and application building (29), retained from 182 candidates after multi-party review and sandbox trials. Grading combines rule checks, binary LLM-judged rubrics and a browser-driven evaluation agent, reported as Completion (fraction of checks passed) and Pass rate (all checks passed). Six models run under Hermes Agent and Codex CLI. Runs average 122.8 minutes; mean Completion is 68.0%, Pass rate stays between 0.4% and 10.4%, and 54 tasks are never passed. The authors attribute part of the gap to "self-contented completion", and a fixed re-check prompt raises mean Pass rate by 26.7% relative on three models.

Critical strengths

The problem is well-motivated as the agents are now deployed on these office-type tasks.

Critical weaknesses

The biggest concern is that the headline gap may be mostly a property of the metric and grader, and nothing in the paper measures how much. Under all-or-nothing scoring over many binary checks, I would expect a near-zero Pass rate at 68% Completion from the metric alone, and every LLM-judged rubric or browser-probe step is another chance for a false rejection to zero the task. Reviewer prompts reportedly probe valid solutions that might be rejected, but the paper never says what grader errors that audit found, and reports no checks per task, no reference deliverable passing every check (tasks are retained "regardless of trial-agent success"), and no agreement or false-rejection rate for the judges.

With 54 of 86 tasks never passed and half the groups at 0.0 deep-research Pass rate, it is difficult to tell hard tasks from unpassable ones.

[1]. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

The feedback part can also use more careful design: there is no vanilla "keep working" baseline, no comparison to self-refinement loops, and no report of extra runtime or tokens, so it is difficult to credit the gain to redirecting verification toward the task contract.

Questions for authors

It seems beneficial to show more fine-grained grading, such as can the authors validate the grader first by showing a reference deliverable passing every task, reporting check counts and judging agreement with experts, quantifying self-contented completion over all failing runs, and rerunning feedback against a generic continuation control with variance over repeats.

Happy to raise my rating if the authors can show more detailed metrics and analysis as discussed above.

3 comments

Thank you for your comments; we'd love to show the details of our metrics, task creation, and judgments below.

For metrics applied, as discussed with Reviewer pkdV, Completion and Pass follow established practice, and the gap is not produced by all-or-nothing scoring alone: the same models show a 17–29 point gap on SWE-bench, against 60–70 points on our tasks. Failures are also not spread thinly across checks: 53% of failing runs miss six or more checks.

With respect to the per-task checks, the 86 tasks comprise a total of 2,378 scored checks: 1,557 scripted rule checks, 440 LLM judgments, and 381 browser-based agent judgments, accounting for 65%, 19%, and 16% of the total, respectively. Each task contains between 6 and 143 scored checks, with a mean of 27.7 and a median of 22; specifically, 85 tasks incorporate rule checks, 61 employ LLM judges, and 46 utilize agent judges. In addition, 70 tasks include 94 prerequisite checks in total, ranging from 0 to 3 per task, which verify solely that the required deliverables exist and can be parsed or that the application starts. Although these prerequisite checks are unscored, any failure results in a score of 0 for the run, as no subsequent grading can be performed.

The table below breaks down these checks by task family, reporting the total count alongside the per-task mean in parentheses.

FamilyTasksRule (script)LLM judgeAgent judgeTotal scoredPrerequisite
Deep research31373 (12.0)251 (8.1)262 (8.5)886 (28.6)42 (1.4)
Deliverable creation26670 (25.8)182 (7.0)0 (0.0)852 (32.8)15 (0.6)
Application building29514 (17.7)7 (0.2)119 (4.1)640 (22.1)37 (1.3)
All861557 (18.1)440 (5.1)381 (4.4)2378 (27.7)94 (1.1)

We found some grader issues during task construction. Pilot runs exposed several grader failure modes, each of which was fixed and validated by re-grading existing deliverables, as shown below:

IssueResolution
Structure-only evidence checks accepted templated rows or homepage URLs as valid evidenceAdded authenticity verification by fetching sampled sources, matching excerpts against page text, and flagging templated entries
Overly broad prerequisite checks, such as manifest validation, invalidated all downstream checks upon failureRestricted prerequisite checks to deliverable existence and parsability, scoring all other checks independently
LLM judges approved claims without supporting evidenceRequired judges to cite explicit evidence locations and fail unverifiable claims
Rigid exact-wording or exact-rounding matches rejected valid outputsNormalized wording and accepted all variants permitted by the specification
Existence-only checks were trivially satisfied without verifying correctnessIntroduced answer keys and automated recomputation from trusted inputs

For example, re-grading dropped a templated evidence ledger from 1.0 to about 0.68, while a strong research deliverable previously zeroed by a failed prerequisite check recovered to about 0.70.

Prior to the main experiments, to verify alignment with human experts, human annotators independently scored every LLM- and agent-judged check on one GLM-5.3 run per task. The resulting agreement rates with the automated judges are as follows:

JudgeJudgmentsAgreement (95% CI)False PositivesFalse Negatives
LLM44096.1% [93.9, 97.6]161
Agent38198.4% [96.6, 99.3]42
All82197.2% [95.8, 98.1]203

Treating human annotations as ground truth, false positives denote checks accepted by the automated judge but rejected by humans, and false negatives denote the reverse. Overall agreement reaches a Cohen's κ of 0.88, with only 20 false positives and 3 false negatives across all 821 judgments. This strong alignment confirms the methodological soundness and reliability of our automated evaluation.

To distinguish hard tasks from unpassable ones, we analyze check-level performance across all 1,116 runs on the 31 deep-research tasks, spanning six models, two harnesses, and three trials each. Pooling all runs, 97% of the 886 checks are satisfied at least once. The remaining 25 checks, or 3%, require complete deliverables or end-to-end traceability not yet achieved, such as an omitted diligence workbook or verification items unlinked to source extracts. Of the 23 tasks never passed in a single run, 12 have every check individually satisfied across separate runs and 6 miss only one, with the best runs reaching a median Completion of 91%. Thus, most failing tasks are hard rather than unpassable, failing only because no single run meets all requirements simultaneously.

A passing deep-research delivery is more than a well-written report. We grouped the checks by the property they verify; each group corresponds to a component of a correct delivery, and runs fail when that component is missing:

Required PropertyMiss RateShare of MissesObserved Failure Mode
Back every adopted claim with cited source text46%31%Cited sources do not support or contradict the claim
Meet LLM-judged research-quality criteria26%20%Template-filled analyses instead of per-unit judgments
Recompute numerical outputs from underlying inputs38%17%Scores and scenarios inconsistent with their own inputs
Include all required artifacts in parseable form21%13%Missing ledgers or model outputs
Propagate revised findings into final results32%9%Logged follow-ups whose final values remain unupdated
Maintain consistent entity identities across files30%6%Identical source IDs mapped to different entities across files
Deliver a functional research site46%6%Script errors leaving report sections empty

The research narrative itself usually passes; what fails is the evidence, computation, and revision work that the narrative depends on. This reflects agents' current inability to sustain such work consistently over hours, rather than unsatisfiable criteria. We will include the breakdown method and this further analysis in the appendix in the future.

What's more, we added a "keep working" baseline to compare against the original non-feedback and fixed-feedback experiments, using the prompt: "Please continue working on the task. You may take more time if you need it; submit again once you are finished." We ran this baseline using Kimi-K3 models across three independent runs while keeping all other experimental settings unchanged. The results are as follows:

HarnessExperimentCompletionPassRuntime (min)Input tokens (M)Output tokens (k)Tool callsSubmissions
HermesNo feedback76.69.5110.620.81741701
HermesKeep working70.79.593.923.61361902.64
HermesFixed-feedback77.610.7133.626.31972032.27
CodexNo feedback82.49.2160.019.01921901
CodexKeep working81.012.4134.521.51891992.82
CodexFixed-feedback84.718.3187.223.22092222.29

Merely "keep working" fails to deliver comparable gains: on Hermes, Pass remains unchanged and Completion drops, while on Codex, Pass rises far less than under our feedback. Trajectory analysis explains why. Following the generic prompt, agents continue editing deliverables in 82–85% of runs but rarely revisit the task specification: only 13–24% re-read it, compared with 31–37% under our feedback. Moreover, relative to our feedback, their initial responses more frequently propose polishing, expanding, or "hardening" existing work, at 29% versus 7%, and less frequently propose checking outputs against requirements, at 33% versus 67%. Such unguided edits leave unmet requirements unaddressed and can invalidate previously satisfied checks. In contrast, our feedback directs additional effort toward re-examining the task, converting near-complete work into full passes.

The realized cost of our feedback, averaged over all six model–harness cells of Figure 4, is modest relative to its effect on full delivery:

No feedbackFixed-feedbackChange
Completion (%)66.167.2+1.1 pp
Pass (%)5.637.13+26.7%
Runtime (min)102.3147.4+44.0%
Input tokens (M)22.427.6+23.0%
Output tokens (k)176192+9.3%
Tool calls189222+17.4%
Submissions12.20—

Regarding self-refinement loops, we explored designs in which the agent receives automatically computed feedback. However, signals informative enough to drive refinement closely mirror the final acceptance checks, which lets agents optimize against the grader rather than the task; to keep the evaluation reliable, we did not pursue this path. Self-improvement methods, including recursive self-improvement, ultimately improve agents through feedback, and our results show that even a static, grader-independent prompt makes feedback a viable lever. We would explore richer contract-grounded feedback for future work.

Reviewer_pkdVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WorkThrough-Bench to benchmark agents on long-horizon productivity work. The benchmark includes 86 tasks from professional requests across dozens of industries and execution averages 122.8 minutes per run. With the benchmark, the authors identify a gap between partial progress and reliable delivery. The authors also find that using fixed feedback, we can already improve the mean task pass rate by 26.7% relative to the no-feedback baseline, which highlights the potential of further feedback design to improve agents' completion judgments and final delivery.

Critical strengths
  1. The paper contributes 86 long-horizon tasks with average execution time as 122.8 minutes per run.
  2. The experiments on self-contented completion and feedback reveal interesting insights. Using a fixed prompt to inform agents to check against original instructions can directly improve results. This finding may be interesting to the community; although there is a caveat that the relative gain of 26.7% feels misleading as the absolute result is 5.63% to 7.13%.
Critical weaknesses
  1. While the paper contributes 86 long-horizon tasks, the overall methodology lacks novelty and scientific insights. The separation of task completion and pass rate is not new, as it has been a practice in benchmarks like Collaborative Gym, SWE-Bench, GDPval, etc. In those benchmarks, they've seen similar patterns that the agent has higher completion rate and lower pass rate.
  2. For task criteria, it's unclear how the quality is controlled. Based on the description in "Review and filtering" paragraph in Section 3.1, the creation process largely depends on LLM generation.
  3. The analysis in the paper stays general and does not touch the special aspects of long-horizon tasks. For example, for findings in this paper (e.g., completion vs. pass, the gain from fixed feedback), it's unclear whether it's specific to long-horizon tasks or it is true for any agent failure. If it's specific to long-horizon tasks, I'd be curious about what the underlying cuases are.
Questions for authors
  1. By emphasizing the results on fixed feedback, do you have further insights on how to improve agents' performance on long-horizon tasks? This finding is interesting but the current analysis can probably go deeper.
6 comments

We thank you for your comments. Regarding novelty and scientific significance, our contribution does not lie in proposing a new evaluation methodology; for the metrics, we follow established practice [1]. Rather, it lies in the problem we measure: long-horizon, self-contained, everyday office work spanning research, deliverable creation, and application building. Existing benchmarks do not cover this space; they focus on a single computer interface [2], one-shot deliverables [3], or minute-scale tasks (Table 2). Much as the SWE-bench line evolved from Verified to Pro to Multilingual by refining which problems are measured rather than by inventing new metrics, we aim for the same refinement for long-horizon office work.

This scope exposes limitations that narrower settings hide. Agents fail in recurring ways: research claims and numbers are not carried through to the final deliverable, batch outputs degrade into templated content that no longer reconciles with the summaries, and applications work at the API level but break in end-to-end user flows. More fundamentally, agents are self-contented: they declare completion after checking their own implementation rather than the task contract. We further show that feedback redirecting verification to the original requirements improves full delivery, pointing to a concrete direction for improving long-horizon agents.

[1] TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks.

[2] OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks.

[3] GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks.

For the task criteria creation procedure, LLMs only draft and audit; humans define and accept every task and criterion. Beyond §3.1, the criteria follow explicit rules: Each check maps a stated requirement to an observable outcome, so a deliverable that fully meets the specification can always receive full marks, and each check must name the failure or success it is meant to catch, or it is dropped. Deterministic properties are checked by scripts against recomputed values, while LLM or browser judges handle only semantic or interactive requirements and must cite evidence.

We separate generation and review. One agent proposes candidate checks, and an independent, stronger model keeps, deletes, or revises each one, looking for checks that a templated deliverable could pass or that could reject a valid one. This loop runs for up to three rounds, with the task author settling any remaining disagreement. Each script is then tested on paired samples: a correct sample must pass, and one with a perturbed number, source, or state must fail.

Finally, we validate tasks by execution. After expert, peer, and agent-assisted review, candidates run in a sandbox across models; for application building, the grader is also tested against an independent implementation built only from agent-visible materials. Every failed check is traced either to a task or grader defect, which is fixed and re-run, or to the agent. Tasks that cannot be graded reliably are removed: we dropped 96 of 182 candidates, leaving 86 for the final benchmark.

To test whether the completion–pass gap is specific to our setting, we evaluated the same models on code benchmarks and compared the average runtime, completion, and pass rate with ours. We report the average numbers below:

BenchmarkModelRuntime (min)CompletionPass
SWE-bench Verified [1]GLM-5.324.798.8%80.8%
SWE-bench Verified [1]Kimi-K323.399.2%82.4%
SWE-Bench Pro [2]GLM-5.341.191.2%64.5%
SWE-Bench Pro [2]Kimi-K343.390.5%61.1%
Ours (Hermes / Codex)GLM-5.3148.3 / 82.173.8% / 67.1%7.4% / 3.9%
Ours (Hermes / Codex)Kimi-K3118.3 / 157.975.5% / 79.8%9.3% / 10.3%

On code tasks, Completion and Pass differ by 17–29 points; on our tasks, by 60–70 points. Code tasks come with a test suite the agent can run, so self-verification and grading largely coincide, whereas our contracts are spread across long documents that the agent must reconstruct. The gap therefore reflects the long-horizon work regime rather than the metric.

Fixed feedback targets this failure. Without feedback, nearly every GLM-5.3 and Kimi-K3 run executes its own checks near the end, yet only 24% re-read the task specification. With feedback, 37% re-read it, 85% revise their deliverables, and Pass rises from 8.0% to 10.1%. These runs, however, repair only about a quarter of the checks they usually miss; the remaining errors lie in requirements spanning many entities, source verification, and numerical tie-outs. Feedback thus corrects what agents verify against, while the remaining errors mark the capability boundary the benchmark is designed to expose.

[1] SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

[2] SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

Comments on the paper

1 comment

General Response

We sincerely thank the Area Chair, Senior Area Chair, and reviewers for their constructive feedback. Our work WorkThrough-Bench evaluates 86 long-horizon, integrated office workflows via partial progress (Completion) and strict full delivery (Pass). Our study reveals that frontier agents reach 68.0% mean Completion yet only 6.9% mean Pass—driven by self-contented completion, where agents verify against their own outputs rather than task requirements—and our new experiments and human audits support the benchmark's value, solvability, and grading validity.

Recognized Strengths

  • Practical significance (sAjU, cnD9 and pkdV): Evaluates 86 hours-long tasks (122.8 min/run avg.) across interdependent research, deliverables, and applications.
  • Diagnostic metrics (cnD9): Separating Completion from Pass prevents partial credit from masking incomplete delivery.
  • Mechanistic insight (pkdV, cnD9): Diagnoses self-contented completion and validates requirement-anchored feedback.
  • Evaluation rigor (cnD9): Multi-stage filtering and human validation support grading reliability.

Core Concerns and Resolutions

1. Novelty of our benchmark (pkdV)

In follow-up remarks, pkdV questioned adding evaluation data alongside benchmarks such as Terminal-Bench or commercial datasets. Our contribution extends beyond data collection in two respects:

  • Open standard for general workflows: Proprietary vendor datasets remain closed training assets, while existing open benchmarks isolate system tools or short tasks. WorkThrough-Bench establishes an open standard for broad, hours-long office workflows.
  • Isolating a distinct failure regime: On standard coding benchmarks (SWE-bench Verified/Pro), the same frontier models exhibit a 17–29 point Completion–Pass gap versus 60–70 points on our benchmark. Because office requirements are dispersed across long documents without built-in test suites, agents verify against their own artifacts rather than the specification—yielding concrete harness-design principles.

2. Task Solvability and Baseline Reliability (sAjU, cnD9)

Reviewers asked whether low Pass rates reflect unpassable tasks or noisy judges. We verified solvability and grading accuracy directly from executed runs:

  • Task solvability: 65% of checks are deterministic scripts validated on paired valid and perturbed outputs, and application-building graders are tested against an independent implementation built only from agent-visible materials. Across more than 1,000 deep-research runs, 97% of checks are satisfied at least once, and 12 of the 23 never-passed deep-research tasks have every check satisfied across separate runs, indicating that most failing tasks are hard rather than unpassable.
  • Baseline audit: Independent human grading of all 821 LLM- and browser-judged checks on a single no-feedback GLM-5.3 + Hermes run per task (rather than all three trials) achieved 97.2% agreement with κ equal to 0.88; correcting every disagreement flipped zero of the 73 audited task-level Pass outcomes.

3. Further Analysis of the Feedback Gain (sAjU, cnD9, pkdV)

Prompting agents to verify against original requirements raises strict Pass by +1.50 pp (95% CI [+0.05, +2.98], paired bootstrap). Reviewers questioned whether extra compute or judge bias under feedback (cnD9 follow-up) explains this improvement:

  • Requirement anchoring vs. extra effort: Under identical time and submission budgets, two generic controls ("keep working" and "check your work again") consumed comparable tokens and more submissions yet failed to match fixed feedback, as agents continued polishing their own outputs rather than re-reading the specification.
  • Ruling out grading bias under feedback: While manually re-grading ~2,900 multi-hour runs is infeasible within a rebuttal window, two analyses directly address cnD9's concern: (1) the share of runs clearing all deterministic script checks, which no learned judge can influence, rises by +2.02 pp (95% CI [+0.55, +3.47]), positive in every model–harness cell, with no sign of more lenient judging under feedback in the script-dominated families or in the audited cell; (2) a paired human audit of a single fixed-feedback GLM-5.3 + Hermes run (rather than all three trials) on each of the 14 baseline-disagreement tasks yielded higher agreement and fewer false positives than the no-feedback audit, and applying every human correction leaves the gain essentially unchanged.

Summary

WorkThrough-Bench addresses an under-explored frontier as AI agents take on hours-long knowledge work. Beyond establishing an open standard for realistic office workflows, it identifies self-contented completion as a fundamental long-horizon failure mode and validates specification-anchored feedback as an effective remedy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.