- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
WorkThrough-Bench evaluates agents on 86 long-horizon productivity tasks derived from professional requests. The tasks cover deep research, deliverable creation, and application building. An isolated grading package combines executable checks, LLM-based judgments, and interactive behavioral tests. The benchmark reports both mean task completion and full-task pass rate. The authors evaluate six models with two agent harnesses and report a large gap between partial progress and complete delivery. They analyze cases in which agents verify their own implementation rather than the original requirements. A fixed feedback prompt asks agents to reread those requirements and inspect their deliverables. In the reported feedback experiment, mean pass rate rises from 5.63% to 7.13%.
Critical strengths
- A practical benchmark of usable deliverables and interdependent work.
The benchmark addresses a real need. It evaluates whether agents can complete research, produce files, and build working applications, rather than only answer questions. Tasks include dependencies between steps and require calculations, business rules, and multiple artifacts to remain consistent. This makes the task design closer to practical work than isolated generation tasks.
- A useful distinction between partial progress and complete delivery, with an actionable direction for improvement.
Reporting both Completion and Pass rate makes an important failure mode visible: an agent can satisfy many checks while still failing to produce an acceptable final deliverable. This distinction helps avoid overstating capability from partial credit alone. The trajectory examples also connect failures to a concrete behavior: agents check their own implementation instead of the original requirements. The fixed reminder to revisit those requirements offers a simple, practical direction for improving verification.
Critical weaknesses
- The overall aggregation and effective evaluation coverage are not sufficiently transparent.
Table 3 lists 31 research tasks, 26 deliverable-creation tasks, and 29 application-building tasks, with equal weight per task. For Qwen-3.8-Flash with Codex, the displayed category Completion scores give (31 × 59.7 + 26 × 68.9 + 29 × 78.9) / 86 = 68.96%, rather than the reported 69.3%. Rounding the category scores to one decimal place cannot explain this difference. Section 3.2 states that grading infrastructure failures produce no valid score, and Section 4.2 mentions missing coverage and shared-task comparisons. However, the paper does not provide enough per-group task counts and exclusion details to reconstruct the aggregate. Missing coverage could bias the results if it is related to task difficulty. Please report the effective task counts by category, excluded task IDs and reasons, repeat counts, and the exact aggregation procedure. This is a concern about transparency and comparability, not evidence of deliberate task selection.
- The feedback gain is reported without sufficient uncertainty analysis.
The 26.7% relative gain is arithmetically consistent with the reported increase in mean Pass rate from 5.63% to 7.13%. However, the absolute gain is 1.50 percentage points. Figure 4 reports comparisons on shared tasks, but does not provide confidence intervals, task-level paired outcomes, or sufficient repeat-level detail to judge the stability of the improvement. Please report absolute and relative gains together, the number of tasks and repeats in each comparison, and uncertainty estimates that respect the paired task structure. Without this information, it is difficult to assess whether the gain is robust or concentrated in a small number of successful runs.
- The feedback experiment does not isolate the benefit of the specific reminder from additional verification effort.
Section 4.3 allows up to three submissions within the original 24-hour limit. This keeps the maximum time allowance fixed, but does not establish matched realized compute between conditions. Feedback may improve results simply by prompting more inspection and revision. A generic reminder such as "check your work again" would be an important control. Please compare it with the proposed requirement-focused prompt under comparable continuation opportunities and resource budgets, and report actual token usage, tool calls, and runtime. Otherwise, the improvement cannot be attributed specifically to the proposed feedback content.
- The reliability of the learned graders is not sufficiently quantified.
The paper describes expert review, independent review, and agent-assisted audits during benchmark construction. These are useful safeguards. However, it does not provide a quantitative validation of the final LLM-based judgments and agent-based behavioral checks against human judgments on evaluated outputs. Agreement rates, false-positive and false-negative rates, and judge repeatability are not sufficiently reported. This matters because full-task Pass rate requires every scored check to pass, and a failed mandatory gate sets Completion to zero. A false rejection on one check can therefore change the task-level result. A human-adjudicated audit, including borderline cases and diverse valid solutions, would help establish whether the low pass rates reflect agent failures rather than grading errors.
Questions for authors
-
What are the effective task counts and aggregation weights for each model–harness group in Table 3? For Qwen-3.8-Flash with Codex, weighting the displayed category Completion scores by 31, 26, and 29 tasks gives 68.96%, rather than 69.3%. Could you provide the included task IDs, exclusions and reasons, repeat counts, and aggregation code? Does evaluating all groups on a common task set change the overall scores or rankings?
-
How stable is the feedback improvement from 5.63% to 7.13% mean Pass rate? Please report the paired task-level outcomes, task and repeat counts for each model–harness comparison, and confidence intervals for the absolute gain of 1.50 percentage points. How many tasks improve, regress, or remain unchanged, and is the gain concentrated in a small subset of tasks?
-
How much of the feedback gain comes from its requirement-focused wording rather than additional inspection and revision? Both conditions share a 24-hour limit, but feedback allows up to three submissions. Could you compare the proposed prompt with a generic "check your work again" prompt under the same continuation opportunities and comparable resource budgets? Please report actual token usage, tool calls, and runtime for these conditions.
-
How accurately and consistently do the LLM-based judges and behavioral evaluators assess final outputs? Could you report agreement with human judgments, false-positive and false-negative rates, and repeatability on a human-adjudicated sample that includes borderline cases and valid alternative solutions? Since a single failed check prevents full-task success, how often do grading errors change a task's pass/fail result or the measured feedback gain?
6 comments
Here, we report absolute and relative performance gains alongside paired uncertainty estimates. Each model–harness cell evaluates tasks paired across both conditions with up to three trials per task, and 95% confidence intervals are estimated via a paired bootstrap over tasks. Here, A and B denote the no-feedback and fixed-feedback conditions, respectively, with Pass and Completion reported as percentages. Following the aggregation scheme, the confidence interval is constructed by resampling tasks within each cell. The results are shown below:
| Model | Harness | Pass A → B | Δ Pass (95% CI) | Rel. Δ | Completion A → B | Δ Completion (95% CI) |
|---|---|---|---|---|---|---|
| GLM-5.3 | Hermes | 8.0 → 10.8 | +2.7 [−2.1, +7.4] | +34% | 75.3 → 75.2 | −0.1 [−3.2, +2.9] |
| GLM-5.3 | Codex | 4.0 → 4.0 | +0.0 [−4.4, +4.0] | 0% | 67.0 → 68.5 | +1.5 [−1.4, +4.4] |
| Kimi-K3 | Hermes | 9.6 → 10.8 | +1.2 [−1.6, +4.4] | +12% | 76.5 → 77.5 | +1.0 [−2.1, +3.5] |
| Kimi-K3 | Codex | 10.6 → 14.6 | +4.1 [−0.8, +8.9] | +38% | 79.8 → 81.5 | +1.7 [−0.8, +4.3] |
| MiniMax-M3 | Hermes | 0.4 → 1.8 | +1.4 [+0.0, +3.2] | +350% | 50.4 → 52.0 | +1.6 [−1.6, +4.7] |
| MiniMax-M3 | Codex | 1.2 → 0.8 | −0.4 [−2.0, +0.8] | −33% | 47.7 → 48.7 | +1.0 [−1.8, +3.8] |
| Average | 5.63 → 7.13 | +1.50 [+0.05, +2.98] | +26.7% | 66.1 → 67.2 | +1.1 [−0.0, +2.3] |
The macro-averaged Pass gain is reliably positive, with a 95% confidence interval strictly above zero at +1.50 percentage points, corresponding to a +26.7% relative improvement. By contrast, overall Completion rises by only +1.1 percentage points. This divergence reflects the underlying mechanism: rather than boosting partial credit uniformly across all tasks, fixed feedback primarily helps near-complete deliverables resolve residual omissions, thereby converting borderline attempts into full passes.
To evaluate the generic reminder "check your work again", we conducted experiments with Kimi-K3 across both harnesses using the prompt: "Please check your work again before finishing. You may take more time if you need it. If you find anything that should be fixed, fix it. Then, submit again once you are finished." Aside from the prompt itself, all experimental parameters matched our feedback setting, including the tasks, grader, 24-hour time limit, three-submission budget, and three trials per task. The table below reports realized runtime, token usage, and tool calls alongside Completion and Pass:
| Harness | Experiment | Completion | Pass | Runtime (min) | Input tokens (M) | Output tokens (k) | Tool calls | Submissions |
|---|---|---|---|---|---|---|---|---|
| Hermes | No feedback | 76.7 | 9.6 | 109.4 | 20.5 | 173 | 168 | 1 |
| Hermes | Generic reminder | 71.9 | 8.8 | 92.1 | 23.1 | 132 | 188 | 2.77 |
| Hermes | Fixed-feedback | 77.6 | 10.8 | 132.3 | 25.8 | 194 | 201 | 2.26 |
| Codex | No feedback | 82.0 | 9.0 | 153.6 | 17.7 | 185 | 180 | 1 |
| Codex | Generic reminder | 81.3 | 11.8 | 136.2 | 22.6 | 184 | 209 | 2.94 |
| Codex | Fixed-feedback | 84.7 | 18.1 | 180.6 | 22.2 | 203 | 213 | 2.31 |
The generic reminder fails to reproduce the performance gains, and this shortfall cannot be attributed to differences in effort. On Codex, it consumes comparable input tokens and tool calls to our feedback while requiring more submissions, averaging 2.94 versus 2.31, yet achieves a Pass rate of only 11.8% compared with 18.1%; on Hermes, it yields no improvement over the no-feedback baseline. A generic "continue working" prompt exhibits similar behavior, as detailed in our response to Reviewer sAjU.
Trajectory analysis indicates that the divergence lies not in whether agents verify their work, but in the reference standard they verify it against. Following the generic reminder, agents announce a re-check in 74–82% of initial responses, compared with 86% under our feedback, yet they evaluate against their own prior output: only 10–28% re-read the task specification and 22–35% of initial responses reference the task requirements, compared with 32–37% and 61–71% under our feedback, respectively. Representative responses focus on "the remaining details I flagged" or "fields I set hastily" rather than the task itself. This pattern exemplifies the self-contented completion discussed in the paper, which our feedback redirects toward the original requirements.
The following table shows the average of all six model–harness cells of Figure 4 against our proposed fixed-feedback costs:
| No feedback | Fixed-feedback | Change | |
|---|---|---|---|
| Completion (%) | 66.1 | 67.2 | +1.1 pp |
| Pass (%) | 5.63 | 7.13 | +26.7% |
| Runtime (min) | 102.3 | 147.4 | +44.0% |
| Input tokens (M) | 22.4 | 27.6 | +23.0% |
| Output tokens (k) | 176 | 192 | +9.3% |
| Tool calls | 189 | 222 | +17.4% |
| Submissions | 1 | 2.20 | — |
We thank the reviewer for careful examination. We apologize for the computation error for Qwen-3.8-Flash with the Codex harness and will correct Table 3 in the future. We confirm that no task is excluded on the basis of difficulty or model outcome. Runs interrupted by infrastructure errors, such as API instability, are not counted for the final scores. We re-run these failed runs to ensure the results are complete. The sentence in §3.2 refers to this re-run policy, and we will revise it to state this explicitly.
Difficulty likewise plays no role in task selection. During benchmark construction (§3.1), candidate tasks are retained or removed solely based on realism, solvability, genuine dependencies, and verifiable grading, resulting in 182 candidates being filtered to the final 86 tasks, and all valid tasks are retained regardless of trial-agent success (Figure 2). Consequently, task coverage does not depend on how difficult a task is for any given model.
For score aggregation, we average three validly graded runs for each task, assign a score of 0 to any run that fails a prerequisite check, like no file output, and weight all tasks equally within each family and overall. We will document this detailed aggregation procedure in the Appendix and release both the per-run grading records and the aggregation script so that every cell in Table 3 can be independently reconstructed.