- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces WorkFit Bench, a benchmark of 162 executable tasks across nine work-scenario categories. Tasks were synthesized from a public agent Skill. A task passes only if all of its delivery conditions pass; these are mostly programatic checks, with some graded by an LLM judge. 12 models run once per task inside Claude Code. Overall completion ranges from 19.8% to 43.8%. From per-scenario and per-difficulty results and cost–completion frontiers, the authors argue that cheaper models often come close to the top scorer and that the best model choice depends on scenario and difficulty.
Critical strengths
- Reproducible tasks. The benchmark ships executable, containerized tasks, and most verification is rule-based rather than done by an LLM judge. That makes results reproducible and easy to re-run as new models come out.
- A practical framing. The paper treats model choice as a cost question within each work scenario rather than producing one metric. This matches how practitioners actually deploy agents.
Critical weaknesses
- Each task is scored on a single "selected" run (Sec. 4.1). The paper does not say how that run was selected, and gives no confidence intervals, run-by-run variance, or significance tests.
- In general the benchmark has low spread. With 162 tasks, Claude Opus 4.8's 71/162 has a 95% Wilson interval of roughly [36%, 52%]. The nine models between 38.9% and 43.8% are separated by at most 8 tasks, and their intervals overlap heavily.
- The paper claims the tasks are "real world works", however the tasks were derived through synthetic generation from public skills. It is unclear how these works are anchored on real world workflows. Similarly, positioning against the closest work is missing - GDPval, the Remote Labor Index, Agents' Last Exam or APEX-Agents leverages domain experts or real projects, and grading is anchored to human expert deliverables and real professional workflows. The differentiator of the paper is unclear.
- Judge using Gemini 3.8 Flash. This is susceptible to judge bias. A study of performance on different judge models and human agreement to the judge are missing.
- Missing critical details and justifications on reasoning effort setting.
- Missing critical details on harness setup - paper claims models run inside Claude Code through ACP. That is the native harness for Anthropic models but presumably requires an adapter for the others. Also it is questionable if this causes unfair disadvantage to other commercial models which should use their own native harness.
- Missing quality control details - the paper miss details about how quality was checked. The tasks are largely synthetic and the only step with an explicit human is the verifiers: they were "reviewed by humans for quality, consistency, and acceptance of equivalent solutions" (Sec. 3.2).
Questions for authors
- How was the selected run chosen? Report at least three independent runs with per-model variance and confidence intervals.
- What evidence shows that the synthetic tasks represent real workflows? Were they reviewed by practitioners or compared with real user requests?
- How does WorkFit Bench differ from GDPval, Remote Labor Index, Agents’ Last Exam, and APEX-Agents in task sources, grading, and evaluation goals?
- How well does the Gemini 3.8 Flash judge agree with human labels? Do results change when using a judge from another model family?
- What reasoning setting did each model use? Test how different settings affect completion and cost for leading models.
- How are non-Anthropic models connected to Claude Code? How would that compare vendor-native tools, such as Gemini CLI and Codex?
- Describe the human quality-control process, reviewer expertise and agreement, revision and rejection rates, reference-solution coverage, and audits of unsolved tasks.
2 comments
Thank you for the clarifications and additional experiments. The reasoning settings, harness details, and judge-agreement results address several of my concerns. However, the validation of synthetic tasks against real professional workflows remains limited. The repeated evaluations also do not establish stable spread of rankings: their three-run means differ by only about 0.2 percentage points. Finally, condition-level judge agreement does not establish whether task-level scores and rankings remain robust under alternative adjudication. These gaps still limit confidence in the benchmark’s verdict. I therefore maintain my rating.
Author Response to Reviewer 9SGc
Thank you for your detailed review and for recognizing our approach to reproducible task packaging and our cost evaluation framework. We also appreciate your specific questions about ranking uncertainty, task provenance, and evaluation settings.
W1–W2/Q1 Ranking uncertainty
"Author Response to Reviewer gKZ4" (Q1) explains run selection. "Author Response to Reviewer z7Jn" (W1) reports results and standard deviations across repeated runs.
W3/Q2–Q3 Task realism and novelty
Task sources and screening. We collected 1,917 skills from the open-source community. We considered their sources, community use, and citations to see whether practitioners recognized the workflows. We checked skill quality by looking for clear steps, constraints, and deliverables. We reviewed use cases and external dependencies to see what work each skill covered and whether it could become a reproducible task. This helped ensure task realism and quality from the outset.
Task authenticity and quality. We combine model generation and human review ("Author Response to Reviewer gKZ4", W1). A model drafts instructions, local data, and acceptance criteria from each selected skill. Reviewers spot-check task realism, data completeness, and fit with the skill's use case. We use their feedback to refine the synthesis pipeline.
Differences from existing benchmarks. Related benchmarks typically draw on work produced by experts, completed real-world projects, or professional tasks written by experts. They focus on deliverable quality, long-horizon work, or end-to-end project automation. WorkFit Bench draws on publicly available, reusable skill workflows for everyday work and covers multiple scenarios and difficulty levels. We ask which model users should choose when they already have a specific workflow or skill for a common work scenario. We therefore compare completion rates and call costs by scenario in one framework. This shows where a lower-cost model suffices and where paying more helps complete additional tasks.
W4/Q4 Judge bias
See "Author Response to Reviewer z7Jn" (W2) for the human and cross-model agreement results.
W5/Q5 Reasoning settings
The table lists the reasoning effort setting for each model. We will add it to the appendix in a future revision.
We tested DeepSeek 4.1 Flash and Gemini 3.8 Flash at low reasoning effort on the original 162 tasks. We estimated costs using the paper's published unit prices and cache reuse assumptions. For DeepSeek 4.1 Flash, low effort reduced estimated cost by 48.2% and the completion rate by 2.47 percentage points. For Gemini 3.8 Flash, it reduced estimated cost by 51.5% and the completion rate by 10.49%.
W6/Q6 Harness fairness
For non-Anthropic models, LiteLLM converts Claude Code requests to each model's API format, then converts responses and tool calls back to the format Claude Code expects.
We understand your concern that the framework can affect fairness. We ask how completion rates and call costs vary by model and work scenario within a fixed agent framework. We therefore use Claude Code and keep task packages, skill invocation, and verification criteria consistent. With vendor-native tools, both the model and the framework would vary, and the results would reflect both. We will clarify this design choice and limit our conclusions to the model–framework combinations we evaluated.
W7/Q7 Quality Control reporting
Human reviewers conducted task review and verifier review. All held at least a bachelor's degree.
One person reviewed the tasks. The reviewer checked whether the instructions reflected practical needs in the scenario, whether the inputs were de-identified as needed and followed the relevant business logic, and whether the materials and environment were sufficient to produce the requested deliverables. When the reviewer found issues, we revised the instructions, data, or environment and checked them again against the delivery requirements.
Two people reviewed the verifiers. They checked whether the acceptance conditions covered user requirements, whether the criteria were clear, and whether substantively equivalent solutions could pass. We then compared their judgments to assess agreement. See "Author Response to Reviewer z7Jn" (W2) for the agreement results.
"Author Response to Reviewer gKZ4" (W1) describes the full task construction process. We revised the dataset four times, updating the full task set and the data synthesis pipeline each time.