acceptodds
Under review as a conference paper at ICLR 2027

WorkFit Bench: Benchmarking Model–Scenario Fit across Real-World Agent Workflows

Abstract

Deploying agents for real-world work requires balancing task completion against model invocation cost. Yet existing benchmarks emphasize general capabilities and aggregate rankings, offering limited guidance for choosing a model for a specific work scenario. We introduce WorkFit Bench, a benchmark of 162 executable tasks across nine categories of work, constructed from reusable Skills. We evaluate 12 models across task categories and difficulty levels, measuring completion, estimated invocation cost, token use, and elapsed time. Overall completion rates range from 19.8% to 43.8%, while estimated costs for one pass through all 162 tasks range from USD 8.83 to USD 536.54 under an idealized cache-reuse assumption. Model rankings vary across scenarios and difficulty levels, and several lower-cost models achieve completion rates close to the highest overall score. WorkFit Bench provides evidence for choosing lower-cost models when they meet a scenario’s task-completion requirements. The repository is available at https://github.com/sbench-maker/WorkFit-Bench.

Primary area:
datasets and benchmarks
Keywords:
Agent Benchmarks, Scenario-Aware Evaluation, Real-World Work Tasks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9SGcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces WorkFit Bench, a benchmark of 162 executable tasks across nine work-scenario categories. Tasks were synthesized from a public agent Skill. A task passes only if all of its delivery conditions pass; these are mostly programatic checks, with some graded by an LLM judge. 12 models run once per task inside Claude Code. Overall completion ranges from 19.8% to 43.8%. From per-scenario and per-difficulty results and cost–completion frontiers, the authors argue that cheaper models often come close to the top scorer and that the best model choice depends on scenario and difficulty.

Critical strengths
  • Reproducible tasks. The benchmark ships executable, containerized tasks, and most verification is rule-based rather than done by an LLM judge. That makes results reproducible and easy to re-run as new models come out.
  • A practical framing. The paper treats model choice as a cost question within each work scenario rather than producing one metric. This matches how practitioners actually deploy agents.
Critical weaknesses
  • Each task is scored on a single "selected" run (Sec. 4.1). The paper does not say how that run was selected, and gives no confidence intervals, run-by-run variance, or significance tests.
  • In general the benchmark has low spread. With 162 tasks, Claude Opus 4.8's 71/162 has a 95% Wilson interval of roughly [36%, 52%]. The nine models between 38.9% and 43.8% are separated by at most 8 tasks, and their intervals overlap heavily.
  • The paper claims the tasks are "real world works", however the tasks were derived through synthetic generation from public skills. It is unclear how these works are anchored on real world workflows. Similarly, positioning against the closest work is missing - GDPval, the Remote Labor Index, Agents' Last Exam or APEX-Agents leverages domain experts or real projects, and grading is anchored to human expert deliverables and real professional workflows. The differentiator of the paper is unclear.
  • Judge using Gemini 3.8 Flash. This is susceptible to judge bias. A study of performance on different judge models and human agreement to the judge are missing.
  • Missing critical details and justifications on reasoning effort setting.
  • Missing critical details on harness setup - paper claims models run inside Claude Code through ACP. That is the native harness for Anthropic models but presumably requires an adapter for the others. Also it is questionable if this causes unfair disadvantage to other commercial models which should use their own native harness.
  • Missing quality control details - the paper miss details about how quality was checked. The tasks are largely synthetic and the only step with an explicit human is the verifiers: they were "reviewed by humans for quality, consistency, and acceptance of equivalent solutions" (Sec. 3.2).
Questions for authors
  • How was the selected run chosen? Report at least three independent runs with per-model variance and confidence intervals.
  • What evidence shows that the synthetic tasks represent real workflows? Were they reviewed by practitioners or compared with real user requests?
  • How does WorkFit Bench differ from GDPval, Remote Labor Index, Agents’ Last Exam, and APEX-Agents in task sources, grading, and evaluation goals?
  • How well does the Gemini 3.8 Flash judge agree with human labels? Do results change when using a judge from another model family?
  • What reasoning setting did each model use? Test how different settings affect completion and cost for leading models.
  • How are non-Anthropic models connected to Claude Code? How would that compare vendor-native tools, such as Gemini CLI and Codex?
  • Describe the human quality-control process, reviewer expertise and agreement, revision and rejection rates, reference-solution coverage, and audits of unsolved tasks.
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer 9SGc

Thank you for your detailed review and for recognizing our approach to reproducible task packaging and our cost evaluation framework. We also appreciate your specific questions about ranking uncertainty, task provenance, and evaluation settings.

W1–W2/Q1 Ranking uncertainty

"Author Response to Reviewer gKZ4" (Q1) explains run selection. "Author Response to Reviewer z7Jn" (W1) reports results and standard deviations across repeated runs.

W3/Q2–Q3 Task realism and novelty

Task sources and screening. We collected 1,917 skills from the open-source community. We considered their sources, community use, and citations to see whether practitioners recognized the workflows. We checked skill quality by looking for clear steps, constraints, and deliverables. We reviewed use cases and external dependencies to see what work each skill covered and whether it could become a reproducible task. This helped ensure task realism and quality from the outset.

Task authenticity and quality. We combine model generation and human review ("Author Response to Reviewer gKZ4", W1). A model drafts instructions, local data, and acceptance criteria from each selected skill. Reviewers spot-check task realism, data completeness, and fit with the skill's use case. We use their feedback to refine the synthesis pipeline.

Differences from existing benchmarks. Related benchmarks typically draw on work produced by experts, completed real-world projects, or professional tasks written by experts. They focus on deliverable quality, long-horizon work, or end-to-end project automation. WorkFit Bench draws on publicly available, reusable skill workflows for everyday work and covers multiple scenarios and difficulty levels. We ask which model users should choose when they already have a specific workflow or skill for a common work scenario. We therefore compare completion rates and call costs by scenario in one framework. This shows where a lower-cost model suffices and where paying more helps complete additional tasks.

W4/Q4 Judge bias

See "Author Response to Reviewer z7Jn" (W2) for the human and cross-model agreement results.

W5/Q5 Reasoning settings

The table lists the reasoning effort setting for each model. We will add it to the appendix in a future revision.

Reasoning effortModels
maxGPT-5.6-Sol, Claude Opus 4.8, Claude Sonnet 4.6
highGemini 3.8 Flash, Gemini 3.1 Pro Preview, DeepSeek 4.1 Flash, DeepSeek 4 Pro 0813, GLM 5.3, GLM 5.3 Flash, Kimi K3
xhighQwen 3.8 Max, Qwen 3.8 27B

We tested DeepSeek 4.1 Flash and Gemini 3.8 Flash at low reasoning effort on the original 162 tasks. We estimated costs using the paper's published unit prices and cache reuse assumptions. For DeepSeek 4.1 Flash, low effort reduced estimated cost by 48.2% and the completion rate by 2.47 percentage points. For Gemini 3.8 Flash, it reduced estimated cost by 51.5% and the completion rate by 10.49%.

ModelHigh completion rateLow completion rateHigh costLow cost
DeepSeek 4.1 Flash40.74% (66/162)38.27% (62/162)$8.83$4.58
Gemini 3.8 Flash42.59% (69/162)32.10% (52/162)$62.68$30.37

W6/Q6 Harness fairness

For non-Anthropic models, LiteLLM converts Claude Code requests to each model's API format, then converts responses and tool calls back to the format Claude Code expects.

We understand your concern that the framework can affect fairness. We ask how completion rates and call costs vary by model and work scenario within a fixed agent framework. We therefore use Claude Code and keep task packages, skill invocation, and verification criteria consistent. With vendor-native tools, both the model and the framework would vary, and the results would reflect both. We will clarify this design choice and limit our conclusions to the model–framework combinations we evaluated.

W7/Q7 Quality Control reporting

Human reviewers conducted task review and verifier review. All held at least a bachelor's degree.

One person reviewed the tasks. The reviewer checked whether the instructions reflected practical needs in the scenario, whether the inputs were de-identified as needed and followed the relevant business logic, and whether the materials and environment were sufficient to produce the requested deliverables. When the reviewer found issues, we revised the instructions, data, or environment and checked them again against the delivery requirements.

Two people reviewed the verifiers. They checked whether the acceptance conditions covered user requirements, whether the criteria were clear, and whether substantively equivalent solutions could pass. We then compared their judgments to assess agreement. See "Author Response to Reviewer z7Jn" (W2) for the agreement results.

"Author Response to Reviewer gKZ4" (W1) describes the full task construction process. We revised the dataset four times, updating the full task set and the data synthesis pipeline each time.

Thank you for the clarifications and additional experiments. The reasoning settings, harness details, and judge-agreement results address several of my concerns. However, the validation of synthetic tasks against real professional workflows remains limited. The repeated evaluations also do not establish stable spread of rankings: their three-run means differ by only about 0.2 percentage points. Finally, condition-level judge agreement does not establish whether task-level scores and rankings remain robust under alternative adjudication. These gaps still limit confidence in the benchmark’s verdict. I therefore maintain my rating.

Reviewer_z7JnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorkFit Bench evaluates model–scenario fit using 162 executable tasks across nine balanced work categories. Each task provides a preselected Skill, local data, and an execution environment. Twelve model configurations are evaluated through a shared agent framework, with completion determined by rule-based and semantic checks. The paper analyzes overall, scenario-specific, and difficulty-conditioned completion, then combines these results with idealized-cache cost estimates to derive value-dependent model choices. The results show substantial scenario heterogeneity and large estimated cost differences among models with similar aggregate completion rates.

Critical strengths
  1. Relevant practical motivation. Comparing task completion against invocation cost addresses an important consideration when selecting models for agent workflows.

  2. Balanced scenario coverage. The nine-category design and shared execution framework provide a useful starting point for comparing model configurations across different types of work.

Critical weaknesses
  1. Ranking stability is insufficiently established. Only one selected run per model–task pair is scored, with 18 tasks per scenario; the leading overall gap is just two tasks (Section 4.1; Table 4). Please clarify run selection and provide repeated paired evaluations with uncertainty estimates.

  2. Verifier validity needs stronger evidence. Human review is reported, but 124 conditions use Gemini 3.8 Flash, also an evaluated model, without a human-agreement audit or alternative-judge analysis (Table 3; Section 4.1). Since every condition must pass, please establish whether scoring errors could change the close rankings.

  3. Execution and missing-submission accounting are incomplete. Section 4.1 does not report task-level time/turn budgets, retry rules, or counts and causes of missing submissions. These details are needed to reproduce the comparisons and distinguish task-solving failures from execution failures.

  4. The cost-based recommendations need sensitivity analysis. Table 5’s thresholds depend on idealized caching and the pricing assumptions in Appendix C.2, without token breakdowns or alternative cache scenarios. Please show whether model choices remain stable under plausible cache hit rates and clarify the accounting of failed attempts.

Questions for authors

See Critical Weaknesses above.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer z7Jn

Thank you for recognizing the practical motivation behind WorkFit Bench and its balanced coverage across scenarios, and for your suggestions on where it needs improvement. Below, we provide the key metrics and experiments you identified as missing. We will include them in the main text or appendix of a future revision.

W1 Uncertainty in model rankings

The phrase "one selected run" is ambiguous. We retry a run up to twice when infrastructure or network issues cause an error. We report the score from the run that finishes, or from the last attempt if none finishes. Earlier failed runs are excluded. We will clarify this wording in a future revision. We evaluated three models multiple times to assess stability. The table shows the results.

ModelReported resultRepeat 1Repeat 2Three-run meanStandard deviation
Claude Opus 4.843.83% (71 tasks)40.12% (65 tasks)42.59% (69 tasks)42.18%1.89
Gemini 3.8 Flash42.59% (69 tasks)41.98% (68 tasks)41.36% (67 tasks)41.98%0.62
DeepSeek V4 Pro 081338.89% (63 tasks)38.27% (62 tasks)39.51% (64 tasks)38.89%0.62

After repeated evaluation, the relative ranking of the three models remained the same. Run-to-run variation on the full dataset was ±1.20 percentage points, equivalent to about two completed tasks out of 162. Claude Opus 4.8 showed greater variation across runs than the other two models.

W2 Verifier validity and scoring errors

We added an agreement analysis covering repeated judgments by Gemini 3.8 Flash, the judge used in the paper, as well as judgments by other models and by humans.

We included every semantic verification item from the evaluation results of Claude Opus 4.8, Gemini 3.8 Flash, DeepSeek 4.1 Flash, and GPT‑5.6‑Sol, for a total of 124 × 4 = 496 samples. Repeated judgments by the same model generally agreed. Gemini 3.8 Flash and DeepSeek V4.1 Flash made similar judgments. GPT‑5.6‑Sol agreed less often with either model and was stricter. Mean agreement with human judgments across the three judge models was 93.14%. Human judgments aligned more closely with Gemini 3.8 Flash and DeepSeek V4.1 Flash, which used more lenient criteria.

1. Agreement within a single judge model

Repeat roundsAgreement rate
Rounds 1 and 299.80%
Rounds 1 and 399.60%
Rounds 2 and 399.80%
All three rounds99.60%

2. Agreement across judge models

Judge pairAgreement rate
Gemini 3.8 Flash ↔ DeepSeek V4.1 Flash95.77%
Gemini 3.8 Flash ↔ GPT‑5.6‑Sol86.49%
DeepSeek V4.1 Flash ↔ GPT‑5.6‑Sol87.10%

3. Human agreement

Judge modelMatching judgmentsOverall agreement rate
Gemini 3.8 Flash474 / 49695.56%
DeepSeek V4.1 Flash467 / 49694.15%
GPT-5.6-Sol445 / 49689.72%

W3 Execution limits and non-submissions

The phrase "one selected run" is ambiguous. We retry a run up to twice when infrastructure or network issues cause an error. We report the score from the run that finishes, or from the last attempt if none finishes. Earlier failed runs are excluded. Across all tasks, the initial error rate was about 4.42%, and 2.78% of tasks still had errors after retries. Errors related to model stability and infrastructure accounted for 1.4% of all tasks. To approximate real working conditions, we set a one-hour time budget with no turn limit.

W4 Cost sensitivity to cache assumptions

We determined the preferred models at cache reuse rates of 25% and 75%. The two rates yield broadly consistent choices across scenarios. The order changes slightly from the ideal-cache setting but is largely preserved. In particular, GLM-F offers good value at low cost across all scenarios at both 25% and 75% cache reuse.

v is the value (USD) of one more completed task. Adjacent models tie at each threshold. "All v" means v ≥ 0.

25% cache reuse

ScenarioSelected model by v
SoftwareGLM-F (v<0.45), DS-F (v>0.45)
OperationsGLM-F (v<1.88), DS-F (1.88<v<8.10), Gemini (v>8.10)
RetrievalGLM-F (v<39.82), Claude Opus (v>39.82)
ContentGLM-F (all v)
PlanningGLM-F (v<23.82), GLM (v>23.82)
AnalysisGLM-F (v<9.33), Qwen (v>9.33)
BusinessGLM-F (v<6.77), DS-Pro (6.77<v<14.18), Gemini (14.18<v<115.87), GPT (v>115.87)
AI/agentsGLM-F (v<13.98), DS-Pro (13.98<v<83.69), Claude Opus (v>83.69)
CollectionGLM-F (v<1.10), DS-F (1.10<v<11.86), Qwen (v>11.86)

75% cache reuse

ScenarioSelected model by v
SoftwareGLM-F (v<0.10), DS-F (v>0.10)
OperationsGLM-F (v<0.59), DS-F (0.59<v<4.01), Gemini (v>4.01)
RetrievalGLM-F (v<24.40), Claude Opus (v>24.40)
ContentGLM-F (all v)
PlanningGLM-F (v<13.93), GLM (v>13.93)
AnalysisGLM-F (v<5.20), Qwen (v>5.20)
BusinessGLM-F (v<3.11), DS-Pro (3.11<v<7.10), Gemini (7.10<v<68.35), GPT (v>68.35)
AI/agentsGLM-F (v<6.41), DS-Pro (6.41<v<50.12), Claude Opus (v>50.12)
CollectionGLM-F (v<0.26), DS-F (0.26<v<6.27), Qwen (v>6.27)

We calculated costs only for the runs selected for scoring and excluded failed model calls within those runs.

Thank you for the additional experiments and clarifications.

However, important concerns remain: 1) The top two models reverse order in Repeat 1, and their three-run means differ by only approximately 0.2 percentage points. Preserving the ordering of the means does not establish stable rankings, particularly for scenario-level comparisons based on 18 tasks; 2) The human-agreement results are encouraging, but condition-level agreement does not establish whether task-level completion rates and close rankings remain unchanged under human or alternative-judge adjudication.

These concerns affect the reliability of the benchmark’s comparative conclusions and practical model-selection recommendations. I therefore maintain my original rating of 2.

Reviewer_gKZ4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

WorkFit Bench contains 162 executable tasks across nine work scenarios, constructed from curated agent Skills. Twelve model configurations are evaluated using a shared Claude Code scaffold, with binary completion determined by rule-based and model-based verifiers. The paper analyzes completion by scenario and annotated difficulty, estimates invocation costs, and derives scenario-specific model choices under an assumed value per completed task.

Critical strengths

The practical motivation is compelling: aggregate capability rankings provide incomplete guidance for choosing models for specific workflows.

Critical weaknesses
  1. How the categories of tasks are designed are unclear. And how to ensure the realism of the tasks is unclear.

  2. I am not sure the design of verification is comprehensive, whether there is strict state check to confirm the task is done? e.g., could a task pass by writing a convincing output file without performing the operation the user requested? Whether internet connection could introduce reward hacking, etc. These needs more clarification and testing.

Questions for authors

What does “one selected run” mean? How many attempts existed, how was selection performed, and how many failures were infrastructure-related?

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer gKZ4

Thank you for recognizing the practical motivation behind WorkFit Bench. Thank you for raising these questions. They give us an opportunity to explain in more detail how we construct the tasks, what the verifiers check, and how we select runs for scoring.

W1 Unclear taxonomy and task realism

We based the task categories on the categories of the underlying skills. We prioritized high-quality skills that are widely used and frequently referenced. Using a model, we grouped the 1,917 skills that passed screening by function and asked human reviewers to check the groupings. Reviewers corrected misclassified skills and merged overly narrow categories, leaving 23 functional categories. We then grouped those into nine work scenarios. We took this bottom-up approach because widely used and frequently referenced skills better reflect how people use agents in their day-to-day work.

We iterated through several rounds of task synthesis, model evaluation, and human review to make sure the tasks were realistic and reasonable. Reviewers checked whether each request fit its workflow and reflected a real need, and whether its verifiers covered the expected deliverables. We used their feedback to refine the synthesis pipeline and improve the tasks, helping ensure their realism and quality.

W2 Verification coverage and reward hacking risk

A task counts as complete only if it meets every delivery requirement. Each task has 3.88 verifiers on average. The verifiers check the final deliverable as well as required steps, task constraints, and evidence supporting the conclusions. In a simulated trading task, for example, the verifiers read the submitted receipt and the persisted account state separately. They check whether orders were canceled, funds were released, and holdings were updated, then compare the receipt with the account state.

To keep results consistent, we prohibit models from accessing the internet during task construction and evaluation. When building tasks, we package local inputs and use snapshots or mocks for workflows that involve remote services, so the tasks can be completed offline. During evaluation, we disable networking in the container and prohibit web searches.

Q1 Run selection and infrastructure failures

The phrase "one selected run" is ambiguous. We retry a run up to twice when infrastructure or network issues cause an error. We report the score from the run that finishes, or from the last attempt if none finishes. Earlier failed runs are excluded. We will clarify this wording in a future revision. Across all tasks, the initial error rate was about 4.42%, and 2.78% of tasks still had errors after retries. Errors related to model stability and infrastructure accounted for 1.4% of all tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.