acceptodds
Under review as a conference paper at ICLR 2027

-Bench: An Environment for End-To-End, Realistic Agent Construction

Abstract

LLM agents are rapidly becoming production software, deployed to handle customer service, adjudicate disputes, and operate internal systems. Notably, the work of building them is increasingly handed to coding agents themselves, yet existing benchmarks say little about whether an AI system can deliver one under the conditions of a real client engagement. We introduce -bench, a benchmark that makes agent construction the task. A developer agent is given the records a business actually keeps, a client who holds requirements, a codebase to inherit, and limits on serving cost and models: the same starting point a real engagement provides. From these it must deliver a complete customer-service agent, scored by deploying that agent against held-out simulated users. Across 53 tasks spanning four domains, the strongest developer configuration, Claude Opus 5 under Claude Code, passes just 23.9% of evaluation simulations. Meanwhile, a human-authored reference ceiling scores 82.2%. The failures are the ones agent developers already see every day: models issue shallow queries in place of deep comprehension of the records, communicate almost nothing to the client, and experiment too little with agent architecture and serving spend, shipping the first design that runs. We aim for -bench to turn the work of cooperative agent building into a measurable target for coding agents.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Evaluation, Agents, Autoresearch

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_VFwdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces τττ^τ-Bench, a benchmark for evaluating whether coding agents can build and deliver complete customer-service agents for business clients. It combines requirements recovery from incomplete information with end-to-end agent construction in a unified evaluation framework, using atomic policy facts to support controlled variations in the development environment. The strongest evaluated configuration achieves an overall budget-adjusted score of 23.9, compared with 82.2 for an expert-authored oracle reference. Analysis of development trajectories reveals recurring shortcomings in requirements recovery, client consultation, validation of inherited code, API fault handling, and cost management.

Critical strengths
  1. The idea is straightforward and well implemented. Although the evaluation covers only six configurations, developing the benchmark and conducting the experiments clearly required substantial time and effort. And the analyses of agents' behavior are insightful.
  2. The design of deliberately flawed APIs and atomic policy facts is well motivated. The former introduces practical integration challenges, while the latter supports controlled task construction and systematic tracking of requirements.
  3. The paper is clear and well-presented.
Critical weaknesses
  1. The evaluation metric may be misleading. The paper frequently refers to the budget-adjusted score as the "pass rate," which does not necessarily reflect the actual task success rate, especially given the substantial impact of budget penalties. I suggest reporting both the raw pass rate and the budget-adjusted score in the main results table.
  2. The latency constraints are not clearly defined. Although the paper treats latency as a hard deployment constraint, it does not specify the thresholds or measurement methodology, nor does it report latency results. Moreover, since the constructed agents rely on external model APIs, measured latency is affected by provider-side delays and network fluctuations. The authors should clarify how latency constraints are enforced and how external latency variability is controlled to ensure fair comparisons across developer agents.
  3. The analysis could be more in-depth. While the paper identifies several failure modes through trajectory analysis and case studies, it lacks systematic ablations to quantify the impact of different benchmark components (e.g., evidence complexity, client interaction, and API defects) and attribute failures to different stages of agent construction. Moreover, some conclusions rely on correlations rather than controlled experiments. For example, Figure 4 shows that developers who ask more client questions achieve higher scores, but this may reflect differences in developer capabilities rather than the benefit of client interaction itself. Comparing the same developer under different client interaction settings would provide stronger evidence.
Questions for authors
  1. The benchmark is heavily dominated by banking tasks. Banking accounts for 35 of the 53 tasks, which may disproportionately influence the overall ranking. What motivates this imbalance and are there plans to expand the other domains?
  2. Claude Opus 5 achieves the highest overall score and performs particularly well on Retail, but falls behind GPT-5.6-sol on Banking. What explains these differences?
  3. Interestingly, Kimi K3 consistently performs better with OpenCode than with Kimi Code across all domains. What might explain this difference?
  4. Why was GPT-5.6-sol evaluated with xhigh reasoning effort rather than max, as used for the other models?
Reviewer_4xiAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ττ\tau^\tau-bench, a benchmark where the task is to build a customer-service agent rather than act as one. A developer agent receives a business's records (2,868 multimodal artifacts generated from atomic policy facts), a simulated client who holds some requirements not in the records, a client REST API that may contain deterministic defects, optionally an inherited codebase, and a fixed model menu with a per-conversation credit budget. The submitted agent is scored on held-out τ-bench-style tasks with simulated users. Score is mean reward minus a linear overspend penalty. The 53 release tasks span airline, retail, telecom, and banking, and a private set of 53 tasks is held back.

The authors evaluate six developer configurations across Codex, Claude Code, Kimi Code, and OpenCode. The best, Claude Code with Opus 5, scores 23.9%, against an 82.2% expert-authored reference. Trajectory analysis reports several recurring failures: developers grep the corpus instead of reading it, rarely ask the client, rewrite inherited code without running it, miss quiet API defects, mismanage the budget, converge on a single tool-loop architecture, and sometimes weaken their own tests.

Critical strengths
  1. Important and novel task formulation. Coding agents are already being used to build agents, but no prior benchmark measures this end to end, with spec recovery from records, an interactive stakeholder, and cost/model constraints. This is the main reason I lean toward acceptance: the benchmark opens a useful and timely evaluation direction.

  2. Careful, measurable environment design. Several design choices make failures diagnosable rather than just scored:

    • Policies are decomposed into atomic facts, so spec recovery can be checked.
    • The client's knowledge is a set of fact IDs, so elicitation is measurable.
    • API defects are deterministic and replay identically.
    • Tasks are built from independent levers, which allows controlled variants.

    The artifact generation rules (Appendix F) also show real effort to block shortcuts. This makes the benchmark likely to be reusable and extendable.

Critical weaknesses

1. The evaluation protocol is not rigorous enough to support the reported numbers and rankings.

  • Overspend penalty. The penalty subtracts the overspend ratio directly from the pass rate. This implies a 1:1 trade-off: 10% overspend costs 10 points, and any spend ≥2× budget scores 0 regardless of quality. The paper does not justify this form or coefficient, yet the choice clearly matters: 21 builds overshot the budget, and 10 of them lost an otherwise positive score. Because Table 2 reports only post-penalty scores, readers cannot see how much each configuration lost to the penalty. Adding pre-penalty quality to Table 2 would fix this. A short sensitivity check would also show whether the ranking holds under other reasonable designs, such as a smaller coefficient, a hard budget constraint, or a quality–cost Pareto view.

  • Single run per task. Each configuration is run once per task, and no variance or confidence intervals are reported. Close results, such as 23.9 vs 22.0 (Opus 5 vs GPT-5.6-sol) or 17.9 vs 16.1 (the two Kimi K3 harnesses), therefore cannot be ranked.

2. The main claim about the source of the gap is not supported by experiments.

The paper attributes the gap to "research" skills: reading the records, asking the client, exploring designs, and validating against ground truth. The evidence, however, is mostly qualitative trajectory excerpts plus a few correlations. No experiment isolates any of these stages, so the gap cannot be split between spec recovery, construction, and cost optimization.

This matters because several of the reported failures may be partly caused by the environment rather than by model capability:

  • The corpus is deliberately designed so that keyword search cannot isolate the fact-bearing artifacts (Appendix F).
  • The client is instructed to push back on broad questions and answer only narrow ones (Appendix K).
  • The developer brief in Appendix H does not mention the client.
  • The steep overspend penalty, with no reward for staying under budget, encourages a large safety margin.

As a result, the observations that developers "don't read, don't ask, and underspend" may partly describe how models respond to this particular environment. A few ablations on a small subset of tasks would help:

  • Giving the developer the ground-truth policy would isolate construction and cost optimization from spec recovery.
  • Stating the client's role explicitly in the brief would separate "doesn't know the client has information" from "doesn't think to ask".
  • Removing the budget would show how much the cost constraint itself contributes.
Questions for authors
  1. How was the budget b set for each task? The paper says only that easy budgets fit frontier models and hard budgets fit small models. Was b calibrated from the reference agents' measured spend? If so, the reference using 0.96× of the budget is close to true by construction, and the comparison with developer spend (≈0.5×) becomes circular. Please describe the calibration procedure.

  2. What architectures do the reference agents use? The paper uses the reference several times as evidence: "far more varied designs," more open-weight models, and near-full budget use. However, it never describes these designs. A short description, or a breakdown in the format of Fig. 5, would make the comparison verifiable and would also show readers what a good build looks like.

  3. Is the developer told that some requirements live only with the client? If developers are not told that the records are incomplete, the finding that developers rarely ask the client is partly an information issue rather than a capability issue. Please clarify what the developer is told about the client.

  4. Were the LLM judges and simulators validated against human judgment? All scores depend on GPT-5.5 user and client simulators and on LLM judges for messaging rules, but no agreement rates are reported. Please report judge–human agreement on a sample of conversations, any fidelity checks of the user simulator, and statistics from the three-auditor review of the artifacts (e.g., inter-auditor agreement and rejection rates).

Reviewer_WUUeOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes tau^tau bench, a benchmark for evaluating whether Agents can build customer-service agents. Instead of testing a finished agent, the benchmark gives a developer agent a set of business records, a simulated client, and some constraints, and asks the agent system to deliver a complete agent, which is then scored on held out tasks against simulated users. The benchmark covers 53 tasks in 4 domains, with another 53 kept private. The authors evaluate several developer configurations and find a large gap to an expert-written reference. Their analysis traces the gap to a few recurring failure modes in how the systems gather requirements and manage the budget.

Critical strengths
  1. timely problem. Whether AI systems can build new agents instead of just acting it is a real and interesting question.
  2. Solid construction. Policies are split into atomic facts and turned into realistic artifacts, with machine checks that every fact stays recoverable.
  3. Informative failure analysis. The paper reads the build trajectories and finds concrete failure modes: developers grep instead of read, almost never interview the client, and mismanage the budget. Claims are backed by trajectory excerpts.
Critical weaknesses
  1. agentic evaluation is noisy, and with only 6 tasks per non-banking domain, small differences could be noise. It could be better to repeate on a small subset of tasks and give an estimate of the variance.
  2. Why do banking tasks make up 35 of 53, when it is by far the hardest domain? Will it consider a rebalanced task mix or report score per domain?
  3. The evaluation itself relies heavily on LLM judges. Could the author report the agreement rate with human labels, and how much do scores move if judge model changes?
Questions for authors

See Weaknesses.

Reviewer_zFbqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ττ-bench, which asks a coding agent to build a customer-service agent instead of acting as one. The developer gets business records generated from τ-bench policies, on some tasks a simulated client who knows requirements missing from the records, a client API that may have planted bugs, sometimes an existing codebase, and a model list with a serving budget. The built agent is then tested on held-out simulated users. On 53 tasks across four domains, the best setup (Claude Opus 5 in Claude Code) gets 23.9%, while an expert-built reference gets 82.2%. The authors attribute the gap to developers grepping the records instead of reading them, rarely asking the client, and misusing the budget.

Critical strengths

Building an agent end to end, rather than running a given one, is a new and useful target. The setup reflects real projects: unclear specs, a client to ask, a buggy API, and cost limits. Each of these is set independently, so the benchmark supports controlled comparisons.

The construction is careful. Policies are split into atomic facts and every artifact is checked against them, so we know which facts the client holds and which ones a developer recovered. Scoring only looks at the deployed agent's behavior, not its implementation. The trajectory analysis (Section 6) and the cheating audit are informative.

Critical weaknesses

Each configuration is run once per task, and there are no confidence intervals. 35 of the 53 tasks are banking, where scores are 3.2–9.0%, so the overall number mostly reflects banking. Many differences in Table 2 are small (23.9% vs. 22.0% for the top two; 16.1% vs. 17.9% for Kimi K3 across harnesses), and without variance estimates it is unclear whether the rankings hold. All simulated parts (client, user simulator, judges, architecture classifier) are LLMs, and none are validated against humans; the classifier is only "spot-verified by hand." The artifacts are generated partly by the same models tested as developers (Claude Opus 5, Sonnet 5, GPT-5.6-sol), which could bias results. The claim that the failures match those of human developers has no human study behind it. Some analysis claims are weak. The 3x gain from asking the client is not controlled for model or task difficulty. The telecom jump from 31% to 67% with an intent-routing hint comes from one probe. The budget numbers are unclear: Figure 4 reports a 0.44× mean, while Table 2 averages about 0.58× and Section 6 says "about half."

Questions for authors

Can you repeat a few configurations on a subset of tasks to estimate variance, and add confidence intervals and a domain-balanced overall score? Does the Table 2 ranking still hold? Can you add a baseline without spec recovery, e.g., giving the developer the original τ-bench policy and tools under the same budget? Is the ceiling per task or per domain/tier, and which model did the expert use? How often do the judges and the classifier agree with humans? Does the client simulator ever volunteer facts without being asked, or refuse ones it should share? Do developers from the model families that generated the artifacts score differently? Does the client-question effect hold within the same configuration and task? Does the intent-routing result hold across domains and repeated runs? How does ττ-bench differ from the Meta-Agent Challenge (Lu et al., 2026), which is in Table 1 but not discussed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.