- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work introduces -Bench, a benchmark for evaluating whether coding agents can build and deliver complete customer-service agents for business clients. It combines requirements recovery from incomplete information with end-to-end agent construction in a unified evaluation framework, using atomic policy facts to support controlled variations in the development environment. The strongest evaluated configuration achieves an overall budget-adjusted score of 23.9, compared with 82.2 for an expert-authored oracle reference. Analysis of development trajectories reveals recurring shortcomings in requirements recovery, client consultation, validation of inherited code, API fault handling, and cost management.
Critical strengths
- The idea is straightforward and well implemented. Although the evaluation covers only six configurations, developing the benchmark and conducting the experiments clearly required substantial time and effort. And the analyses of agents' behavior are insightful.
- The design of deliberately flawed APIs and atomic policy facts is well motivated. The former introduces practical integration challenges, while the latter supports controlled task construction and systematic tracking of requirements.
- The paper is clear and well-presented.
Critical weaknesses
- The evaluation metric may be misleading. The paper frequently refers to the budget-adjusted score as the "pass rate," which does not necessarily reflect the actual task success rate, especially given the substantial impact of budget penalties. I suggest reporting both the raw pass rate and the budget-adjusted score in the main results table.
- The latency constraints are not clearly defined. Although the paper treats latency as a hard deployment constraint, it does not specify the thresholds or measurement methodology, nor does it report latency results. Moreover, since the constructed agents rely on external model APIs, measured latency is affected by provider-side delays and network fluctuations. The authors should clarify how latency constraints are enforced and how external latency variability is controlled to ensure fair comparisons across developer agents.
- The analysis could be more in-depth. While the paper identifies several failure modes through trajectory analysis and case studies, it lacks systematic ablations to quantify the impact of different benchmark components (e.g., evidence complexity, client interaction, and API defects) and attribute failures to different stages of agent construction. Moreover, some conclusions rely on correlations rather than controlled experiments. For example, Figure 4 shows that developers who ask more client questions achieve higher scores, but this may reflect differences in developer capabilities rather than the benefit of client interaction itself. Comparing the same developer under different client interaction settings would provide stronger evidence.
Questions for authors
- The benchmark is heavily dominated by banking tasks. Banking accounts for 35 of the 53 tasks, which may disproportionately influence the overall ranking. What motivates this imbalance and are there plans to expand the other domains?
- Claude Opus 5 achieves the highest overall score and performs particularly well on Retail, but falls behind GPT-5.6-sol on Banking. What explains these differences?
- Interestingly, Kimi K3 consistently performs better with OpenCode than with Kimi Code across all domains. What might explain this difference?
- Why was GPT-5.6-sol evaluated with xhigh reasoning effort rather than max, as used for the other models?