- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AppSci-Bench, which aims to evaluate the ability to build a runnable agent from scratch. The benchmark covers seven types of real-world business tasks, requiring models to deliver executable agent systems, and evaluates their task performance and reliability via a hidden test set. Experiments reveal that the best-performing model achieves an overall score of only 59.1%, and that models generally suffer from insufficient requirement acquisition, distorted internal evaluation, and limited self-optimization effectiveness, indicating a substantial gap between current LLMs and reliable end-to-end agent development.
Critical strengths
- The benchmark includes the evaluation process overlooked by prior benchmarks, including requirement elicitation and self-optimization.
- The shortcomings the authors identify in existing LLMs' agent construction can inspire future related work.
Critical weaknesses
The authors explicitly state in Lines 455–456 that their benchmark's advantage over -Bench lies in "adopting this paradigm in a broader applied-science setting." However, the newly added domain covers only seven task scenarios, which focus on relatively narrow areas such as "Customer Call Insights" and "CRM Insights," making it insufficient to fully evaluate LLMs' agent construction capabilities.
If the authors could release the benchmark via an anonymous repository during the rebuttal period, and I can confirm that the data quality is good, I would be happy to raise my score.
Questions for authors
- During the review period, the authors did not release the benchmark through an anonymous repository, which limits the assessment of this benchmark's validity.
- Pages 8–9 look somewhat sparse overall. It would be better to add content to make these pages more substantial.
2 comments
Thanks for your response. I have checked that the benchmark is okay and have increased my score.
W, Q1, Q2: Benchmark Release, Adding more analysis to the paper
Thanks for your comment! We fully agree with the questions and provide explanations for the weakness below.
W1: We wanted to sample our task domains from realistic business scenarios and came up with these seven diverse domains as our initial candidates. We will support newer agentic use-cases in future releases as we come across more of them, to make the benchmark useful to the community.
Q1: You can find the entire benchmark along with the datasets here. All task related data can be found in the tasks/ directory: https://anonymous.4open.science/r/appsci-bench-CD3F.
Q2: We added another study Section 3.5: statistical significance studies on the effect of enabling optimization into the main paper. Please feel free to take a look! We also have ablation studies comparing a no-business user setting with the default setting (with user simulator) in Appendix I, that we can move into the main paper when we get the larger page limit. Some other analysis pointed out by reviewers, like benchmark's LLM-judge-human alignment and reasons for performance gap between native harnesses v/s AppSciAgent will also be moved up into the main paper.