- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FDE-Bench contains 49 real-project tasks requiring agents to clarify underspecified business requirements and deliver executable optimization or machine-learning artifacts. It combines a customer simulator, case-specific evaluators, and customer-accepted reference solutions. Across 18 model–scaffold configurations, agents remain below these references; information ablations show that clarification helps, while complete information still leaves a delivery gap.
Critical strengths
- Initial requests, hidden requirements, and accepted reference solutions originate from the same real projects. Connecting requirement elicitation to executable delivery makes the benchmark a useful complement to tasks with fully specified instructions (Sections 3.1–3.3).
- The evaluation is extensive: 18 configurations, three repeats per task, paired information ablations, and human audits of customer answers, clarification judgments, and artifact extraction. These support the benchmark’s practical value and make its reliability assessable (Section 4.3; Appendix A.8).
Critical weaknesses
The paper uses GLM-5.3 as the customer simulator. My only question is that whether the use of a llm to serve as a customer will influence the performance of llm on this bench. In other words, whether different model as customer will influence the final result, such as the ranking of different model on this bench. I regard this as a minor limitation.
Questions for authors
whether the use of a llm to serve as a customer will influence the performance of llm on this bench. In other words, whether different model as customer will influence the final result, such as the ranking of different model on this bench
Response to Reviewer 9jdb
We sincerely thank you for the positive review and for this question. It led us to run new experiments with other customer models, now Section 5.5 of the revised paper. Changes in the revised PDF are in blue.
Question.
Answer: In our new experiments it does not change the ranking: the five systems we re-ran keep the same order under two other customer models, and we find no sign that a customer model favors agents built on its own model family.
The five systems keep the same order under two other customer models. In 600 new runs on 20 cases (five opencode systems, three repeats; same customer prompt, case information, and scoring) with DeepSeek-V4-Pro and Claude Opus 5 as the customer, the order is the same as under GLM-5.3. Quality per customer (new Section 5.5, Table 3; the reference solution scores 1.0):
The ranking of agents by requirements obtained is stable across seven customer models. We sent the 3,456 question messages from 337 main-run dialogues to six other customer models and ranked the seven agent models by the share of requirements they obtain under each customer. The rankings under any two customers are close (Kendall τ at least 0.71), and GPT-6-Astra is first and Gemini-3.1-Pro last under all seven (added to Appendix A.6).
We find no sign that a customer model conveys more requirements to agents built on its own model family than to other agents. We tested this because GLM-5.3 is both the customer and one of the evaluated models. For GLM-5.3 the difference is −0.2 points, 95% CI [−4.3,3.8]; for the other five customer models whose family is evaluated, the estimates are −3.9 to +2.5 points, and every interval includes zero (Appendix A.6).
Score levels largely follow how completely each customer model answers. Given the same question messages, Claude Opus 5 conveys 2–10 percentage points more of a case's requirements than GLM-5.3, and DeepSeek-V4-Pro 16–31 points fewer, depending on the agent (Appendix A.6). We chose GLM-5.3 before the experiments by a vote of five FDEs, who compared candidate models' replies with the original customers' communication style. The new experiments are consistent with this choice: when a question targets a requirement, GLM-5.3 (95.5%) and Claude Opus 5 (96.6%) convey it most often among the seven customer models (the other five: 82.3–91.5%), and a five-model judge panel finds 0.1% of GLM-5.3's answers in conflict with the case's business information (Appendix A.6).