acceptodds
Under review as a conference paper at ICLR 2027

FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests

Abstract

Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experience. Forward-deployed engineers (FDEs) must elicit these requirements through dialogue and iteratively optimize solutions until they meet customer acceptance criteria. Existing optimization benchmarks typically start from explicit objectives, while clarification benchmarks often introduce information gaps into existing tasks. We introduce FDE-Bench, the first benchmark built from real customer projects to evaluate agents on end-to-end delivery tasks that require interactive requirement clarification and iterative optimization. Its 49 cases span combinatorial optimization and machine learning. Clarification targets are distilled from discussions between frontline practitioners and FDEs. Agents elicit missing requirements from a simulated customer, and case-specific evaluators assess final artifacts against acceptance criteria using customer-accepted FDE solutions as quality references. Across 18 agent configurations, the highest mean delivery score is only **0.5510**, compared with an FDE reference of **1.0**. The highest delivery success rate (DSR), defined as the fraction of planned runs reaching this reference, is only **6.12%**. These results show that reliable FDE-level end-to-end delivery remains out of reach for the evaluated agents on underspecified real-world business tasks. The code and dataset for FDE-Bench are anonymously available at: [https://anonymous.4open.science/r/FDE-Bench/](https://anonymous.4open.science/r/FDE-Bench/).

Primary area:
datasets and benchmarks
Keywords:
AI Agents, Benchmarking and Evaluation, Human–AI Interaction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9jdbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FDE-Bench contains 49 real-project tasks requiring agents to clarify underspecified business requirements and deliver executable optimization or machine-learning artifacts. It combines a customer simulator, case-specific evaluators, and customer-accepted reference solutions. Across 18 model–scaffold configurations, agents remain below these references; information ablations show that clarification helps, while complete information still leaves a delivery gap.

Critical strengths
  1. Initial requests, hidden requirements, and accepted reference solutions originate from the same real projects. Connecting requirement elicitation to executable delivery makes the benchmark a useful complement to tasks with fully specified instructions (Sections 3.1–3.3).
  2. The evaluation is extensive: 18 configurations, three repeats per task, paired information ablations, and human audits of customer answers, clarification judgments, and artifact extraction. These support the benchmark’s practical value and make its reliability assessable (Section 4.3; Appendix A.8).
Critical weaknesses

The paper uses GLM-5.3 as the customer simulator. My only question is that whether the use of a llm to serve as a customer will influence the performance of llm on this bench. In other words, whether different model as customer will influence the final result, such as the ranking of different model on this bench. I regard this as a minor limitation.

Questions for authors

whether the use of a llm to serve as a customer will influence the performance of llm on this bench. In other words, whether different model as customer will influence the final result, such as the ranking of different model on this bench

1 comment
AuthorsOpenReview ↗

Response to Reviewer 9jdb

We sincerely thank you for the positive review and for this question. It led us to run new experiments with other customer models, now Section 5.5 of the revised paper. Changes in the revised PDF are in blue.

Question.

The paper uses GLM-5.3 as the customer simulator. My only question is that whether the use of a llm to serve as a customer will influence the performance of llm on this bench. In other words, whether different model as customer will influence the final result, such as the ranking of different model on this bench.

Answer: In our new experiments it does not change the ranking: the five systems we re-ran keep the same order under two other customer models, and we find no sign that a customer model favors agents built on its own model family.

  • The five systems keep the same order under two other customer models. In 600 new runs on 20 cases (five opencode systems, three repeats; same customer prompt, case information, and scoring) with DeepSeek-V4-Pro and Claude Opus 5 as the customer, the order is the same as under GLM-5.3. Quality per customer (new Section 5.5, Table 3; the reference solution scores 1.0):

    SystemDeepSeek-V4-ProGLM-5.3 (main)Claude Opus 5
    GPT-6-Astra0.4790.5800.658
    GLM-5.30.4310.5330.590
    DeepSeek-V4-Pro0.4070.5060.521
    Kimi-K30.3460.3690.381
    Gemini-3.1-Pro0.2460.2910.282
  • The ranking of agents by requirements obtained is stable across seven customer models. We sent the 3,456 question messages from 337 main-run dialogues to six other customer models and ranked the seven agent models by the share of requirements they obtain under each customer. The rankings under any two customers are close (Kendall τ\tau at least 0.71), and GPT-6-Astra is first and Gemini-3.1-Pro last under all seven (added to Appendix A.6).

  • We find no sign that a customer model conveys more requirements to agents built on its own model family than to other agents. We tested this because GLM-5.3 is both the customer and one of the evaluated models. For GLM-5.3 the difference is −0.2-0.2 points, 95% CI [−4.3,3.8][-4.3, 3.8]; for the other five customer models whose family is evaluated, the estimates are −3.9-3.9 to +2.5+2.5 points, and every interval includes zero (Appendix A.6).

  • Score levels largely follow how completely each customer model answers. Given the same question messages, Claude Opus 5 conveys 2–10 percentage points more of a case's requirements than GLM-5.3, and DeepSeek-V4-Pro 16–31 points fewer, depending on the agent (Appendix A.6). We chose GLM-5.3 before the experiments by a vote of five FDEs, who compared candidate models' replies with the original customers' communication style. The new experiments are consistent with this choice: when a question targets a requirement, GLM-5.3 (95.5%) and Claude Opus 5 (96.6%) convey it most often among the seven customer models (the other five: 82.3–91.5%), and a five-model judge panel finds 0.1% of GLM-5.3's answers in conflict with the case's business information (Appendix A.6).

Reviewer_KWi5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FDE-Bench, comprising 49 real customer projects that require agents to clarify underspecified requirements and iteratively develop optimization or machine-learning solutions. It contributes a simulated-customer interaction framework and case-specific evaluators grounded in customer-accepted expert deliverables. Across 18 agent configurations, the best mean delivery score is 0.551 relative to an expert reference of 1.0, and the highest reference-level success rate is 6.12%. Information ablations show that clarification improves delivery, but complete requirements alone do not close the performance gap.

Critical strengths

Real-project grounding creates a meaningful evaluation setting (Section 3). The initial requests, implicit requirements, and accepted expert solutions originate from the same customer projects. This links clarification to consequential delivery outcomes and provides a concrete quality reference beyond synthetically introduced ambiguities.

Information ablations distinguish requirement acquisition from execution limitations (Section 5.4). Comparing no-answer, interactive, and full-information conditions shows that clarification substantially improves delivery, while complete requirements still leave a performance gap. This makes the benchmark diagnostically useful: it identifies challenges in both acquiring requirements and translating them into effective solutions.

Critical weaknesses

All systems interact with a single fixed customer model. Although answer-grounding audits are reassuring, annotators identify cross-turn inconsistencies in 9–10 of 105 audited dialogues. The paper does not establish whether these inconsistencies materially affect requirement acquisition or final scores, or whether conclusions persist with another simulator. Because interactive clarification is the benchmark’s distinguishing contribution, this uncertainty limits confidence that measured differences reflect agent capabilities rather than simulator behavior.

Submitted artifacts are normalized by model-based extractors before executable evaluation. The human audit and estimated 97.9% extraction accuracy provide useful validation, but the paper does not quantify how identified extraction errors change delivery scores or system rankings. This matters because a hard-constraint violation produces zero credit, and reference-level successes are rare. Agreement among extractors cannot exclude shared interpretation errors. Reporting scores before and after human correction on the audited subset would better establish the reliability of the headline metrics.

Questions for authors

For the 9–10 audited dialogues containing cross-turn inconsistencies, did those inconsistencies concern requirements relevant to feasibility or scoring? Did affected agents receive incorrect or conflicting information that could explain their delivery failures?

Do you have existing comparisons using another customer model or human responses? If so, do the relative system performance and benefits of clarification remain consistent?

For artifacts judged incorrectly extracted in the human audit, how do delivery scores change after correction? Could these corrections materially affect system rankings or the reported reference-level success rates?

3 comments
AuthorsOpenReview ↗

Response to Reviewer KWi5

We sincerely thank you for the careful review and concrete questions. Revisions are in blue.

W1. All systems interact with a single fixed customer model; the paper does not establish whether its inconsistencies materially affect requirement acquisition or scores, or whether conclusions persist with another simulator.

Answer: The revision tests both: replacing the nine affected runs changes any system's Quality by at most 0.004, and the five re-run systems keep their order under two other customer models (Q1, Q2).

Q1. For the 9–10 audited dialogues with cross-turn inconsistencies: did they concern requirements relevant to feasibility or scoring? Did affected agents receive incorrect or conflicting information that could explain delivery failures?

Answer: Of the nine dialogues both annotators marked, four concern such a requirement and five do not; the affected runs do not score significantly lower than their repeats (Appendix A.8).

  • In three of the four, the customer's last answer agrees with the business information: it corrected a wrong premise in the question, returned to the documented definition, or corrected a column name. In the fourth, the customer gave a default not in the business information and, once the agent reported it could not be met, replaced it with another, following the documented priority. The other five are about data availability: the agent asked for material the case lacks; the customer offered it, then withdrew it.
  • Effect on delivery: the nine runs do not score significantly lower than the other two repeats of the same system and case (sign test p=0.29p = 0.29; nine runs give little power); replacing them with the mean of those repeats changes any system's Quality by at most 0.004.

Q2. Do you have comparisons using another customer model or human responses? If so, do relative system performance and benefits of clarification remain consistent?

Answer: Yes, for two other customer models: the five systems keep their order and each still gains from clarification. We have no runs with human responses.

  • Relative performance: 600 new runs on 20 cases with DeepSeek-V4-Pro and Claude Opus 5 as customer. Quality per customer (new Section 5.5, Table 3; Hidden: Table 2 and Appendix A.6):

    SystemDeepSeek-V4-ProGLM-5.3 (main)Claude Opus 5Hidden
    GPT-6-Astra0.4790.5800.6580.180
    GLM-5.30.4310.5330.5900.170
    DeepSeek-V4-Pro0.4070.5060.5210.142
    Kimi-K30.3460.3690.3810.124
    Gemini-3.1-Pro0.2460.2910.2820.122
  • Benefit of clarification: each system beats its Hidden score under every customer, by 0.12–0.30 with DeepSeek-V4-Pro and 0.16–0.48 with Claude Opus 5.

  • Score levels largely follow how completely each customer model answers. In a replay of the same questions (Appendix A.6), Claude Opus 5 conveys 2–10 percentage points more of a case's requirements than GLM-5.3, and DeepSeek-V4-Pro 16–31 fewer, across agents. We chose GLM-5.3 before the experiments by a vote of five FDEs comparing candidate models' replies with the original customers' communication style. The replay is in line with this choice: when a question targets a requirement, GLM-5.3 (95.5%) and Claude Opus 5 (96.6%) convey it most often among the seven customer models (others 82.3–91.5%).

W2. The paper does not quantify how extraction errors change scores or rankings; extractor agreement cannot exclude shared interpretation errors; report scores before and after human correction on the audited subset.

Answer: The revision reports scores before and after correction on the audited subset (Q3).

  • Shared interpretation errors: agreed; the human audit, not extractor agreement, checks for them. Every audited run judged incorrectly extracted is corrected in Q3, whether or not its extractors agreed.

Q3. For artifacts judged incorrectly extracted in the human audit, how do delivery scores change after correction? Could this materially affect rankings or the reported reference-level success rates?

Answer: On the audited subset, little: no system loses more than 0.007 Quality or changes rank, and one reference-level run is lost (Appendix A.8).

  • Scores before and after: correction sets to zero every audited run that either annotator judged incorrectly extracted. Five runs change, from 0.42–1.00 to zero; the others already score zero. Five systems each lose 0.003–0.007 Quality (the top-ranked one: 0.5510 to 0.5481); 13 are unchanged.
  • Zero credit: the runs already at zero were recorded as invalid, and neither annotator judged any of their artifacts valid, so the audit found no valid artifact zeroed by an extraction error.
  • Reference-level success: one changed run had reached the reference (50 of 2,646 runs, not 51; 8 of 147, not 9, for the system with the most). Correction can only lower this rate, so reference-level delivery remains rare.

Thanks for the responses. Most of my concerns are being addressed. Since my original score is high (3), I will keep the score.

Dear Reviewer KWi5,

Thank you for your thoughtful reassessment and for considering our responses and revisions. We appreciate your time and constructive feedback, which have helped us improve the paper.

Sincerely,

The Authors

Reviewer_YwgnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

FDE-Bench is a benchmark for forward-deployed engineering (FDE) delivery: agents must (1) elicit tacit, unstated business requirements through dialogue with a simulated customer, and (2) iteratively optimize a deliverable (an optimization plan or an ML artifact) until it meets the project's acceptance criteria.

Critical strengths
  1. Coupling interactive requirement clarification with iterative optimization on real customer projects, with customer-accepted solutions as the delivery reference, is not covered by existing benchmarks. The Table 3 comparison is comprehensive and the distinguishing axis (real-project requirements + reference + clarification + execution) is credible.

  2. The target-retention criteria (unresolvable from visible inputs; answer changes feasibility/objective/acceptance; answer expressible as a fact the evaluator applies), explicit split/merge rules with boundary examples (Table 5), second-pass review against evaluator code, validity/difficulty/diversity selection (only projects with >1 month of FDE effort), and the per-case evaluator reference table together give a transparent, auditable data card per case.

  3. The diagnostics are informative and the null results are reported honestly. Notable findings: 78.5% of zero-score runs are constraint/feasibility violations (agents submit well-formed artifacts that break business rules); the Case-10 trace ("asked about a rule but did not enforce it," 347 violations, 327 units available in the local warehouse yet purchasing from the supplier) is a vivid, well-documented failure mode; mandatory clarification multiplies question count by 1.4–3.0 but yields no significant Quality change (Table 12); and the oracle nested-answer experiment (Figure 5) shows fragile, nonlinear value of information, a few missing requirements can void an otherwise good solution.

  4. The authors test the causal story they care about. The within-system/case Recall comparison, the information ablation, and the answer-coverage oracle all attempt to move beyond leaderboard rankings, and Appendix A.4 includes the capability-deconfounded Recall–Quality correlation (ρ=−0.31, p=0.22).

Critical weaknesses
  1. The simulated customer is an omniscient oracle reading a written file, which inverts the paper's central premise of tacit knowledge.

  2. The most direct validation of the simulator — replay against the original discussion transcripts — is missing, and the audit's agreement is weak where it matters.

  3. Statistical resolution at scale n=49 makes the headline metrics thin, especially DSR.

  4. The LLM extraction layer is a non-trivial measurement surface, and its error mode interacts with the zero gate.

Questions for authors

How many of the 2,646 runs ended with no extractor majority (no recorded score)? What is the leaderboard under a single deterministic extractor vs. the five-model panel?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Ywgn

We sincerely thank you for the careful reading and constructive comments. Revisions are in blue.

W1. The simulated customer is an omniscient oracle reading a written file, which inverts the premise of tacit knowledge.

Answer: The customer holds the full information by design, but the submission did not explain this clearly (clarified in Section 4.1).

  • By tacit we mean unstated, not unknown: in the original projects the practitioner knew the rule, did not think to state it, and gave it once the FDE asked. The simulated customer plays that practitioner, so the benchmark measures whether the agent works out what to ask, not whether the customer knows the answer.

W2. Replay against the original transcripts is missing, and the audit's agreement is weak where it matters.

Answer: The transcripts are the customers' confidential business material and cannot be released, so we cannot report a verifiable replay; we instead check the simulated customer against the record of what the discussions established. The low κ\kappa comes from almost all answers receiving the same label. Under two other customer models, five re-run systems keep their order.

  • The record: each case's business-information file, written by the FDE responsible for the original delivery. Two annotators checked 1,303 customer answers against this file and labeled 95.9% and 90.5% as supported and 0.2% and 0.5% (2 and 7 answers) as contradicted.
  • Audit agreement: the annotators agree on 92.1% of answers, and on 99.6% of the 1,170 that neither marked uncertain (Table 17). κ\kappa is 0.385 because it subtracts the agreement expected by chance, which is high when both label over 90% as supported. The prevalence- and bias-adjusted κ\kappa (PABAK) is 0.881 (now in Section 4.3).
  • Other customer models (new Section 5.5): with DeepSeek-V4-Pro or Claude Opus 5 as the customer (600 new runs), five opencode systems keep their order. Re-sending 3,456 main-run question messages to six other customer models keeps the ranking of agents by requirements obtained stable (pairwise Kendall τ≥0.71\tau \geq 0.71).

W3. With n=49 the headline metrics are thin, especially DSR.

Answer: We rank systems by Quality, and 49 cases resolve the larger Quality gaps. Reference-level delivery is rare for every system, so DSR values differ by only a few runs; we use DSR to show this rarity, not to rank systems.

  • Quality: every run receives a graded score, so a system's Quality is the mean of 147 runs. In a paired bootstrap over the 49 cases, the 95% interval of the gap excludes zero for every pair more than 0.08 apart, and for all 60 pairs between the 12 systems at 0.499–0.551 and the 5 at 0.387–0.420. Gaps of 0.04 or less are not resolved (Section 5.2; Appendix A.4). 49 cases is in line with open long-horizon agent benchmarks (CO-Bench 36 tasks, Frontier-Eng 47, MLE-bench 75).
  • DSR: only 51 of 2,646 runs (1.9%) reach the reference. With so few, gaps of one or two runs are expected: the four highest systems have 9, 8, 7 and 6 of 147, all inside the Wilson 95% interval of the highest value, [3.3%, 11.2%] (now in Section 5.2).
  • More cases: we will keep adding real customer projects under the same review (Section 7).

W4. The LLM extraction layer is a non-trivial measurement surface, and its error mode interacts with the zero gate.

Answer: We agree that extraction is an important measurement step, so no score comes from a single model: a score is recorded only when at least three of up to five extraction models agree.

  • Stability: any single extractor's score differs from the panel score on 14–66 of 2,646 runs (Table 19).
  • Zero gate: since three extractors must agree, one extractor's error cannot set a run to zero. Errors shared by several extractors are checked by the human audit of 120 runs, which found no valid artifact zeroed by an extraction error; extraction accuracy is 97.9% (reweighted to all runs).

Question. Runs with no extractor majority; leaderboard under a single deterministic extractor vs. the five-model panel.

Answer: No run went unscored for lack of a majority, and the leaderboard under a single extractor is nearly the same as the panel's.

  • Runs with no recorded score: none. All 2,643 runs that submitted an artifact have at least three agreeing extractor scores; the other 3 submitted no artifact and score zero (Appendix A.8).
  • Single-extractor leaderboard: the artifacts are free-form files, so no rule-based parser covers all cases; the closest baseline scores each run with one extraction model's score from the main evaluation. Where that model has none (route skipped, failed, or no artifact), we use the panel score: 6–267 runs per extractor. Under each extractor, 140–148 of the 153 system pairs keep their panel order (Kendall τ\tau 0.830–0.935), and every pair that swaps is under 0.021 apart in panel Quality. No system's Quality moves by more than 0.03 (Tables 19–20).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We sincerely thank the three reviewers for their careful reading and constructive comments. This comment lists the strengths they noted and what the revision adds; the details are in our response to each reviewer. Each item names the reviewers whose comments it addresses. Changes in the revised PDF are in blue.

Strengths noted by the reviewers

  • Real-project grounding. Ywgn: "Coupling interactive requirement clarification with iterative optimization on real customer projects, with customer-accepted solutions as the delivery reference, is not covered by existing benchmarks"; KWi5: "Real-project grounding creates a meaningful evaluation setting"; 9jdb: "Connecting requirement elicitation to executable delivery makes the benchmark a useful complement to tasks with fully specified instructions".
  • Auditable case construction. Ywgn: "a transparent, auditable data card per case".
  • Diagnostic value. Ywgn: "The diagnostics are informative and the null results are reported honestly"; KWi5: "This makes the benchmark diagnostically useful".
  • Extensive evaluation. 9jdb: "The evaluation is extensive: 18 configurations, three repeats per task, paired information ablations, and human audits of customer answers, clarification judgments, and artifact extraction".

New experiments and analyses

  • Other customer models (9jdb, KWi5, Ywgn). In 600 new runs on 20 cases with DeepSeek-V4-Pro or Claude Opus 5 as the customer, the five opencode systems we re-ran keep the order they have under GLM-5.3. Each stays above its Hidden score (request and data only, no customer answers) under every customer, so each still gains from clarification (new Section 5.5 and new Table 3; the tables that follow are renumbered). When 3,456 main-run question messages are replayed to six other customer models, the ranking of agents by requirements obtained stays stable (pairwise Kendall τ≥0.71\tau \geq 0.71), and no customer model measurably favors agents built on its own model family (Appendix A.6).
  • Cross-turn inconsistencies of the customer (KWi5). In nine of the 105 audited dialogues, both annotators found a cross-turn inconsistency. These runs do not score significantly lower than the other two repeats of the same system and case (sign test p=0.29p = 0.29; nine runs give little power), and replacing them with the mean of those repeats changes any system's Quality by at most 0.004 (Section 4.3; Appendix A.8).
  • Extraction (Ywgn, KWi5). No run went unscored for lack of an extractor majority. When every run is scored with one extractor alone (the panel score where that extractor has none), 140–148 of the 153 system pairs keep their panel order (Kendall τ\tau 0.830–0.935; new Tables 19–20). On the audited subset, setting to zero every run that either annotator judged incorrectly extracted lowers no system's Quality by more than 0.007 and changes no rank (Appendix A.8).

Clarifications

  • Tacit knowledge and the customer (Ywgn). Section 4.1 now states that the customer holds the full case information by design: the requirements we target are rules that the practitioner knew but did not state.
  • Audit agreement (Ywgn). Section 4.3 now reports the prevalence- and bias-adjusted κ\kappa (0.881) next to Cohen's κ\kappa for the customer-answer audit.
  • Statistical resolution with 49 cases (Ywgn). Section 5.2 now states in the main text which Quality gaps a paired case bootstrap resolves (every gap above 0.08, none of 0.04 or less; details in Appendix A.4) and adds the Wilson 95% interval of the highest DSR. We use DSR to show that reference-level delivery is rare, not to rank systems.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.