acceptodds
Under review as a conference paper at ICLR 2027

ProdBench: Evaluating Requirement Recovery and Realization in Producer-oriented Software Development

Abstract

Real software requests leave crucial decisions unstated. We introduce PRODBENCH, a benchmark for evaluating an LLM Producer in agentic software development. A Producer turns an underspecified user request into an explicit implementation brief and oversees the development and validation of the resulting software. In our controlled protocol, the Producer gathers prior context, questions a fixed User Simulator, and directs a fixed Code Agent through implementation and revision. PRODBENCH contains 300 source-grounded Python tasks from 101 pseudonymized users, with 8,273 atomic requirements and 12,918 executable tests. Three matched request views vary the initial information while preserving each task’s target. Five metrics measure requirement recovery, delivery completeness, preservation of recovered intent, targeted clarification, and revision gains. Two annotators audit construction quality and validate recovery judgments, with agreement between the panel and each annotator of 91.9% and 90.8%. Across 13 Producers, delivery completeness exceeds a request-only coder using the same backend by 19.9–38.7 percentage points on single-file tasks and 10.1–22.3 on repository tasks. Yet richer context improves recovery far more than delivery. These results identify a persistent gap between recovering requirements and realizing them in software.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Agentic Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8834OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors build a new benchmark to assess whether coding agents sufficiently retrieve all requirements about a requested delivery when given an unclear task and the opportunity to interact with a "user" (represented by another model with hidden information). They claim that their results demonstrate models consistently fail to fetch all hidden requirements from a simulated user.

Critical strengths
  • The overall idea of measuring whether coding models can extract missing intended details makes sense and is an important metric, given that coding agents are widely used and not provided with concrete and detailed specifications in practice.
Critical weaknesses
  • The writing is very difficult to understand. For example
    • There is no motivation or take away for any of the experimental results (except the main results).
    • Many terms are used before introduction (eg "L0", L.191, RQ1-RQ3).
    • There are many signs of AI writing, and overly detailed, terse text that does not help explaining the content (for example the construction example in L.205 ff)
  • The experimental results are very unclear.
    • The source of the dataset is ominous ("from pseduonomized users") - did the authors collect these from human AI interactions? GitHub? Synthesize?
    • The human validation might be biased as it is not clarified if the annotators are also authors.
    • It is not ablated how much variance the used code and user agents introduce nor whether their particular choice has a large effect on the results.
    • Is the user agent even faithful to a human interacting with the model?
    • Are there any instances where the produce model asks "Give me the full list of hidden requirements?" and does the user model reply to this?
    • Concerning RG, are the consistently missing 5-8% in the two datasets ldue to a capability limitation of the coding model?
    • What happens to RG and DC when the producer model is not limited by the capability of the coding model but can implement the code as well?
    • I don't see a review of whether the extracted atomic requirements are meaningful/useful/realistic. For example, the construction example mentions ", ''great product''" being graded as an exact string that has to be present, this appears almost completely unrelated to the initial user request.
    • As mentioned, for many side results (RQ2 and onwards) it is unclear what their purpose is.
    • No reports on statistical significance.
Questions for authors
  • Did you ablate the user agent and coding model? What is the effect of using a different model here? What harnesses were used?
  • What is the relationship between the authors and the annotators if any? What is the qualification of the annotators (CS degree or similar) if any?
  • Are the results statistically significant? What is the variance between reruns of the coding agent on the same brief?
  • Please refer to the questions raised in the second critical weakness.
  • What are RQ1-4 if formulated as questions? What is the purpose of the results presented in RQ2-4?
11 comments

Dear Reviewer 8834:

Thank you for your careful reading and for your detailed questions about the data, human audit, simulator, coder, and experimental presentation. We address the weaknesses first, then answer the specific questions with the completed controls and their implications.

ProdBench links requirements communicated before coding to behavior in the delivered software. IR measures hidden-requirement recovery in the brief, DC delivery completeness, RIR realization conditional on recovery, and RG the change in DC through revision.

Weakness 1: The dataset source and meaning of the reference requirements are unclear

Response 1. The tasks originate from human–AI software-development conversations, not GitHub issues or wholly synthetic requests. Source users opted into the model service's data-use agreement. Model-assisted construction converts these interactions into self-contained Python tasks with frozen requirements and tests; neither those requirements nor simulator replies constitute subsequent confirmation by the original users.

The benchmark contains 300 Python tasks derived from interactions associated with 101 pseudonymized users: 200 single-file tasks and 100 repository tasks. A source interaction supplies the task kernel; construction supplies a stable callable contract, atomic requirements, reference implementation, and executable tests. Profile/history evidence must precede the source interaction and exclude that interaction and related sessions. History-based implications require support from at least two earlier sessions. These restrictions are designed to keep the source interaction and related solution context out of the admissible history.

“Pseudonymized” describes how user identity is represented. It does not mean that the source requests were generated as fictional conversations. The release excludes raw source conversations, identifying traces, and source-session metadata, while retaining de-identified task context needed for evaluation. We will make this source-to-task transformation explicit in Section 3.1 and point to Appendix B.2 for source evidence and B.3 for the two-pass target drafting procedure.

Weakness 2: Coder limitations and review effectiveness are conflated

Response 2. Delivery depends on the coding configuration. Our transfer experiments measure this dependence; the feedback audit in Answer 9 separately evaluates the Producer’s diagnosis. Holding the coder fixed in the original benchmark controls the backend, rather than removing its influence.

We additionally evaluated two alternative coder configurations on the original 90-task subset. Frozen-brief replay uses 13 Producers and two coders, yielding 2,340 implementations. Initial-DC rank correlations with the original coder are 0.956/0.758 for DeepSeek-V4.1-Flash and 0.923/0.588 for Gemini-3.8-Flash (single-file/repository). Complete-workflow runs with the DeepSeek configuration cover six Producers and 540 L0 episodes, yielding final-DC rank correlations of 0.943 [0.486, 1.000] on single-file tasks and 0.143 [−0.657, 0.600] on repositories.

The replay holds briefs fixed; the full workflow permits changed briefs, feedback, and artifacts. The alternative coder caps are 24,000/49,152 output tokens, compared with the original reported 16,000/24,000, and the full-workflow runs allow 20,000 Producer output tokens while retaining the action quotas. Consequently, these are transfers across declared configurations rather than isolated causal estimates of coder identity. They establish stronger transfer on single-file tasks and leave repository ranking stability unestablished. The same-model experiment described in Answer 8 uses its own 30-task design and must not be pooled with these 90-task results.

Weakness 3: The writing introduces terms too late and obscures experimental motivation

Response 3. We will introduce the workflow and terms before the experimental details, and give each subsection an explicit question and takeaway. The organizing sequence will be gathering information, writing a specification, implementing it, and reviewing the result. Answer 11 sets out RQ1–RQ4 and the purpose of their analyses; Answer 3 gives a clearer replacement for the construction example.

Question 1: Who were the annotators, and how were they related to the authors?

Answer 1. The two original annotators were non-authors, did not participate in task construction, and had no prior knowledge of this project. They were dedicated data annotators in an enterprise setting with computing backgrounds. Each audited 90 construction packets; the separate 360-pair recovery-judgment study blinded them to Producer identities, panel votes, and each other's labels. We will make these qualifications and study roles explicit.

The construction audit covers source grounding, temporal separation, visibility and channel policies, hint quality, and reference validity. Both annotators examined the same 90 packets, including a deliberately risk-enriched subset. Source-grounding compliance is 100.0%/94.4%, and channel-assignment compliance is 96.6%/95.8%. These are within-task fractions averaged across tasks, not percentages of tasks passing every check or population estimates.

On the separate 360 requirement–brief pairs, the three-judge majority panel agrees with the two humans on 91.9%/90.8%, with precision 95.8%/91.0%; human–human agreement is 95.0%. This checks recovery judgments, not whether every reference is the source user’s unique intention. The simulated pilot and all new model audits are separate from this human evidence.

Question 2: Are the atomic requirements relevant, useful, and realistic?

Answer 2. The new audit supports relevance and usefulness under model assessment, while identifying alternatives and possible overconstraint. It is LLM-only: three judges assessed 90 requirements from 30 tasks, one per recovery route per task.

Majority judgment on 90 sampled requirementsCount
Relevant to the task90/90
Useful for specifying the behavior89/90
Admits a reasonable alternative77/90
Possible test overconstraint9/90

Source support was judged present for 28/30 history targets, 11/30 clarification targets, and 4/30 defaults. Sampling equally across routes makes this a diagnostic audit, not an estimate weighted by the benchmark’s route frequencies. Relevance does not imply a unique user-validated choice. We will distinguish historical preferences from constructed decisions and report the overconstraint flags.

Separately, restricting evaluation to clarification/history hidden requirements gives IR/DC/RIR rank correlations of 0.989/1.000/0.995 on single-file tasks and 0.995/0.940/0.989 on repositories. DC also excludes surface targets; RIR remains undefined when no selected target was recovered. Model ordering largely survives removal of defaults. This sensitivity result does not resolve the nine overconstraint flags or turn the model audit into human validation.

Question 3: Why is the exact string “, great product” tested?

Answer 3. The string is the expected result of a specified transformation. Requirement r3 removes configured blacklist phrases case-insensitively and normalizes whitespace:

ComponentValue
Input titleIn conclusion, great product
Configured blacklist phraseIn conclusion
Behavior under testRemove the phrase case-insensitively; normalize whitespace
Expected title, great product

The comma remains because the rule removes the blacklist phrase; removing punctuation would be a separate behavior. Thus the test checks a transformation of supplied input, rather than requiring an unrelated slogan. We will replace the compressed example with this input–rule–output explanation and its requirement-to-test link.

Question 4: What changes when the User Simulator backend changes, and which harness is used?

Answer 4. Simulator choice changes scores and some model orderings, while incomplete recovery and delivery persist. We ran 270 L0 episodes: three Producers × three simulators × 30 tasks (20 single-file, 10 repository), with Kimi-K2.7-Code as coder. The models and results appear below.

Tasks were selected from the original 90-task audit pool using task-family and hidden-requirement information, not model scores. We reran the original simulator as an anchor for paired comparisons under the same supplementary protocol. These are full workflows: an answer can affect later questions, the brief, and review decisions.

We used the original JSON-action harness, environment, Code Agent, and orchestration interfaces. Limits were 40 total actions, 18 gathering actions, 14 simulator questions shared across gathering and review, 10 profile/history lookups, six example executions, and four revisions. Agents had no hidden-test access. Producer output was capped at 6,000 tokens and coder output at 16,000/24,000 for single-file/repository tasks.

Across 12 paired simulator changes, DC differences range from −7.48 to +11.35 percentage points (pp), and IR from −12.38 to +3.85 pp. The highest configuration means are 35.77%/33.40% IR and 45.55%/46.86% DC on single-file/repository tasks. The recovery and delivery shortfall persists; model ordering is not backend-independent. IR uses complete three-judge panels under the final atomic_batches_v2 protocol, with full brief context and the same entailment rule; earlier protocol attempts are excluded. Each table cell is IR / DC (%). All rows use Kimi-K2.7-Code.

Track / ProducerDeepSeek-V4-Flash simulatorGemini-3.5-Flash simulatorGPT-5.5 simulator
Single-file / DeepSeek-V4-Flash33.87 / 38.1626.05 / 39.1831.18 / 35.13
Single-file / Gemini-3.8-Flash33.50 / 39.4329.86 / 45.5535.77 / 44.84
Single-file / Gemini-3.5-Flash7.39 / 25.218.88 / 29.618.30 / 17.73
Repository / DeepSeek-V4-Flash33.40 / 36.6621.02 / 40.8827.26 / 35.84
Repository / Gemini-3.8-Flash26.84 / 35.5122.53 / 46.8630.69 / 39.86
Repository / Gemini-3.5-Flash10.28 / 37.2610.36 / 32.497.75 / 33.02

Question 5: Does asking for all hidden requirements reveal the target?

Answer 5. Broad requests occur and can reveal multiple requirements. A three-model audit of 60 keyword-enriched dialogues found 32 broad questions: 25 disclosed at least one requirement and nine disclosed several. This sampling establishes occurrence, not corpus-wide prevalence. A controlled study generated 1,080 answers (30 tasks × three simulators × six question types × two calls). The first repetition was content-audited. The original simulator refused explicit “all hidden requirements” and natural “all remaining rules” requests on all 30 tasks, yet one refusal disclosed a requirement. Refusal labels therefore do not establish nondisclosure. These content audits used the original three-model panel, not human annotators.

The one-target matching rule limits interaction-scoring credit, not the information in an answer. A Producer can use everything disclosed when writing its brief. We will clarify that distinction and report the observed multi-requirement disclosures.

Question 6: Is the simulator faithful to a human user?

Answer 6. The simulator is a controlled specification-answering interface; human behavioral fidelity has not been established. A model supplied with the reference has access to information that a real user may recall, express, or negotiate differently.

The 120 additional LLM roleplay answers cover ten tasks, two roleplayer models, three question types, and two calls, with model audits of the first repetition. They test another configuration, not human fidelity. Confirmation probes restating defaults or history also do not demonstrate discovery of unknown preferences. Because instructions and interaction configurations differ, effects cannot be assigned to one prompt phrase. We will keep these model-only controls separate from the original human audits.

Reviewer_E3pXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ProdBench, a benchmark for evaluating how LLMs within a fixed Producer Agent framework perform in requirement recovery and realization in software development. Existing benchmarks primarily rely on end-to-end code correctness, making it difficult to distinguish failures in requirement discovery from failures in implementation. To address this limitation, ProdBench separates the responsibilities of the Producer and Code Agent, enabling stage-wise evaluation of requirement recovery and realization. The benchmark constructs 300 tasks from real-world software development conversations, decomposing explicit and hidden requirements into testable requirements. Each task includes a target specification, a reference implementation, and executable tests, along with three task views (L0–L2) with different levels of information. The experiments evaluate 13 Producer LLMs, using metrics such as IR, RIR, DC, and ReviewGain to assess requirement recovery, requirement realization, and revision effectiveness. The paper also conducts extensive validation of benchmark reliability. However, I think further investigation is needed into whether the target specifications fully reflect actual user intent and whether multiple reasonable default behaviors may exist.

Critical strengths
  1. This paper separates the evaluation of the Producer Agent from that of the Code Agent, focusing on the Producer’s ability to understand and recover requirements, review code, and guide iterative revisions, rather than solely assessing the correctness of the final code.

  2. The paper evaluates both task construction quality and scoring reliability. Automated tests and human audits are used to assess source grounding, requirement annotation quality, and reference implementation correctness. In addition, agreement between multiple LLM judges and human annotations is examined to validate the reliability of requirement recovery scoring.

  3. The paper constructs three user request views (L0, L1, and L2) with varying levels of information for the same task while keeping the Gold Requirements unchanged, enabling a systematic analysis of how information availability affects the Producer's requirement recovery and final delivery performance.

Critical weaknesses
  1. The benchmark relies on predefined Gold Requirements, potentially omitting alternative valid default behaviors. It also does not directly assess additional requirements in the Brief, which may be incorrect or conflict with existing requirements.

  2. ProdBench assumes a fixed Producer–Code Agent workflow, limiting its direct applicability to software development agents with different architectures, such as single-agent or dynamically coordinated multi-agent systems. For example, the benchmark uses a single fixed Code Agent, without evaluating Producers across different LLMs or Code Agents, so Producer performance may depend on the specific coder's capabilities and compatibility.

  3. Although ReviewGain measures improvements in code quality resulting from code review and iterative revisions, it is influenced by the Producer's error diagnosis, revision guidance capabilities and the Code Agent's ability to execute revisions, making it difficult to independently assess the Producer's Delivery Control capabilities.

  4. The paper introduces certain terms and workflow stages, such as Tracks, Gold, and Both Phases, without sufficient context, making it difficult for readers to follow. Furthermore, in the main-text description of task construction, the authors state, “A second pass identifies further behavioral decisions,” but provide no explicit reference to the appendix where this process is detailed.

Questions for authors

see weaknesses

3 comments

Dear Reviewer E3pX:

Thank you for your detailed assessment and for recognizing the value of separating requirement recovery from implementation. We address reference ambiguity, dependence on the coding configuration, review diagnosis, and terminology in the four responses below.

Weakness 1: The reference may omit valid defaults and additional brief requirements

Response 1. A frozen reference specifies the behavior an agent should elicit and implement; alternative behaviors may still be reasonable. Default-routed decisions can be queried through specification-grounded simulator answers (Appendix C.4). In a new three-LLM audit, 343 of 344 sampled defaults had complete panels. A strict majority judged 233/343 (67.9%) ambiguous. Another 34 received three different labels; assigning these ties conservatively to ambiguity gives 267/343 (77.8%). The rubric excludes fixed identifier and error-string spellings from behavioral ambiguity. We report both aggregation rules because the conservative count includes disagreements. These are model judgments, not original-user validation.

The comparative question is whether these default choices drive the model ordering. We tested this by restricting evaluation to the other recovery routes.

Using the 11,700 recorded episodes, we recomputed Intent Recovery (IR), Delivery Completeness (DC), and Recovered Intent Realization (RIR) on clarification- and history-routed hidden requirements, excluding all 2,789 defaults. The correlations with the original 13-Producer rankings are:

Restricted metricSingle-fileRepository
IR0.9890.995
DC1.0000.940
RIR0.9950.989

Here DC is also restricted to the selected hidden targets; surface requirements are excluded. RIR is undefined when no selected target was recovered and is omitted for that metric rather than replaced with zero. Model ordering is largely preserved after defaults are removed. This result supports the comparative use of the benchmark beyond default matching; it does not certify the uniqueness or human endorsement of every remaining reference behavior.

Additional brief requirements

We directly audited brief additions. Across 155 L0 briefs covering all 13 Producers, each of three LLM judges assessed up to 15 additions per brief for conflicts with the reference. Majority agreement on the implicated requirement identified an average of 2.65 distinct reference requirements per brief. The counting unit is a distinct reference requirement implicated in a conflict, counted once per brief when at least two judges flag it. It is not a count of erroneous additions and cannot be divided by the average number of extracted additions to obtain an error rate. The extraction cap also means this is not an exhaustive inventory of every extra decision.

This measures brief-side inconsistency separately from execution: DC reflects a conflict only when the implemented behavior violates its bound tests. We will report this audit alongside the reference-sensitivity analysis and clarify that reference coverage alone does not evaluate every additional decision.

Weakness 2: Producer performance may depend on the coding configuration

Response 2. IR evaluates the brief before implementation; delivery scores describe the declared Producer–coder configuration. We tested transfer on the same 90-task subset (60 single-file, 30 repository), using L0 requests and the same hidden tests:

ExperimentProducersNew implementations/episodesDC rank correlation: single-file / repository
Frozen-brief replay with DeepSeek-V4.1-Flash131,170 implementations0.956 / 0.758
Frozen-brief replay with Gemini-3.8-Flash131,170 implementations0.923 / 0.588
Complete workflow with DeepSeek-V4.1-Flash6540 episodes0.943 / 0.143

The replay keeps the previously recorded brief fixed while changing the implementation configuration, asking whether brief-quality comparisons transfer to a different implementer. The complete-workflow study allows the Producer to interact with the alternative coder throughout implementation and review, so it includes changes in feedback and revision behavior. Replay therefore compares initial artifacts, whereas complete-workflow runs compare final artifacts; the two studies cannot be pooled into one estimate of a coder effect. The single-file replay correlations have 95% task-bootstrap intervals [0.797, 0.984] and [0.725, 0.967]. Complete-workflow intervals are [0.486, 1.000] for single-file and [−0.657, 0.600] for repository. The evidence supports transfer of single-file comparisons across these configurations; stable repository ordering is not established. We will state this track-specific conclusion explicitly.

The alternative configurations used coder output limits of 24,000/49,152 tokens (single-file/repository), versus the original reported 16,000/24,000. Complete-workflow runs also allowed 20,000 Producer output tokens while retaining the action quotas. These are configuration-transfer tests, not an isolated estimate of coder identity. The intervals use 2,000 paired task-bootstrap draws and are pointwise, without simultaneous coverage guarantees for all model orderings. In particular, the low repository correlation cannot be dismissed as only an issue of sample size.

The original fixed coder provides a controlled comparison under one declared backend. The transfer tests show where that comparison generalizes across the tested configurations: single-file ordering transfers strongly, whereas repository ordering remains configuration-dependent. We will report the tracks separately.

Applicability to other agent architectures

The executable delivery target also supports evaluation without a separate Producer. An interactive direct coder with the same information interfaces and action quotas completed 270 episodes across L0–L2, achieving DC 29.4%/31.3%. It gathers context and implements without a separate Producer model or distilled brief. This demonstrates one single-agent design. DC requires a delivered artifact; IR/RIR additionally require a pre-implementation brief. Applying recovery metrics to dynamically coordinated architectures would require exposing a comparable specification at that stage. A description inferred afterward from the finished code would not measure what was communicated before implementation. We will distinguish the demonstrated single-agent evaluation from this broader extension.

Weakness 3: ReviewGain mixes Producer diagnosis with coder execution

Response 3. ReviewGain measures net delivery change under the stated coder. To assess the Producer’s contribution more directly, we evaluated whether its feedback identifies currently failing requirements, then tracked subsequent repair. Diagnosis and implementation can diverge: useful feedback may not be executed, unmentioned failures may be repaired, and revisions may introduce regressions.

On the 60-task single-file L0 subset spanning all 13 Producers (780 episodes), reconstruction recovered 885 revision steps; 871 had complete three-LLM diagnosis panels. Against the currently failing test-bound requirements, majority-labeled diagnosis recall ranges from 4.7% to 28.4% across Producers. Subsequent repair occurred for 437/1,299 identified failure requirement-steps (33.6%), versus 710/9,917 other failure requirement-steps (7.2%). These are conditional associations, not a causal decomposition: identified and unidentified failures may differ in difficulty.

Diagnosis and available improvement headroom

Diagnosis recall evaluates the feedback itself using majority model judgments, independently of whether the next artifact implements that feedback successfully. The repair rates describe the next artifact. They separate observable stages but do not assign causal credit.

We also computed RG* = [DC(A_F) − DC(A_0)] / [1 − DC(A_0)] for episodes with positive initial headroom. Its model rankings correlate with raw ReviewGain at 0.879/0.868 (single-file/repository). RG* expresses net gain relative to available headroom; it does not remove coder dependence or all difficulty effects. We will present diagnosis recall, conditional repair, and net gain as distinct measurements.

Weakness 4: Terms and construction details need clearer introduction

Response 4. We will define “track” at first use as the single-file or repository partition, replace “gold” with “frozen reference,” and replace “both phases” with “evidence gathering and implementation review,” explicitly stating that they share the question quota. The second-pass construction sentence will point to Appendix B.3, “Two-pass target drafting,” in the submitted PDF. A terminology table will define Producer, brief, episode, recovery source, and L0–L2, with main-text pointers to the new analyses.

The revised workflow introduction will explain that the Producer gathers context and writes a brief, then reviews the coder’s artifacts and requests revisions. L0 supplies a sparse request and callable contract; L1 names unresolved dimensions; L2 adds qualitative guidance while preserving the target. These definitions will precede the experiments, whose introductions will state the question being tested and the comparison used to answer it.

Reviewer_b17UOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PRODBENCH, a benchmark for evaluating an LLM Producer in agentic software development. A Producer is defined as an agent that turns an underspecified user request into an explicit implementation plan and oversees the development process. The Producer gathers prior context, asks questions of a fixed User Simulator, and directs a fixed Code Agent through implementation and revision.

Critical strengths
  • Paper addresses an important and underexplored problem: turning underspecified user requests into explicit implementation briefs.

  • Paper provides comprehensive evaluation metrics and a controlled evaluation protocol.

Critical weaknesses
  • There is a budget mismatch between the Producer workflow and the request-only coding baseline.

  • The benchmark's ground-truth requirements are largely constructed from historical interactions with LLM assistance, rather than directly verified with the original users.

  • The Recovered Intent Realization metric may introduce selection bias. Intent Recovery (IR) may favor verbose Producers.

Questions for authors

Major:

(Q1) How many hidden requirements are supported by user evidence versus introduced design decisions? Could the original users validate a representative sample?

(Q2) Could the authors compare against budget-matched iterative coding agents without an explicit Producer or requirement-recovery stage?

Minor:

(Q3) Could the authors discuss metrics mentioned in the Weaknesses?

5 comments

Dear Reviewer b17U:

Thank you for your careful review and for highlighting the importance of requirement recovery and controlled evaluation. We address the budget comparison, the provenance of the reference, and the interpretation of the recovery metrics below.

Weakness 1: The request-only comparison gives the workflows different budgets

Response 1. On the paired 90-task audit subset, all 13 Producer configurations achieve higher mean Delivery Completeness (DC) than an interactive, iterative coder with matched action allowances. We evaluated two baselines using the paper's Kimi-K2.7-Code backend, simulator, execution environment, and grader:

  • A: self-revision coder, with up to six self-authored example executions and four revisions.
  • B: interactive coder, additionally given the Producer's information sources and allowances: 10 profile/history lookups, 14 simulator questions, 18 gathering actions, and 40 total actions. It implements from the gathered context without a separate Producer or distilled implementation brief.

Both use the corresponding Producer action caps, output limits, and timeouts. Each runs on 60 single-file and 30 repository tasks at all three request views: 270 episodes per baseline, 540 in total. Producer comparisons use the same task–view pairs. DC is the fraction of atomic requirements satisfied by the final artifact.

WorkflowSingle-file DC (%)Repository DC (%)
Request-only coder24.623.9
A: self-revision coder24.031.0
B: interactive coder29.431.3
Producer configurations, range across 13 models44.4–61.832.7–46.4

The comparison keeps the task–view pairs and executable target fixed. L0 supplies the sparse request and callable contract; L1 identifies unresolved dimensions; L2 supplies qualitative guidance. These are three information conditions for each task, rather than three stochastic repeats. The table is computed on the same 90-task subset for every workflow, so its values should not be compared with full-benchmark averages as if their task distributions were identical.

Giving the direct coder access to interaction and revision therefore does not eliminate the observed delivery gap. For example, Kimi-K3 as Producer reaches 48.5%/40.1%, with paired gains of 19.1/8.9 percentage points over B. These are means of task–view differences before rounding. All 13 configurations have higher mean DC on this subset; this statement does not assert that every individual model–baseline contrast is statistically separated. This comparison matches action allowances; it does not establish equal token cost or isolate a causal effect of role separation from model choice. We will add these baselines and explicitly distinguish action budgets from measured or estimated token usage.

Weakness 2: Constructed references may differ from the original users' intentions

Response 2. The reference combines historical evidence and constructed decisions; original-user endorsement has not been measured. Its 7,377 hidden requirements have recorded recovery routes: 640 profile/history (8.7%), 3,948 clarification (53.5%), and 2,789 constructed defaults (37.8%). These are routes for obtaining the target behavior, not a partition into user-verified versus invented requirements. History-derived implications require support from at least two earlier sessions. Clarification answers are constructed reference answers, not responses collected retrospectively from the source users; defaults fix engineering conventions (Appendix B).

These counts do not establish a user-supported versus author-introduced split: adding history and clarification targets would not give an original-user validation rate. Original-user validation has not been conducted. The existing human evidence is the independent two-annotator audit of 90 construction packets: mean source-grounding compliance is 100.0%/94.4%, and channel-assignment compliance is 96.6%/95.8% (Table 3A). Both annotators were non-authors, did not participate in task construction, and were unfamiliar with the project beforehand. The sample deliberately includes construction risks. It checks source evidence and construction rules, rather than estimating endorsement by the source users.

The evaluation target is explicit: all systems face the same frozen requirements and tests, with information interfaces that provide routes to the reference behavior. This measures whether systems recover and implement a declared specification, even when other choices would be reasonable in an unconstrained setting. We will explain this distinction in the construction section.

Model rankings largely persist when constructed defaults are excluded. From the 11,700 original episodes, we recomputed Intent Recovery (IR), DC, and Recovered Intent Realization (RIR) using only clarification- and history-routed hidden targets. This restricted DC also excludes surface requirements; RIR remains conditional on a nonempty recovered set. Rank correlations with the original metrics are:

Restricted metricSingle-file Spearman ρRepository Spearman ρ
IR0.9890.995
DC1.0000.940
RIR0.9950.989

Thus the model ordering is largely preserved under this target restriction. We will use “frozen reference” consistently and make the distinction between recovery routes, construction audits, and user confirmation explicit.

Weakness 3: RIR selection effects and IR sensitivity to verbosity

Response 3. RIR needs a common-target sensitivity analysis because its denominator depends on recovery; IR needs a content control to test whether irrelevant length earns credit. We completed both analyses.

RIR selection effects

RIR measures preservation of recovered requirements, not overall task completion. IR measures hidden requirements expressed in the brief; DC measures all atomic requirements satisfied by the final artifact. RIR restricts both numerator and denominator to recovered hidden requirements. High RIR can therefore accompany low recovery, while DC can credit behavior supplied by the coder but absent from the brief. The three metrics distinguish these outcomes.

To test selection sensitivity, we evaluated every model on the same targets recovered by at least 7 of the 13 Producers. These sets average 12.8/18.7 requirements per task–view; realization rankings correlate with original RIR at ρ = 0.984/0.940. The all-13 intersection averages only 1.9/3.5 targets, including empty intersections, and is nonempty on 274/600 single-file and 134/300 repository task–view pairs. On those nonempty intersections, ρ = 0.945/0.637. Thus ordering is stable on majority-common targets but less stable on the strict repository intersection. We will report both analyses and their set sizes.

IR sensitivity to verbosity

Generic padding does not improve mean IR in the completed stress test. The scoring rule requires an atomic requirement to be entailed by the brief; generic advice does not specify task behavior. Of 130 sampled briefs, 123 have complete three-judge results for both manipulated variants. Missing judgments are excluded. Original scores come from frozen records; the variants received additional model judgments. Approximately doubling brief length with irrelevant engineering boilerplate changes mean IR from 58.2% to 57.2%; a generic best-practices brief of approximately matched length scores 0.0%. Across 13 Producers, mean length and IR have Spearman ρ = −0.01. This tests generic padding, not every verbosity strategy. The separate human audit reports panel precision of 95.8%/91.0% against the two annotators on 360 requirement–brief pairs (Table 3B). We will report the padding control alongside that validation.

Question 1: How many hidden requirements have user support, and can original users validate a sample?

Answer 1. The 640 history/profile, 3,948 clarification, and 2,789 default targets describe recovery routes, not endorsement counts. History implications require earlier-session evidence; clarification answers and defaults are construction-derived. Original-user validation has not been conducted. Response 2 gives the construction audit and the ranking sensitivity after defaults are excluded; neither is a substitute for original-user confirmation.

Question 2: Can an iterative coding agent receive matched budgets without a separate Producer?

Answer 2. Yes. Baselines A and B each completed 270 episodes on the same 90 tasks and three request views. B gathers information and implements without a separate Producer or distilled brief. All 13 Producer configurations achieve higher mean DC on both tracks. As detailed in Response 1, action allowances are matched; equal token cost and a causal effect of role separation alone are not established.

Question 3: How should the metric concerns be addressed?

Answer 3. We will report RIR jointly with IR and DC, add both common-target analyses, and state their eligibility rules. For IR, the 123 complete paired cases show that irrelevant padding earns no mean recovery gain. Response 3 retains the weaker result on the strict repository intersection and limits the verbosity conclusion to the tested manipulation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overll Summary

Dear Reviewers, Area Chair, Senior Area Chairs, and Program Chairs,

Thank you for the careful reviews and discussion. We revised the manuscript in response to all three reviews and marked the key changes in blue.

Highlights of ProdBench

An implementation brief may omit a requirement, or the code may fail to implement a requirement the brief conveys. In ProdBench, a Producer gathers context, writes the brief, and reviews the coder’s work. We score the brief recorded before coding and test the delivered code against the same reference.

  1. Requirement-level evidence. ProdBench contains 300 Python tasks, 8,273 test-linked requirements, and 12,918 executable tests. Each hidden requirement has an intended recovery route through prior context, clarification, or an engineering default. The release includes requirement-level judgments and test outcomes for the original evaluation.
  2. Matched requests with a fixed executable target. Each task has a sparse request, a request that names the unresolved decisions, and a request that provides qualitative guidance. All three use the same requirements and tests, so comparisons vary the initial information without changing what counts as a correct implementation.
  3. Recovery and delivery improve at different rates. Across 13 Producers, moving from the sparse request to qualitative guidance increased hidden-requirement recovery in the brief by 29.1–43.6 percentage points and delivery completeness by 5.8–17.3 points. On repository tasks, mean realization of hidden requirements already expressed in the brief ranged from 37.9% to 54.0% across Producers (§4.2, Table 3).

Strengths recognized in the reviews

b17U and 8834 highlighted the importance of evaluating how agents recover missing requirements. b17U also noted the controlled protocol and breadth of metrics. E3pX highlighted the separation between requirement recovery and implementation, the matched request views, and the audits of task construction and recovery scoring. In the separate recovery-scoring study, the three-model panel agreed with the two independent annotators on 91.9% and 90.8% of 360 sampled requirement–brief pairs, respectively (§4.1).

Addressing the major concerns

  1. b17U — a direct-coder comparison with matched action allowances. The interactive coder can look up history, ask questions, run examples, and revise its code. On the same 90 tasks, all 13 Producer configurations achieved higher mean delivery completeness on both tracks (§4.2). We also clarify the origin of reference requirements (§§3.1–3.2), test realization on common sets of requirements, and show that padding briefs with irrelevant advice changed mean recovery from 58.2% to 57.2% (§4.1). b17U thanked us for addressing their comments and maintained their weak-accept rating.
  2. E3pX — reference sensitivity and the scope of the workflow results. E3pX thanked us for the response and asked us to “revise the manuscript accordingly.” The main text addresses all four points. We audit ambiguous defaults and conflicts from added instructions. Restricting scoring to hidden requirements routed to history or clarification largely preserves the original rankings (Spearman ρ = 0.940–1.000, §4.1). When we reran the full workflow under an alternative coding configuration, the six Producers’ delivery-rank correlations were 0.943 on single-file tasks and 0.143 on repositories (§4.2, Table 4). We also include the interactive single-agent baseline, distinguish diagnosis in feedback from subsequent repair (§4.4), and define the workflow terms and point to the two-pass construction procedure (§§1, 3.1, 3.3).
  3. 8834 — traceable data provenance and independent source audit. Following the reviewer’s requests, §3.1 explains how 260 software-development conversations with an internal company AI tool yielded 300 tasks. Seven adapt self-contained problems and 293 were designed around a source goal or domain. The catalogue example puts a paraphrased source request next to the benchmark task and identifies the rule added during construction. Restricting the analysis to the 85 of 90 tasks whose core goals both non-author annotators traced to the source changed the best mean sparse-request recovery and delivery scores by about one percentage point or less on each track (§4.1). The revision explains the simulator comparisons and the relation between the experimental subsets (§4.3, Appendix P). After our second response, 8834 raised their rating from 1 to 2.

Thank you for considering the revised manuscript and the additional evidence.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.