- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors build a new benchmark to assess whether coding agents sufficiently retrieve all requirements about a requested delivery when given an unclear task and the opportunity to interact with a "user" (represented by another model with hidden information). They claim that their results demonstrate models consistently fail to fetch all hidden requirements from a simulated user.
Critical strengths
- The overall idea of measuring whether coding models can extract missing intended details makes sense and is an important metric, given that coding agents are widely used and not provided with concrete and detailed specifications in practice.
Critical weaknesses
- The writing is very difficult to understand. For example
- There is no motivation or take away for any of the experimental results (except the main results).
- Many terms are used before introduction (eg "L0", L.191, RQ1-RQ3).
- There are many signs of AI writing, and overly detailed, terse text that does not help explaining the content (for example the construction example in L.205 ff)
- The experimental results are very unclear.
- The source of the dataset is ominous ("from pseduonomized users") - did the authors collect these from human AI interactions? GitHub? Synthesize?
- The human validation might be biased as it is not clarified if the annotators are also authors.
- It is not ablated how much variance the used code and user agents introduce nor whether their particular choice has a large effect on the results.
- Is the user agent even faithful to a human interacting with the model?
- Are there any instances where the produce model asks "Give me the full list of hidden requirements?" and does the user model reply to this?
- Concerning RG, are the consistently missing 5-8% in the two datasets ldue to a capability limitation of the coding model?
- What happens to RG and DC when the producer model is not limited by the capability of the coding model but can implement the code as well?
- I don't see a review of whether the extracted atomic requirements are meaningful/useful/realistic. For example, the construction example mentions ", ''great product''" being graded as an exact string that has to be present, this appears almost completely unrelated to the initial user request.
- As mentioned, for many side results (RQ2 and onwards) it is unclear what their purpose is.
- No reports on statistical significance.
Questions for authors
- Did you ablate the user agent and coding model? What is the effect of using a different model here? What harnesses were used?
- What is the relationship between the authors and the annotators if any? What is the qualification of the annotators (CS degree or similar) if any?
- Are the results statistically significant? What is the variance between reruns of the coding agent on the same brief?
- Please refer to the questions raised in the second critical weakness.
- What are RQ1-4 if formulated as questions? What is the purpose of the results presented in RQ2-4?
11 comments
Question 1: Who were the annotators, and how were they related to the authors?
Answer 1. The two original annotators were non-authors, did not participate in task construction, and had no prior knowledge of this project. They were dedicated data annotators in an enterprise setting with computing backgrounds. Each audited 90 construction packets; the separate 360-pair recovery-judgment study blinded them to Producer identities, panel votes, and each other's labels. We will make these qualifications and study roles explicit.
The construction audit covers source grounding, temporal separation, visibility and channel policies, hint quality, and reference validity. Both annotators examined the same 90 packets, including a deliberately risk-enriched subset. Source-grounding compliance is 100.0%/94.4%, and channel-assignment compliance is 96.6%/95.8%. These are within-task fractions averaged across tasks, not percentages of tasks passing every check or population estimates.
On the separate 360 requirement–brief pairs, the three-judge majority panel agrees with the two humans on 91.9%/90.8%, with precision 95.8%/91.0%; human–human agreement is 95.0%. This checks recovery judgments, not whether every reference is the source user’s unique intention. The simulated pilot and all new model audits are separate from this human evidence.
Question 2: Are the atomic requirements relevant, useful, and realistic?
Answer 2. The new audit supports relevance and usefulness under model assessment, while identifying alternatives and possible overconstraint. It is LLM-only: three judges assessed 90 requirements from 30 tasks, one per recovery route per task.
| Majority judgment on 90 sampled requirements | Count |
|---|---|
| Relevant to the task | 90/90 |
| Useful for specifying the behavior | 89/90 |
| Admits a reasonable alternative | 77/90 |
| Possible test overconstraint | 9/90 |
Source support was judged present for 28/30 history targets, 11/30 clarification targets, and 4/30 defaults. Sampling equally across routes makes this a diagnostic audit, not an estimate weighted by the benchmark’s route frequencies. Relevance does not imply a unique user-validated choice. We will distinguish historical preferences from constructed decisions and report the overconstraint flags.
Separately, restricting evaluation to clarification/history hidden requirements gives IR/DC/RIR rank correlations of 0.989/1.000/0.995 on single-file tasks and 0.995/0.940/0.989 on repositories. DC also excludes surface targets; RIR remains undefined when no selected target was recovered. Model ordering largely survives removal of defaults. This sensitivity result does not resolve the nine overconstraint flags or turn the model audit into human validation.
Question 3: Why is the exact string “, great product” tested?
Answer 3. The string is the expected result of a specified transformation. Requirement r3 removes configured blacklist phrases case-insensitively and normalizes whitespace:
| Component | Value |
|---|---|
| Input title | In conclusion, great product |
| Configured blacklist phrase | In conclusion |
| Behavior under test | Remove the phrase case-insensitively; normalize whitespace |
| Expected title | , great product |
The comma remains because the rule removes the blacklist phrase; removing punctuation would be a separate behavior. Thus the test checks a transformation of supplied input, rather than requiring an unrelated slogan. We will replace the compressed example with this input–rule–output explanation and its requirement-to-test link.
Question 4: What changes when the User Simulator backend changes, and which harness is used?
Answer 4. Simulator choice changes scores and some model orderings, while incomplete recovery and delivery persist. We ran 270 L0 episodes: three Producers × three simulators × 30 tasks (20 single-file, 10 repository), with Kimi-K2.7-Code as coder. The models and results appear below.
Tasks were selected from the original 90-task audit pool using task-family and hidden-requirement information, not model scores. We reran the original simulator as an anchor for paired comparisons under the same supplementary protocol. These are full workflows: an answer can affect later questions, the brief, and review decisions.
We used the original JSON-action harness, environment, Code Agent, and orchestration interfaces. Limits were 40 total actions, 18 gathering actions, 14 simulator questions shared across gathering and review, 10 profile/history lookups, six example executions, and four revisions. Agents had no hidden-test access. Producer output was capped at 6,000 tokens and coder output at 16,000/24,000 for single-file/repository tasks.
Across 12 paired simulator changes, DC differences range from −7.48 to +11.35 percentage points (pp), and IR from −12.38 to +3.85 pp. The highest configuration means are 35.77%/33.40% IR and 45.55%/46.86% DC on single-file/repository tasks. The recovery and delivery shortfall persists; model ordering is not backend-independent. IR uses complete three-judge panels under the final atomic_batches_v2 protocol, with full brief context and the same entailment rule; earlier protocol attempts are excluded. Each table cell is IR / DC (%). All rows use Kimi-K2.7-Code.
| Track / Producer | DeepSeek-V4-Flash simulator | Gemini-3.5-Flash simulator | GPT-5.5 simulator |
|---|---|---|---|
| Single-file / DeepSeek-V4-Flash | 33.87 / 38.16 | 26.05 / 39.18 | 31.18 / 35.13 |
| Single-file / Gemini-3.8-Flash | 33.50 / 39.43 | 29.86 / 45.55 | 35.77 / 44.84 |
| Single-file / Gemini-3.5-Flash | 7.39 / 25.21 | 8.88 / 29.61 | 8.30 / 17.73 |
| Repository / DeepSeek-V4-Flash | 33.40 / 36.66 | 21.02 / 40.88 | 27.26 / 35.84 |
| Repository / Gemini-3.8-Flash | 26.84 / 35.51 | 22.53 / 46.86 | 30.69 / 39.86 |
| Repository / Gemini-3.5-Flash | 10.28 / 37.26 | 10.36 / 32.49 | 7.75 / 33.02 |
Question 5: Does asking for all hidden requirements reveal the target?
Answer 5. Broad requests occur and can reveal multiple requirements. A three-model audit of 60 keyword-enriched dialogues found 32 broad questions: 25 disclosed at least one requirement and nine disclosed several. This sampling establishes occurrence, not corpus-wide prevalence. A controlled study generated 1,080 answers (30 tasks × three simulators × six question types × two calls). The first repetition was content-audited. The original simulator refused explicit “all hidden requirements” and natural “all remaining rules” requests on all 30 tasks, yet one refusal disclosed a requirement. Refusal labels therefore do not establish nondisclosure. These content audits used the original three-model panel, not human annotators.
The one-target matching rule limits interaction-scoring credit, not the information in an answer. A Producer can use everything disclosed when writing its brief. We will clarify that distinction and report the observed multi-requirement disclosures.
Question 6: Is the simulator faithful to a human user?
Answer 6. The simulator is a controlled specification-answering interface; human behavioral fidelity has not been established. A model supplied with the reference has access to information that a real user may recall, express, or negotiate differently.
The 120 additional LLM roleplay answers cover ten tasks, two roleplayer models, three question types, and two calls, with model audits of the first repetition. They test another configuration, not human fidelity. Confirmation probes restating defaults or history also do not demonstrate discovery of unknown preferences. Because instructions and interaction configurations differ, effects cannot be assigned to one prompt phrase. We will keep these model-only controls separate from the original human audits.
Dear Reviewer 8834:
Thank you for your careful reading and for your detailed questions about the data, human audit, simulator, coder, and experimental presentation. We address the weaknesses first, then answer the specific questions with the completed controls and their implications.
ProdBench links requirements communicated before coding to behavior in the delivered software. IR measures hidden-requirement recovery in the brief, DC delivery completeness, RIR realization conditional on recovery, and RG the change in DC through revision.
Weakness 1: The dataset source and meaning of the reference requirements are unclear
Response 1. The tasks originate from human–AI software-development conversations, not GitHub issues or wholly synthetic requests. Source users opted into the model service's data-use agreement. Model-assisted construction converts these interactions into self-contained Python tasks with frozen requirements and tests; neither those requirements nor simulator replies constitute subsequent confirmation by the original users.
The benchmark contains 300 Python tasks derived from interactions associated with 101 pseudonymized users: 200 single-file tasks and 100 repository tasks. A source interaction supplies the task kernel; construction supplies a stable callable contract, atomic requirements, reference implementation, and executable tests. Profile/history evidence must precede the source interaction and exclude that interaction and related sessions. History-based implications require support from at least two earlier sessions. These restrictions are designed to keep the source interaction and related solution context out of the admissible history.
“Pseudonymized” describes how user identity is represented. It does not mean that the source requests were generated as fictional conversations. The release excludes raw source conversations, identifying traces, and source-session metadata, while retaining de-identified task context needed for evaluation. We will make this source-to-task transformation explicit in Section 3.1 and point to Appendix B.2 for source evidence and B.3 for the two-pass target drafting procedure.
Weakness 2: Coder limitations and review effectiveness are conflated
Response 2. Delivery depends on the coding configuration. Our transfer experiments measure this dependence; the feedback audit in Answer 9 separately evaluates the Producer’s diagnosis. Holding the coder fixed in the original benchmark controls the backend, rather than removing its influence.
We additionally evaluated two alternative coder configurations on the original 90-task subset. Frozen-brief replay uses 13 Producers and two coders, yielding 2,340 implementations. Initial-DC rank correlations with the original coder are 0.956/0.758 for DeepSeek-V4.1-Flash and 0.923/0.588 for Gemini-3.8-Flash (single-file/repository). Complete-workflow runs with the DeepSeek configuration cover six Producers and 540 L0 episodes, yielding final-DC rank correlations of 0.943 [0.486, 1.000] on single-file tasks and 0.143 [−0.657, 0.600] on repositories.
The replay holds briefs fixed; the full workflow permits changed briefs, feedback, and artifacts. The alternative coder caps are 24,000/49,152 output tokens, compared with the original reported 16,000/24,000, and the full-workflow runs allow 20,000 Producer output tokens while retaining the action quotas. Consequently, these are transfers across declared configurations rather than isolated causal estimates of coder identity. They establish stronger transfer on single-file tasks and leave repository ranking stability unestablished. The same-model experiment described in Answer 8 uses its own 30-task design and must not be pooled with these 90-task results.
Weakness 3: The writing introduces terms too late and obscures experimental motivation
Response 3. We will introduce the workflow and terms before the experimental details, and give each subsection an explicit question and takeaway. The organizing sequence will be gathering information, writing a specification, implementing it, and reviewing the result. Answer 11 sets out RQ1–RQ4 and the purpose of their analyses; Answer 3 gives a clearer replacement for the construction example.