acceptodds
Withdrawn from ICLR 2027

ReqMemBench: Benchmarking Coding Agents for Temporal Requirement-State Reconstruction in Evolving Projects

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Coding agents increasingly join ongoing software projects, where they must modify an evolving codebase in the context of a long-running client collaboration. Before acting, an agent must determine which requirements currently govern the task. This requires tracing how requirements have changed over time and determining which still apply, while distinguishing changes to the requirements themselves from evidence about their implementation. Existing coding and memory benchmarks offer limited visibility into whether agents correctly reconstruct the requirement state that should guide implementation. We introduce ReqMemBench, a benchmark for temporal requirement-state reconstruction built from real freelance project histories that have been rewritten for privacy and include a subset of role-played messages. At intermediate takeover points, ReqMemBench provides evidence-linked gold requirement states and evaluates four capabilities: selecting relevant historical requirements and evidence, reconstructing their pre-task states, updating requirements or identifying material clarification needs, and delivering the requested code changes. A two-phase protocol freezes history-based responses before agents choosing to act on eligible tasks receive repository access. Full History and Oracle Relevant History conditions assess the effect of history filtering and agents’ reconstruction accuracy when relevant evidence is explicitly supplied. Across seven configurations, agents over-select historical evidence and do not benefit from Oracle Relevant History; even the best RQ3 Post-State Score is only 0.1578, and no system passes one quarter of eligible RQ4 repositories. Reliable takeover therefore requires coherent temporal context and stronger update-or-clarify reasoning.

Primary area:
datasets and benchmarks
Keywords:
Coding Agents, Coding Benchmarks, Evolving Requirements, Agent Memory, Real-World Software Projects

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pKt3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a method for structured analysis of language models' ability to first find relevant requirements in software projects, second, reconstruct the state of the requirements (not their solution/implementation),third, update the requirements if needed, and implement requested changes. Fourth, they evaluate their method on chat and repository traces obtained from Upwork by first running an LLM on a filtered subset of the whole chat history of a project (an earlier LLM pass filters out social/admin-only messages first) to find out the requirements, title, and description; afterward, they run language models on each of these requirements separately to build their atomic requirement history - a log of the changes to each requirement. Repository traces only exist for a curated subset of projects/targets, not for every project. This is then saved as a dataset. They evaluate various products from Anthropic and OpenAI by prompting them with a project chat and a request and asking, first, for the touched requirements, measuring precision and recall; second, what the respective requirements' state is; third, if the requirement needs an update; and fourth, for an implementation, now given the code, measuring the predefined tests' outcome.

Critical strengths

The authors address a very important topic in this field and many software engineers' concerns. This is very important because coding agents are writing code increasingly often, and it is an open question what quality the code has, how one could quantify the code's quality, and how to improve it. They use a formalized, structured model, separating requirement selection, state reconstruction, and implementation so one can localize where an agent fails, to approach the problem and develop a potential benchmarking tool. This could be a major improvement because AI companies have gotten increasingly good at improving on benchmarks, and since this is a missing benchmark, it would give an important lever for them to pull to improve their products.

Critical weaknesses

The paper is unusual to read. The paper's forward references, the late definition of the research questions, and the late descriptive figures alongside the formalism made it hard for the authors to tell a compelling story. Furthermore, the captions do not describe the figures well, and the text does not clearly describe them, further increasing the authors' difficulty in clearly presenting their results. Reviewing their reproduction package, I found that the human review is skipped in all cases. None of the proposed gates are actually used in the study. Furthermore, it is ambiguous what a requirement actually is if there were never documented requirements apart from a chat. Since nobody actually read it, it is not clear what a requirement is or whether they are atomic, nor is it clear whether they are non-overlapping. Here, an LLM got a filtered part of the chat, created some requirements, and then, with only shallow, title-level knowledge of the other requirements, parallel searches were run on all requirement chats, and requirement histories were created. The pipeline looks compelling, but since it's not evaluated as described, it's impossible to know whether it actually works. In the end, an LLM created the tests that another LLM - using the authors' method - is tested on.

Questions for authors

Can you clarify that humans reviewed the histories, despite your logs stating otherwise? How did the process went and with what measure did you quantify that the requirements are distinct and correct?

Reviewer_T2u2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReqMemBench, a benchmark for testing whether coding agents can figure out what a project currently requires when they join it midway. Each project has a long history of messages and a corresponding code state, and requirements are modeled as atoms with attributes, lifecycle, scope, ambiguities, and implementation evidence. Requirement events are replayed to build temporal state graphs, which give gold pre- and post-task states at takeover points. The benchmark checks four things: picking relevant history and evidence, reconstructing the pre-task state, updating requirements or asking for clarification, and actually delivering code. It uses a two-phase setup where the agent first reasons without repository access, then gets the repo if it chooses to act, and compares full history with a filtered oracle history. The dataset has 51 projects, 859 requirement atoms, 2,793 events, and 349 takeover targets, with 37 eligible for code delivery. Across seven model-harness setups, agents tend to over-select history, don't improve with oracle history, and struggle to produce complete updates or good clarification questions, and none pass a quarter of the repo targets.

Critical strengths
  1. This work addresses a practical and underexplored setting: coding agents that join an ongoing project midway and must first determine which requirements currently apply before modifying code. The construction pipeline is a clear engineering contribution. It converts long project histories into Requirement Atoms and Events, replays them deterministically into temporal State Graphs, and derives task-local pre- and post-task gold states with evidence links. The two-phase protocol is also well designed, which makes it possible to score requirement reasoning and implementation as distinct stages.
Critical weaknesses
  1. The evaluation cannot attribute failure to specific stages. The four RQs are not causally isolated: an error in RQ2 propagates into RQ3, and an error in RQ3 propagates into RQ4. Low RQ4 PASS rates therefore cannot distinguish poor requirement understanding from poor code delivery. The C2 condition was meant to separate evidence-finding from state reconstruction, but performance drops under C2 across RQ2–RQ4, and the paper itself notes that C2 may remove surrounding context or change task organization. As a result, the headline claim about update-or-clarify reasoning is not supported: the results show agents perform poorly, but not which factor is responsible.

  2. The benchmark construction lacks sufficient detail. The paper does not specify how the 51 projects were selected, how the 44 source projects for the 349 targets were chosen, how repository environments were recovered, or how the 37 RQ4 executable targets were validated. Validator calibration is summarized only as “of 40 proposals, 37 satisfy this contract,” with no per-project breakdown. Without these details, it is hard to judge whether the RQ4 results reflect coding-agent capability or the particular selection of executable targets.

Questions for authors
  1. Could you clarify how the four RQs relate causally? Since an error in RQ2 propagates into RQ3, and an error in RQ3 propagates into RQ4, it is hard to tell whether low RQ4 PASS rates come from poor requirement understanding or poor code delivery. Would it be possible to add a diagnostic condition that isolates each stage?

  2. Could you provide more detail on how the benchmark was constructed? In particular, how were the 51 projects and 44 source projects selected, how were repository environments recovered, and how were the 37 RQ4 executable targets validated? A per-project breakdown of the validator calibration would also help assess whether the RQ4 results are representative.

Reviewer_GhwtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ReqMemBench, a benchmark for coding agents that take over evolving projects with long client-developer histories. The paper argues that an agent must determine not only which historical messages are relevant, but also which requirements currently govern the project, how they have changed, whether unresolved ambiguity requires clarification, and whether the task can be implemented correctly.

The authors define Requirement Atoms, annotate historical messages as state-changing Events, and deterministically replay these Events to construct pre-task and post-task Requirement States. The benchmark evaluates four stages: selecting relevant historical requirements and evidence, reconstructing the pre-task state, updating the state or asking for clarification, and delivering code. The first three stages use only history and the current request; the response is frozen before eligible ACT runs receive a repository for executable validation. The paper compares seven model-harness configurations under Full History and Oracle Relevant History conditions. It reports that agents over-select evidence, achieve low state reconstruction and post-state scores, do not improve with the Oracle Relevant History condition, and pass fewer than one quarter of the 37 executable RQ4 targets.

Critical strengths

The paper studies a meaningful problem. When a coding agent joins an ongoing project, the current request may be underspecified because relevant decisions are distributed across earlier client-developer communication. The distinction between a requirement still being in force and a historical claim about its implementation is useful. This motivates evaluation beyond standard issue-to-patch settings and is the paper's main conceptual contribution.

The RQ1 to RQ4 decomposition is also potentially valuable. It attempts to distinguish selecting historical evidence, reconstructing a pre-task state, updating requirements or identifying missing information, and producing code. The frozen Phase-A response makes the intended separation between historical reasoning and subsequent repository execution explicit. Further, RQ4 uses deterministic validators requiring build success, target-test success, and regression success. This is an important attempt to connect historical reasoning to executable downstream outcomes rather than evaluating only textual outputs.

The construction procedure includes model-assisted annotation, human review, deterministic replay, leakage checks, and validator calibration. These choices show that the authors have made a serious effort to create auditable benchmark instances. The paper also acknowledges several limitations, including semantic judgment in the state representation, the small executable subset, class imbalance, and correlated targets within projects. These strengths make the benchmark direction promising. However, the concerns below affect whether the reported scores measure realistic project takeover rather than performance under a highly specific representation and protocol.

Critical weaknesses
  1. The Phase-A protocol excludes repository access for RQ1 to RQ3. This is difficult to reconcile with the stated project-takeover setting. In practice, current code, tests, configuration, and documentation are important evidence for whether an earlier requirement was implemented, partly implemented, superseded, or still ambiguous. More importantly, the paper's Requirement State explicitly includes latest implementation evidence, but the agent is prevented from accessing the most direct implementation evidence until after its state reconstruction and ACT/CLARIFY decision are frozen. The resulting benchmark evaluates history-only requirement reconstruction, not full project takeover. A history-plus-repository condition, or a stronger justification that history alone suffices for all scored State fields, is necessary.

  2. The Requirement Atom and Requirement State gold targets depend heavily on the authors' custom representation and annotation choices. The authors first split natural-language histories into independently evolving Atoms, classify messages into Events, and then require each Atom to follow a fixed state-machine protocol with prescribed fields and transitions. This is operationally clear, but there is often no unique answer to whether a message creates a new requirement, modifies an old one, changes its scope, or should be treated as a separate requirement. Requirements can also split, merge, overlap, or be implicitly replaced. Deterministic replay guarantees consistency only after these semantic choices have been frozen. The paper does not report annotation agreement for Atom boundaries and Event types, sensitivity to alternative reasonable decompositions, or whether State scores predict real task success. Low RQ2 and RQ3 scores may therefore partly reflect difficulty following the authors' state-machine protocol and structured output convention rather than failure to understand the historical requirements.

  3. The paper does not separate historical requirement understanding from use of the proposed state-machine representation. Given a correct Atom/Event/action list, the paper's deterministic replay rules can compute the relevant pre-task and post-task States. Yet the evaluated agents must directly output complete G- and G+ States from history. A natural baseline would ask an LLM to predict the Atom/Event/action list from the history, then feed this predicted list, rather than the gold annotations, into the authors' deterministic replay rules to compute G- and G+. Extracting the Atom/Event list would remain a difficult semantic task; the rule-based replay would only remove the separate burden of manually maintaining the authors' prescribed State fields and transitions. This comparison would reveal whether the main bottleneck is understanding the history, extracting state changes, applying transitions, or following the authors' complex state-machine protocol and structured output convention. Without it, the paper cannot attribute poor RQ2/RQ3 performance to inability to reason about temporal requirements, nor support the claim that explicit state construction is the needed solution.

  4. The Oracle Relevant History result is over-interpreted. C2 is an author-constructed, gold-dependent subsequence of the history. It changes not only the amount of irrelevant material but also narrative continuity, contextual cues, input length, and how the model organizes the task. The fact that systems perform worse under C2 shows that this particular history packaging does not help. It does not show that explicitly supplying relevant evidence is unhelpful or that the remaining failure is purely state reasoning. The Discussion acknowledges that the cause of the decline is unidentified, but the abstract and conclusion emphasize the stronger Oracle-history claim.

  5. Data and selection transparency remain limited. The histories are privacy-transformed freelance records, with reconstructed or role-played messages and some reconstructed environments. AI tools assisted data transformation and annotation proposals. Human review is helpful, but the paper does not report annotator agreement, the proportion of transformed or role-played records, candidate-target filtering rates, or performance by provenance category. The selector also intentionally excludes context-free tasks and tasks that only report implementation evidence, yielding a stress test enriched for temporally dependent cases. This is a valid benchmark choice, but the claims should be limited to that selected task distribution.

  6. The executable RQ4 result should be interpreted cautiously. It contains only 37 highly filtered Gold-ACT targets. In addition, an agent that incorrectly selects CLARIFY in Phase A is not allowed to attempt code and is counted as an RQ4 failure. The reported PASS rate therefore combines historical understanding, state reconstruction, action selection, and code editing; it is not a pure repository-delivery measurement. The paper appropriately notes the small sample, but should provide uncertainty estimates and state this limitation more prominently in the headline conclusions.

Questions for authors
  1. Why is repository access excluded throughout RQ1 to RQ3, despite implementation evidence being part of the Requirement State? Can the authors report a history-plus-repository condition, or demonstrate that repository access does not materially alter pre-task State reconstruction and ACT/CLARIFY decisions?

  2. Please report the annotation process for Requirement Atom boundaries and Event types, including annotator count, independent annotation or review procedure, disagreement rates, adjudication, and sensitivity to alternative reasonable Atom decompositions.

  3. Can the authors evaluate a structured baseline that extracts Atom/Event changes from history and then applies the deterministic replay rules? This would help distinguish failures in historical understanding from failures in state representation, transition computation, and output formatting.

  4. Please provide a diagnosis of why C2 harms every evaluated system. In particular, can the authors compare multiple context-preserving C2 construction policies or report target-level analyses showing whether removed messages supplied necessary discourse context?

  5. Please report the proportion of transformed, reconstructed, and role-played messages and environments, candidate-target selection/filtering statistics, and results stratified by provenance and historical-dependence level.

Suggestions for future work:

  1. Compare direct history-only reasoning with iterative workflows that allow agents to inspect repositories, tests, and documentation before finalizing a requirement interpretation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.