- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors propose a method for structured analysis of language models' ability to first find relevant requirements in software projects, second, reconstruct the state of the requirements (not their solution/implementation),third, update the requirements if needed, and implement requested changes. Fourth, they evaluate their method on chat and repository traces obtained from Upwork by first running an LLM on a filtered subset of the whole chat history of a project (an earlier LLM pass filters out social/admin-only messages first) to find out the requirements, title, and description; afterward, they run language models on each of these requirements separately to build their atomic requirement history - a log of the changes to each requirement. Repository traces only exist for a curated subset of projects/targets, not for every project. This is then saved as a dataset. They evaluate various products from Anthropic and OpenAI by prompting them with a project chat and a request and asking, first, for the touched requirements, measuring precision and recall; second, what the respective requirements' state is; third, if the requirement needs an update; and fourth, for an implementation, now given the code, measuring the predefined tests' outcome.
Critical strengths
The authors address a very important topic in this field and many software engineers' concerns. This is very important because coding agents are writing code increasingly often, and it is an open question what quality the code has, how one could quantify the code's quality, and how to improve it. They use a formalized, structured model, separating requirement selection, state reconstruction, and implementation so one can localize where an agent fails, to approach the problem and develop a potential benchmarking tool. This could be a major improvement because AI companies have gotten increasingly good at improving on benchmarks, and since this is a missing benchmark, it would give an important lever for them to pull to improve their products.
Critical weaknesses
The paper is unusual to read. The paper's forward references, the late definition of the research questions, and the late descriptive figures alongside the formalism made it hard for the authors to tell a compelling story. Furthermore, the captions do not describe the figures well, and the text does not clearly describe them, further increasing the authors' difficulty in clearly presenting their results. Reviewing their reproduction package, I found that the human review is skipped in all cases. None of the proposed gates are actually used in the study. Furthermore, it is ambiguous what a requirement actually is if there were never documented requirements apart from a chat. Since nobody actually read it, it is not clear what a requirement is or whether they are atomic, nor is it clear whether they are non-overlapping. Here, an LLM got a filtered part of the chat, created some requirements, and then, with only shallow, title-level knowledge of the other requirements, parallel searches were run on all requirement chats, and requirement histories were created. The pipeline looks compelling, but since it's not evaluated as described, it's impossible to know whether it actually works. In the end, an LLM created the tests that another LLM - using the authors' method - is tested on.
Questions for authors
Can you clarify that humans reviewed the histories, despite your logs stating otherwise? How did the process went and with what measure did you quantify that the requirements are distinct and correct?