- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper deals with investigating the quality of agent-generated code over its whole generation duration until the provided final patch. It is indeed interesting to see that the final outcome of agent-based code generation might be wrong whereas in intermediate steps this is not the case. The paper is recent because it focuses on agentic programming, which is a still under-investigated but hot topic of software engineering.
The paper also comes with an initial experimental evaluation, which comprises some agents and some programs from two (maybe connected) repositories. This is fine for an initial evaluation but maybe not enough for a final analysis. Anyhow, the outcome of the paper itself is fine and the final conclusions are justified with its content. The practical implications, however, remain somehow vague. If an agent does not provide a passing outcome, the programming is not done. Even when knowing that there might be an outcome at an intermediate step that fulfills all test cases. When saying this, obviously for providers of agentic coding tools that might make a difference.
To improve the paper, the authors should be more precise when explaining concepts. Everything should be defined in sufficient detail. The experiments could also be improved considering more and more diverse benchmarks.
Critical strengths
- The paper deals with a problem occurring when using agentic-coding tools
- The paper comes with an initial experimental evaluation, showing the existence of the problem
- There might be implications of the paper for agentic-coding tools provider
Critical weaknesses
- The paper is hard to read and understand.
- Some concepts and explanations need more precision and better (formal) definition
- The experimental evaluation is only an initial one.
- The consequences for practice could be discussed much more.
Questions for authors
There are no further questions.