- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
This paper evaluates whether, when given a sequence of queries from a simulated user, an LLM agent is able to continuously update the code without making breaking changes. The paper then quantifies the continuity and whether the LLM agent breaks the code after having subsequently fixed it.
Critical strengths
- The underlying idea is practically relevant: can the same model change a website without breaking it?
- The results are interesting, illustrating that agents break 53% of existing web apps after having already fixed them ( I assume that is what incumbent property refers to)
Critical weaknesses
- Presentation needs some work. A lot of the abstract/main contributions in the introduction sound impressive but don't mean much to me e.g. " ProductWebBench therefore scores a patch against the living system it lands in, not the target alone." is the final sentence of the abstract, it's unclear unless you dig deeper into the paper what this truly means, something that the final sentence of an abstract shouldn't do. Many of the concepts introduced in the introduction lack a clear definition and sound very AI-generated... I think it would be simpler to say something like "ProductWebBench therefore evaluates code changes in live systems, unlike prior systems
- It would be nice if bullet points in the introduction started on a new line.
- Overall, the method is too confusingly written for me to fully evaluate this paper.
- It is unclear how the user simulator is specified
Questions for authors
Where is the simulated user specified and evaluated? How does this process take existing tasks and break them into multi-turn chunks?