- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper takes 46 Python repository-generation tasks from DeNovoSWE (specification, hidden tests, images) and changes the protocol: the agent may call repo-feedback at any time to get the scalar hidden-suite pass rate, is forced to continue after voluntary exits, and runs until pass rate 1.0 or a token budget equal to six hours at the model's calibrated generation speed. Commits are replayed in an isolated evaluator to give a best-pass-rate trajectory over budget. The main metric measures how many tasks reach each pass-rate level, with higher levels weighted more, and how this grows over the budget. Twelve agent systems are each run once. The authors argue that early gains come from implementing the spec and later gains from feedback-guided repair, and that a single-pass run scores lower than the feedback loop. Model rankings also change with the required completion level.
Critical strengths
- The paper studies whether a coding agent keeps improving after its first self-declared completion, instead of stopping early. Forcing continuation and replaying every commit gives a progress curve over the whole run, and makes premature exit visible as a failure mode.
- The experiments cover twelve agent systems with isolated evaluators and network blocked, so upstream source cannot be retrieved. The analysis of gain composition, and of how rankings change with the target pass-rate threshold, is informative.
Critical weaknesses
- The protocol tunes on the test set. Offline scoring uses the same fixed test suite as online feedback, so there is no held-out split. Feedback calls are unlimited, which is like submitting again and again to fit a Kaggle public leaderboard, and the final score is optimistically biased. This setup also does not match real development. A developer, human or agent, does not hold a complete and certainly correct test suite as an oracle before writing the program, and does not develop with that suite as the optimization target.
- The budget analysis uses time as the unit, so the budget depends on serving speed. DSV4.1 Flash at 87 tok/s gets 1,879,200 tokens, Kimi K3 at 26.71 tok/s gets only 576,936. The same model on faster hardware would get more budget. The current comparison and rankings are therefore based on an unfair unit.
Questions for authors
- Beyond using the oracle test pass rate as a verified signal to guide repository generation, which is an unrealistic setup, what is the benchmark's difference and contribution compared to previous work such as ProgramBench?
- Section 3.1 states that the documentation-writing agent's prompt "explicitly requires test-relevant signatures, exceptions, return fields, constants, and defaults to be stated." If the NL specification already carries this information, why is the oracle test pass rate needed as the feedback signal? Why not use self-generated tests to steer the search?