- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes a benchmark to evaluate coding agents throughout the proposal-to-implementation process on large software systems. The authors design a data pipeline to collect high-quality modular development tasks, which involves enhanced proposals, matched PRs and augmented test suits. Extensive evaluations are performed on multiple combinations of models and harnesses.
Critical strengths
- The benchmark is challenging in terms of solution size.
- Extensive experiments are performed with different combination of models and harnesses.
Critical weaknesses
- For the verification of task completion, it is proposed to use the so-called system level public interfaces for F2P tests. But how can such tests be identified and ensured with completeness or sufficiency? The supp describes some enhancement of F2P tests, but it is still not clear why the agent could give a correct testable implementation if not given the exact interfaces. Besides, the completeness and accuracy of tests are not validated. This is particularly important for large-scale coding tasks that involve many modules and span over 2k LoC. It is shown that agents achieve large improvements when given specification. This also raise questions whether previous failures are mainly due to lack of clear interface defintions rather than drawbacks in model capabilities.
- It is unclear what are the conclusions of the assessment. The section 4.2~4.6 provide very limited information. Many paragraphs emphasies the same thing that localization is more of the bottleneck, which is already in wide agreement. The text describes the phenomenon and numbers but give little in-depth analysis and insights. Reading the experimental section makes the reviewer feel confused. What are the drawbacks of model capabilities? How can the assessment results guide the future research on enhancing the model?
- Can the data pipeline be used for scalable task generation so that one can use it for SFT/RL data production?
- Are the tasks verified by human experts to ensure their quality and solvability?
Questions for authors
Were the result numbers obtained with multiple runs for each model-harness combination?