- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
- Introduces a repository-generation benchmark using obfuscation, cross-language implementation, and unmet-specification tests.
- Builds tasks from diverse repositories and evaluates several coding agents.
Critical strengths
- The benchmark targets a meaningful gap in contamination-resistant end-to-end repository evaluation.
- The CLI contract and unmet-specification tests make the evaluation easy to interpret.
Critical weaknesses
- Task synthesis and test quality are central to validity, but the benchmark inherits the assumptions of its source repositories and generated tests.
- The score aggregates heterogeneous tasks, which may make cross-model differences hard to interpret.
Questions for authors
- How are generated specifications and unmet-specification tests independently checked for ambiguity or missing behavior?
- Will the benchmark and evaluation harness be released with fixed versions?
2 comments
Thanks for the response. After reading through the response, I have decided to maintain my original score.
Thank you for your review. Please find our response below.
W1: The benchmark inherits assumptions from both the source repositories and the generated tests.
We acknowledge this trade-off: the specifications originate from real-world repositories, while the tests are generated by an agent—both introduce assumptions into the benchmark. Our guiding principle is that these assumptions must either be explicitly stated in the instruction document (I) or be inferable from it, rather than remaining merely implicit within the tests. When generating T−, we require the agent to provide a justification for the inferability of each assertion—grounded in the author’s documentation, comments, issues, or interface invariants. Consequently, models are not required to guess conventions specific to the source repositories.
W2: Difficulty in interpreting performance differences across models due to task heterogeneity.
CommandBench focuses on whether a model can genuinely understand requirements and implement code accordingly. Table 2 reports T+ and T− separately, clearly distinguishing whether performance differences stem from core functional implementation or edge-case behavior. We provide the following additional statistics to demonstrate that model rankings are robust—insensitive to task difficulty, subject matter, or programming language—further confirming that performance gaps arise from the models' ability to comprehend requirements, rather than from the heterogeneity of the tasks themselves:
Q1: How are ambiguities or omissions in the specifications and T− independently verified?
The core idea is that any ambiguity or omission in a specification will inevitably lead to assertions that cannot be reliably inferred. We employ two independent checks to identify such issues:
Q2: Is it released as a fixed version?
Yes. Version 1.0 locks the tasks, documentation, evaluation environment, and evaluation harness; the specific commit for each task’s source repository is provided in Table 7. Any subsequent updates—such as regenerating obfuscated code, fixing errors, or adding/removing tasks—will result in a new, version-numbered release.
We hope this addresses your concerns.