acceptodds
Under review as a conference paper at ICLR 2027

CommandBench: Toward Diverse, Challenging, and Contamination-Resistant Repository Generation

Abstract

Coding agents are advancing quickly, and automating the entire software engineering process is becoming an increasingly important setting. Existing benchmarks, however, evaluate this setting only weakly: open-source repositories, the single most important source from which benchmarks are built, have been widely absorbed into the training corpora of frontier LLMs, and it is no longer possible to tell whether a high score reflects genuine ability or a replay of the training corpus. Having experts write tasks by hand does resist contamination effectively, but it runs into a clear bottleneck in scale and diversity. We therefore present CommandBench, which asks an agent to implement a complete repository from scratch and encapsulate it as a set of CLI commands. CommandBench is resistant to contamination in three ways: (i) lexical obfuscation, which makes a located repository unusable as it stands; (ii) cross-language implementation, which blocks verbatim copying; and (iii) unmet-specification tests, which target behaviour that the specification requires but that the original repository does not yet achieve. Most distinctively, CommandBench guarantees that an implementation that reproduces the source repository's behaviour scores exactly zero, so that faithfully replaying the source implementation earns no credit by construction. CommandBench is also markedly more diverse and more challenging: its tasks are synthesised from 138 distinct repositories, covering 7 programming languages, with tasks from outside computer science accounting for 48.6% of the benchmark. On CommandBench, models pass 89% of the met-specification tests on average, yet none passes more than 63% of the unmet-specification ones, and the best score is only 58.93 out of 100.

Primary area:
datasets and benchmarks
Keywords:
benchmark, coding agent, llm

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7UuHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary
  • Introduces a repository-generation benchmark using obfuscation, cross-language implementation, and unmet-specification tests.
  • Builds tasks from diverse repositories and evaluates several coding agents.
Critical strengths
  • The benchmark targets a meaningful gap in contamination-resistant end-to-end repository evaluation.
  • The CLI contract and unmet-specification tests make the evaluation easy to interpret.
Critical weaknesses
  • Task synthesis and test quality are central to validity, but the benchmark inherits the assumptions of its source repositories and generated tests.
  • The score aggregates heterogeneous tasks, which may make cross-model differences hard to interpret.
Questions for authors
  • How are generated specifications and unmet-specification tests independently checked for ambiguity or missing behavior?
  • Will the benchmark and evaluation harness be released with fixed versions?
2 comments

Thank you for your review. Please find our response below.

W1: The benchmark inherits assumptions from both the source repositories and the generated tests.

We acknowledge this trade-off: the specifications originate from real-world repositories, while the tests are generated by an agent—both introduce assumptions into the benchmark. Our guiding principle is that these assumptions must either be explicitly stated in the instruction document (I) or be inferable from it, rather than remaining merely implicit within the tests. When generating T−T^-, we require the agent to provide a justification for the inferability of each assertion—grounded in the author’s documentation, comments, issues, or interface invariants. Consequently, models are not required to guess conventions specific to the source repositories.

W2: Difficulty in interpreting performance differences across models due to task heterogeneity.

CommandBench focuses on whether a model can genuinely understand requirements and implement code accordingly. Table 2 reports T+T^+ and T−T^- separately, clearly distinguishing whether performance differences stem from core functional implementation or edge-case behavior. We provide the following additional statistics to demonstrate that model rankings are robust—insensitive to task difficulty, subject matter, or programming language—further confirming that performance gaps arise from the models' ability to comprehend requirements, rather than from the heterogeneity of the tasks themselves:

  • After excluding the 10% most difficult and 10% easiest tasks, the Spearman’s ρ\rho for the rankings of the 7 models is 0.964, indicating that the rankings are not skewed by extreme tasks.
  • When segmented by domain (Computer Science vs. other disciplines), the Spearman’s ρ\rho values ​​for model rankings are 0.857 and 0.786, respectively.
  • When segmented by programming language, the Spearman’s ρ\rho values ​​range from 0.643 to 0.929.

Q1: How are ambiguities or omissions in the specifications and T−T^- independently verified?

The core idea is that any ambiguity or omission in a specification will inevitably lead to assertions that cannot be reliably inferred. We employ two independent checks to identify such issues:

  • Automated verification: We task three different models with implementing the solution from scratch, relying solely on the provided specification. For test cases that passed the reference implementation but failed for all models, we tasked an independent agent with reviewing them to determine whether the issue lay with the specification or the test itself, and then made the necessary adjustments. This process was repeated for three rounds (Appendix A.2).
  • Manual spot-checking: Based on model performance, we identified a subset of assertions most likely to be unverifiable. Two annotators then independently provided their own answers for these; if either annotator’s answer disagreed with the assertion, the assertion was corrected. Out of 316 assertions spot-checked, only 12 (3.8%) required correction—a rate we demonstrate to be within an acceptable range (Appendix A.3).

Q2: Is it released as a fixed version?

Yes. Version 1.0 locks the tasks, documentation, evaluation environment, and evaluation harness; the specific commit for each task’s source repository is provided in Table 7. Any subsequent updates—such as regenerating obfuscated code, fixing errors, or adding/removing tasks—will result in a new, version-numbered release.

We hope this addresses your concerns.

Thanks for the response. After reading through the response, I have decided to maintain my original score.

Reviewer_k2JJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

CommandBench is a benchmark test for end-to-end code repositories. It addresses the dilemma where existing code benchmarks cannot simultaneously achieve "diversity, difficulty, and resistance to pollution". To solve this, it proposes a fully automated pipeline that converts 138 open-source repositories into 7 different languages and interdisciplinary new tasks. The core mechanism involves three aspects: lexical obfuscation, cross-language implementation, and "testing the boundary behaviors that the source repository specifications require but have not yet been implemented itself". This enables the model to score zero on boundary tests even when it memorizes the source code word-for-word. The score is calculated as the product of the pass rate of basic functions and the pass rate of boundary behaviors to prevent mutual compensation.

Critical strengths
  1. The evaluation objective was changed from "reproducing the behavior of the source repository" to "achieving the ideal behavior as specified",specifically tested the boundary cases that the source repository itself failed to handle, resulting in a zero score in the exact reproduction of mathematics - not as a post-event anti-cheating measure, but as a fundamental change in the definition of the evaluation.

  2. Previously, the good anti-contamination benchmarks relied on expert manual writing and only covered a few dozen repositories; CommandBench achieved 138 repositories, 7 languages, and nearly half non-CS disciplines through 8-stage automation, while the anti-contamination ability actually became stronger, proving that "task specifications that deviate from the source behavior" can be automatically derived rather than relying on expensive human labor.

Critical weaknesses
  1. Only 138 tasks, and the T⁻ boundary use cases for each task are automatically generated by the agent rather than being manually verified one by one. If there are erroneous assertions or biased use cases at this scale, they will directly skew the overall score, and it is difficult to dilute the result through multi-task averaging. The rationality of the result is questionable.

  2. The confusion is a one-time static name mapping. Once the task specifications are made public, subsequent model training corpora can directly absorb these confused commands and specifications, leading to lexical confusion and the gradual deterioration of cross-language anti-copying effects as the model iterates. The validity of the model is questionable.

Questions for authors
  1. How extensive were the manual inspections you conducted on the T⁻ assertion? What is the approximate error rate of the assertion? After removing the cases that were judged as incorrect or questionable, did the rankings and total scores of each model change significantly?

  2. How is the distribution of scores for each model across the 138 tasks? Is there a situation where a few extreme tasks (such as particularly easy or particularly difficult) dominate the overall scores? If the conclusions are split by discipline or language, are they consistent?

  3. Have you evaluated the effective lifespan of this benchmark? After the task specifications are made public, how many generations of model training are required for the confusion and cross-language anti-contamination effects to significantly diminish? Is there a plan to maintain the validity of the benchmark by regularly updating the confusion mappings or expanding the task set?

2 comments

Thank you for your review. Please find our detailed response below.

W1 / Q1: Verification and Error Rate of T⁻

We acknowledge that, given the large scale of T⁻ (5,899 test cases), we did not perform a manual, case-by-case verification of every item. However, we implemented rigorous verification measures through the following two approaches:

  • Automated Verification: We implemented the repository from scratch using three different models. Any test cases that failed across all three models were flagged for review by an independent agent; identified errors were subsequently corrected or removed. This process was repeated for three rounds (Appendix A.3).
  • Manual Sampling Verification: We categorized assertions that failed across all models as "high-risk." From this subset, we randomly sampled 316 assertions (21.1% of the high-risk group) and had two annotators independently evaluate them for errors (Appendix A.3). The results showed an error rate of only 3.8% (12 cases) within this group; these 12 instances were corrected before the dataset was finalized.

While we have made significant efforts, eliminating all potential errors is practically impossible. Therefore, the more critical question is: do any residual errors alter the model rankings?

  • The 3.8% error rate reported in Appendix A.3 applies specifically to the high-risk assertions. Even under the conservative assumption that this error rate applies to the entirety of T⁻, it remains well below the 20% error rate we simulated in Appendix A.5. In that simulation—where 20% of the assertions in T⁻ were randomly removed for each task—the standard deviation of the scores, σ(S), was 0.28. This is lower than the σ = 0.37 observed when the same model was evaluated three separate times. Furthermore, the Spearman rank correlation (ρ) was 0.994 (Table 5).
  • Each task contains at least 20 test cases from T⁻, and tasks are weighted equally in the final score. Consequently, a single erroneous test case would shift the total score by at most 5/138 ≈ 0.04 points.

Given these findings, we are confident that the rankings reported in the paper are reliable.

W2 / Q3: Obfuscation and Validity Period

We apologize for not clarifying this in the original manuscript: the obfuscation process is purely programmatic and can be regenerated at any time with minimal cost. It is applied to the released version solely for the sake of consistent comparison and distribution. We will explicitly state this in the revised manuscript.

More importantly, the "zero-score" guarantee in CommandBench does not rely on obfuscation; rather, it stems from how the evaluation targets are defined. In fact, even when the source repository is explicitly disclosed to the model—bypassing obfuscation entirely—the model's score actually decreases slightly.

Regarding the validity period, we believe this is difficult to predict, as it depends heavily on the rate at which model capabilities evolve. In terms of maintenance, we will continuously track the performance of frontier models, fix known issues, regenerate obfuscated content when necessary, retire tasks that have become "saturated" (i.e., too easy), and add more challenging tasks. Each update will be released with a specific version number.

Q2: Score Distribution and Breakdown

The score SS is calculated for each task individually and then averaged across all 138 tasks with equal weighting; thus, any single task contributes at most 100/138 ≈ 0.72 points to the total score. Furthermore, we provide the following detailed statistical analysis:

  • After excluding the 10% most difficult and 10% easiest tasks, the Spearman’s ρ\rho for the rankings of the 7 models is 0.964, indicating that the rankings are not skewed by extreme or outlier tasks.
  • When broken down by Computer Science versus other disciplines, the Spearman’s ρ\rho values ​​for model rankings are 0.857 and 0.786, respectively.
  • When broken down by target language, the Spearman’s ρ\rho values ​​range from 0.643 to 0.929.

As these results show, the model rankings remain largely consistent regardless of whether the data is categorized by discipline or language.

We hope this response addresses your concerns. If so, we would kindly ask you to consider adjusting your score.

Thanks for the response. After reading through the response, I find that my previous concerns remain unaddressed. Taking into account the comments from the other reviewers, I have decided to maintain my original score.

Reviewer_qrTvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CommandBench, a benchmark that evaluates the end-to-end repository generation capability of coding agents. This benchmark contains 138 tasks derived from open-source repositories, spanning seven programming languages and a wide range of disciplines. The main contribution of this paper is the contamination-resistant mechanism based on unmet-specification tests and a new evaluation form, which penalizes the behavior of simply reproducing the original repository. They evaluate multiple frontier models on the benchmark and conduct ablations to investigate the effectiveness of the proposed mechanisms.

Critical strengths
  • Data contamination is an important and practical issue when benchmarking the capability of the models. In this paper, they propose an interesting and novel approach to mitigate this, where they introduce unmet-specification tests to guarantee that the source-faithful replay receives a zero score.

  • The coverage of repositories is broader than existing benchmarks in languages and disciplines, which enables evaluation of models in underexplored domains.

  • This paper provides useful empirical analyses beyond the main leaderboard of benchmarking the frontier models. For example, they include ablations on cross-language implementation and lexical obfuscation, which can provide useful insights on how the proposed mechanisms affect the agents' behavior.

Critical weaknesses
  • In this paper, the unmet-specification suite is constructed from behaviors that the original repo fails to meet. My concern is that while this is central to the contamination-mitigation mechanism, it may introduce a systematic bias silently towards source-specific boundary failures. Therefore, it brings uncertainty about to what extent the performance on CommandBench reflects the general repo-generation capability, or the robustness against particular types of corner cases. A more detailed characterization of the unmet-specification cases and their required software engineering capabilities would help to strengthen the evaluation results.

  • In the proposed pipeline, LLM-based agents are extensively used for repo discovery, specification synthesis, wrapper construction, test generation, etc. However, it's not very clear how strongly the resulting benchmark depends on the capability and the choices of the models.

Questions for authors
  • Which specific LLMs and configurations are used at each construction and validation stage? How sensitive are the resulting benchmarks to the choice of models?
  • What are the rationales behind the standard for repository selection at stage 1? For example, what motivates the thresholds of 4000 lines of code and 100 Github stars?
  • Beyond language diversity and disciplinary coverage, how does CommandBench characterize diversity in the underlying software engineering capability required by the tasks? How do you validate whether the unmet-specification cases are not concentrated in particular categories of boundary failures?
2 comments

Thank you for your review. Our responses are below.

W1 / Q3: Is T⁻ biased toward source-specific boundary failures? What categories and capabilities does it cover?

Every T⁻ case is, by construction, a behaviour that the source repository fails to exhibit. However, the requirement behind a case does not come only from defects in the source code. It can also come from the authors' documented promises, from features that are not yet implemented, or from reasonable inference from the intended use. The table below gives this breakdown. The largest category accounts for less than one third of T⁻. The capabilities needed to pass T⁻ are also diverse: "Silently accepts invalid input" tests input validation and error handling; "Breaks a documented promise" and "Violates a weaker documented property" test adherence to documentation and standards; "Internally inconsistent" tests the internal consistency of the implementation. We also note that the main metric of CommandBench is S rather than T⁻, and general implementation ability is largely checked by T⁺. The revised paper will discuss T⁻ in more detail.

CategoryCasesShare
Silently accepts invalid input1,82030.9%
Breaks a documented promise1,42524.2%
Internally inconsistent1,22620.8%
Acknowledged but unfixed bug4006.8%
Inferred from intended use2894.9%
Violates a weaker documented property2243.8%
Other5158.7%
Total5,899100%

W2 / Q1: Which models and configurations were used for construction, and how sensitive is the benchmark to this choice?

All construction stages use Claude Opus 5, which is not among the evaluated models. We chose it because we wanted the strongest available model to ensure the quality of what the pipeline generates. The three pilot models are GPT-5.6 Sol, Kimi K3 and DeepSeek-V4.1 Flash. Their configuration matches the main evaluation, except for a 12-hour time budget (Appendix A.2).

The choice of construction model does affect what the pipeline produces. However, rebuilding the benchmark with other models is extremely expensive. We believe the key to keeping the benchmark independent of this choice is strict acceptance criteria for its outputs. We verify programmatically that submitting the source repository as it stands passes T⁺ in full and fails every case of T⁻ (Table 4). Pilot runs and human sampling also ensure that nearly all assertions can be inferred from the instruction I (Appendix A.2–A.3). These checks do not depend on which model built the benchmark.

Since the pilot models are also evaluated, we tested whether the correction process favours them. The table below compares each model's score on all cases with its score when only cases never modified during correction are kept (each model at its highest reasoning effort; † marks a pilot model).

ModelS, all cases (rank)S, unmodified cases only (rank)Change
GPT-5.6 Sol †58.93 (1)60.35 (1)+1.42
Kimi K3 †56.67 (2)57.98 (2)+1.31
Grok 4.655.68 (3)54.37 (6)−1.31
GLM-5.3 Flash53.41 (4)55.14 (3)+1.73
Qwen-3.8 Flash52.83 (5)54.50 (5)+1.67
Muse Spark 1.352.66 (6)54.66 (4)+2.00
DeepSeek-V4.1 Flash †52.47 (7)54.18 (7)+1.71

The changes for the three pilot models (+1.31 to +1.71) fall within the range of the other four models (−1.31 to +2.00), and none of the pilot models changes rank. We therefore find no evidence that the correction process favours the pilot models.

Q2: What is the rationale for the thresholds of 4,000 lines of code and 100 stars?

The 4,000-line threshold ensures that each task is repository-scale and spans multiple modules. The 100-star threshold excludes repositories with little maintenance history, because the evidence for T⁻ comes mainly from the authors' documentation and issues. Both are loose lower bounds: across the 138 selected repositories, the median size is about 39k lines of code and the median star count is about 1,900.

We hope these responses address your concerns.

Thank you for providing additional analyses.

The breakdown of the T categories helps clarify the diversity of the unmet-specification tests and alleviates my concern about their concentration on source-specific failures. The clarification of the construction and pilot models, along with the additional analysis of the correction process, is helpful in addressing my concern about the potential bias toward pilot models. While the broader sensitivity of benchmark construction to different LLMs remains an open question, I understand the computational cost of such experiments and do not consider this a major limitation for the current work.

Overall, the rebuttal has addressed most of my concerns and I will maintain my original recommendation.

Comments on the paper

1 comment

We thank all reviewers for their comments. We have uploaded a revised version of the paper. All changes are marked in blue, the main text remains within the page limit, and the rest of the paper is unchanged.

  • Composition of T⁻ (Reviewer qrTv). A new Appendix D (Table 8) breaks down the 5,899 cases of T⁻ by the evidence behind each requirement. Stage 4 in Section 2 now points to it.
  • Models used in construction (Reviewer qrTv). The AI use statement now states that Stages 1–8 use Claude Opus 5, which is not among the evaluated models. Appendix A.2 names the three pilot models and reports the rescoring on cases never modified during correction.
  • Obfuscation mapping (Reviewer k2JJ). Stage 6 in Section 2 now states that M is generated programmatically and can be regenerated.
  • Versioned release (Reviewers 7UuH and k2JJ). Appendix C now describes what version 1.0 fixes and how later changes are released.
  • Minor correction. Stage 4 now reads "at least 20" unmet-specification cases per task, consistent with Figure 2 and Table 7.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.