acceptodds
Withdrawn from ICLR 2027

Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Benchmarks drive AI progress, but are hampered by task under-specification and over-specification. Task under-specification causes guesswork, where solutions have to make things up, or worse, resort to contamination or reward hacking. Task over-specification causes spoilers, where solutions can bypass some of the harder steps the benchmark was supposed to measure.Since future-looking benchmarks should be difficult, over-specification reduces a benchmark's shelflife of community adoption. Repository-level coding benchmarks have helped the field progress but experience unnecessary churn due to both over- and under-specification. For instance, SWE-bench Pro, FEA-bench, Doc2Feat-bench, and Multi-SWE-bench all tackle under-specification by augmenting task statements with hints derived from ground truth. We show that leaking ground-truth information into the task statement causes over-specification in some of these benchmarks. We experiment with various specification levels, trying to thread the needle between spoilers and guesswork. Based on these results, we recommend techniques for deriving parsimonious hints. We release our hints so the community can obtain more reliable signals from popular benchmarks.

Primary area:
datasets and benchmarks
Keywords:
Coding Agent, Repo Level Coding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jzPfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the paper points out the problem that benchmarks can provide either too little (guesswork) or too much information (spoiler). It asks how much information should a coding benchmark give a model so that the test is fair without giving away the solution. The authors' proposal to address this issue has two parts; "DefUse" selects the names that connect the implementation to the hidden tests; "Declaration" reveals that function's name, arguments, and file location, but omits descriptions derived from its implementation. Their experiments show that these limited hints can substantially improve success rates.

Critical strengths
  1. this paper addresses an interesting evaluation problem, that hidden interface requirements can lead to model failures but is not really attributable to models' lack of capability.
  2. the proposed method is automated and simple and addresses the proposed problem.
Critical weaknesses
  1. the issues addressed in the paper is more like suboptimal constructions of the benchmarks themselves, instead of a new academic contribution.
  2. the evaluation subsets in the experiments are small and the authors didn't evaluate the most frontier models.
  3. the insight offered by the authors' proposed method is limited.
Questions for authors
  1. how does the performance of more frontier models, like gpt-5.6-sol, look like in your experiments?
  2. could you go deeper into what's the contribution of the work beyond fixing task instructions in some suboptimal benchmarks?
Reviewer_b3KwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a specification problem in repository-level coding benchmarks: hidden tests may require identifiers absent from the task description, while hints derived from a reference solution may reveal how to solve the task. DefUse selects identifiers defined in one of the reference-code and test patches and used in the other. Declaration then reveals their names, kinds, paths, and signatures without behavioral descriptions. The paper compares hint policies across five benchmarks. On a 68-task SWE-bench Pro subset, Declaration raises DeepSeek resolutions from 7 to 20 and MiniMax resolutions from 2 to 10 relative to no hint.

Critical strengths
  1. The paper identifies a consequential benchmark-design problem: an agent can fail a hidden test because an arbitrary required interface was not specified, while a reference-derived hint can reveal the solution. Separating identifier selection from the information disclosed about each identifier makes this trade-off concrete.
  2. The experiments show that structural interface information can materially affect measured resolution. In Table 6, Declaration raises DeepSeek's resolved tasks from 7 to 20 and MiniMax's from 2 to 10 on the 68-task SWE-bench Pro subset, without adding semantic descriptions.
Critical weaknesses

The main concerns are that the claimed fair disclosure boundary remains unvalidated and the methodological advance over existing identifier-hint pipelines is limited.

  1. The paper does not validate that DefUse and Declaration define a minimal, nonspoiling interface hint. Section 3.1 calls DefUse the minimal selection that removes name-mismatch failures, but a name crossing one reference-code and test patch is not necessarily absent from the issue or base repository, required by every valid implementation, or a legitimate task requirement. Section 4.6 assesses nonspoiling with a judge rubric that treats identity, location, and invocation as non-disclosure, although these are precisely Declaration's attributes. Table 8 therefore cannot independently establish the claimed fair boundary. Case-level evidence is needed to distinguish hints that repair unfair hidden-test failures from hints that reveal reference-specific choices.
  2. The methodological novelty is incremental relative to Doc2Feat and related identifier-hint pipelines. As Section 3.1 explains, Doc2Feat already selects names shared by code and test patches. DefUse adds definition/use direction to that selection. Section 3.2 then limits each hint to structural attributes such as name, path, and signature. These refinements make an existing hint-generation workflow more selective, but do not establish a substantially new approach to benchmark task specification.
  3. The experiments do not establish an advantage over a simpler, solution-blind interface specification. Sections 4.3-4.4 compare no hint and existing or reference-derived hint policies, but not a minimal interface note derived from the issue and public repository without inspecting the reference or hidden-test patches. The results therefore show that hints affect resolution, not that DefUse and Declaration are necessary or better than this simpler alternative. Section 5 also reports only one trajectory per instance and configuration, so small solve-count differences cannot establish a stable advantage.
Questions for authors
  1. Which evaluated DefUse names are genuinely unrecoverable from the issue and base repository, and which resolved tasks demonstrably failed earlier because of those names?
  2. What evidence shows that Declaration outperforms a minimal interface description derived without the reference or hidden-test patches, and that its structural fields do not reveal reference-specific choices?
Reviewer_hehjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the trade-off between under-specification and over-specification in repository-level coding benchmarks. It proposes a framework separating identifier selection from information projection, introducing DefUse to select test-facing identifiers and Declaration to expose structural interface information without implementation descriptions.

Experiments across five coding benchmarks and three language models compare different hint policies in terms of task resolution. An additional LLM-based assessment evaluates the necessity, solution disclosure, and guidance associated with hint attributes.

Critical strengths
  1. Relevant benchmark validity problem. The paper addresses an important trade-off between resolving hidden interface requirements and inadvertently revealing implementation choices. Separating identifier selection from projection provides a useful analytical framework.

  2. Broad empirical comparisons. Evaluations across multiple benchmarks, models, and two programming languages offer useful evidence that hint content and disclosure policies affect measured task resolution.

Critical weaknesses
  1. Insufficient justification for DefUse's minimality and sufficiency. Section 3.1 claims that DefUse identifies the minimal set of identifiers necessary to eliminate naming-related ambiguity. However, the argument assumes that code–test dependencies can be fully captured by explicit def–use relationships. Dynamic access, indirect dependencies, and existing interfaces may violate this assumption. No systematic validation establishes that the selected identifiers are necessary or that excluded identifiers are irrelevant.

  2. Inadequate validation of information leakage. The central claim that Declaration avoids solution disclosure relies heavily on an LLM judge. However, Appendix C explicitly classifies interface identity and addressing as non-leakage, making the zero disclosure scores for structural attributes in Table 8 partly a consequence of the rubric itself. Independent human validation or behavioral evidence is missing. Furthermore, single-trajectory evaluations and substantial differences between related hint configurations (Tables 5 and 6) limit confidence in the reported trade-offs.

Questions for authors
  1. Can the authors validate DefUse against manually annotated interface requirements and provide evidence that its selected identifiers are both necessary and sufficient, particularly for dynamic or indirect code–test dependencies?

  2. Can the authors provide independent human leakage assessments and repeated agent evaluations? In particular, how do they explain the large performance difference between All Hints in Table 5 (49/68) and Pro in Table 6 (19/68), given their related information content?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.