- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
the paper points out the problem that benchmarks can provide either too little (guesswork) or too much information (spoiler). It asks how much information should a coding benchmark give a model so that the test is fair without giving away the solution. The authors' proposal to address this issue has two parts; "DefUse" selects the names that connect the implementation to the hidden tests; "Declaration" reveals that function's name, arguments, and file location, but omits descriptions derived from its implementation. Their experiments show that these limited hints can substantially improve success rates.
Critical strengths
- this paper addresses an interesting evaluation problem, that hidden interface requirements can lead to model failures but is not really attributable to models' lack of capability.
- the proposed method is automated and simple and addresses the proposed problem.
Critical weaknesses
- the issues addressed in the paper is more like suboptimal constructions of the benchmarks themselves, instead of a new academic contribution.
- the evaluation subsets in the experiments are small and the authors didn't evaluate the most frontier models.
- the insight offered by the authors' proposed method is limited.
Questions for authors
- how does the performance of more frontier models, like gpt-5.6-sol, look like in your experiments?
- could you go deeper into what's the contribution of the work beyond fixing task instructions in some suboptimal benchmarks?