- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
this paper extends PBEBench, and proposes a method for certifying what a task in the benchmark actually requires. In particular, each instance has 6 input-out pairs, and the model must produce one common program that correctly transforms every input to its corresponding output. The authors points out that sometimes a two-step program may be solvable with a one-step shortcut. To tackle this issue, they showed that just changing the sixth example can eliminate the shortcut and reducing the success rate across models.
Critical strengths
- the idea of certifying difficulty of tasks in a benchmark is in itself interesting and potentially impactful.
- the experiments are well-conducted, across several settings and models.
Critical weaknesses
- the paper is heavily ai-slopped, and is hard to follow.
- this paper is over-indexed on PBEBench, which is interesting but lacks implication for real programming benchmarks and daily tasks.
- even though certifying difficulty of tasks in benchmarks in itself could be potentially impactful, this paper didn't discuss in depth how to extend their method to general benchmarks. PBEBench in itself is a narrow and manually constructed tasks, and building on top of that offers limited generalizable implications.
Questions for authors
can the authors provide more discussion on how they could generalize their method to certify difficulty of tasks in general benchmarks?