- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents CoSPlay, a test-time unit test generation framework that aims to improve code generation quality with self-generated tests. The proposed method uses a self-play iterative loop to select higher-quality tests, and use those against the generated code to improve its quality. Evaluations on multiple code generation models across four coding benchmarks show improvements compared to the base models.
Critical strengths
-
The idea of self-generating unit tests is sound, and improving code quality with unit test quality at test time is a well-motivated research question. The abstract and introduction are also well written to motivate it.
-
The proposed method operates at test time, saving the cost and effort of training a better unit test generator.
Critical weaknesses
-
The idea of generating unit tests feels like UTGen [1]. Although it is cited in the paper, there is less discussion of the distinction and comparison with this work. The consensus selection part is especially similar to what UTGen uses to select better unit tests. They have also clearly defined the Attack Rate, which reads similarly to "Failure-oriented UT attack idea exploration" in lines 146-152.
-
The proposed pipeline has many design choices. Although the authors did a good job of justifying them with empirical results, it feels bloated and complicated. It is not clear to me how much it costs, compared to using Claude Opus 5.5 or GPT6 to generate the tests. There are also open-source models like UTGen that can serve this purpose.
-
The main results largely compare different models, but there is less focus on different ways of generating unit tests. This is connected to the point above -- how much better is the proposed method at doing unit test generation compared to other methods?
-
The token count analysis for efficiency is great, but it focuses on test-time scaling and doesn't answer points (2) and (3), which is how much cost it saves when compared to other unit test generation methods.
Questions for authors
-
Not really a question but more of a suggestion: there is a lot of negative space throughout the paper, making the layout crowded and hard to read. Figure 2 also has a lot of text. Overall, making more space will improve readability.
-
Figure 3 is not informative and takes up some space within the middle of the method, making it harder to follow. It is not clear to me what Figure 3 wants to convey. The text itself is already self-contained.
-
The method section is overwhelming with a lot of notations, in-line math symbols, and reduced vspace. The connections to figures are also interleaved, so readers have to switch between figure 2,3,4 and equations back and forth.
-
I can't really read anything from Figure 9 (b). It is way too crowded.