- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DivAlgo, a large-scale benchmark for systematically evaluating the algorithmic diversity of code-generation models. Constructed from CodeContests+, DivAlgo contains 7,623 programming problems, each associated with two representative human solutions and a larger pool of correct C++ submissions. Algorithmic differences are characterized at three levels: programming paradigm, problem-solving technique, and data structure, with solution pairs grouped into strong, medium, weak, and identical diversity categories. To support scalable evaluation, the authors distill CodeX-SFT, a lightweight embedding-based similarity metric trained using annotations from an ensemble of proprietary LLM judges, and combine it with the Vendi score to measure population-level diversity. They also introduce AlgoCov to assess coverage of representative human strategies. The benchmark construction and metric are validated through agreement analyses among multiple judges, comparisons with lightweight similarity measures, and evaluation on a small external human-curated dataset. Experiments covering 24 code-generation models, 1,000 test problems, and multiple diversity-oriented generation strategies show that uncontrolled generations recover only about 31% of human algorithmic diversity, while the strongest iterative strategy reaches approximately 70.5% on a matched subset. The results further suggest that models favor more popular human approaches and remain substantially below human coverage even when explicitly steered toward alternative strategies.
Critical strengths
1、The paper substantially expands the scale and granularity of algorithmic-diversity evaluation. DivAlgo contains 7,623 programming problems with human solution pools and representative solution pairs characterized at the levels of programming paradigm, problem-solving technique, and data structure. This is consequential because it enables a broader and more interpretable analysis than prior evaluations based on smaller task sets or coarser notions of similarity. The benchmark could therefore serve as a useful resource for studying dimensions of code-generation capability that are not captured by correctness-based evaluation alone. 2、The empirical evaluation is broad and systematically examines complementary notions of diversity. The paper evaluates 24 code-generation models across uncontrolled sampling, temperature variation, verbalized sampling, iterative constrained generation, heterogeneous model pooling, and explicit strategy steering, while distinguishing population-level diversity from coverage of representative human strategies. This breadth is consequential because the conclusions are not tied to a single model or generation method. The experiments consistently show that current models favor more common solution approaches, that diversity-oriented prompting is more effective than simply pooling heterogeneous models, and that a substantial gap from human algorithmic diversity remains.
Critical weaknesses
1、The paper’s central empirical claims rely on a diversity metric whose validation is not sufficiently independent of the supervision used to construct it. CodeX-SFT is trained on approximately 14,000 solution pairs labeled by proprietary LLM judges and is evaluated primarily against another LLM-ensemble majority on 700 pairs, with overlap in the judge families used for training and evaluation. Related automatic signals and LLM judgments are also used in constructing DivAlgo. The only external human-curated evaluation contains 30 pairs, with the analysis focusing mainly on 23 Type-3 and Type-4 pairs, which the authors acknowledge is too small for meaningful statistical analysis. This is consequential because the headline findings, including the reported 31% and 70.5% human-diversity results, depend directly on CodeX-SFT. Stronger independent human validation or cross-metric robustness evidence is therefore needed to establish that the conclusions reflect genuine algorithmic diversity rather than biases inherited from the automated judging pipeline. 2、The paper’s novelty beyond the closest prior work is not yet sufficiently established. Prior studies have already examined algorithmic diversity in model-generated code, the effects of temperature and heterogeneous-model sampling, diversity-oriented prompting, algorithm steering, and behavior-based measures of algorithmic similarity. DivAlgo clearly extends this line of work in scale, ontology granularity, and evaluation breadth, but the paper does not demonstrate that these additions lead to materially different model rankings, strategy rankings, or scientific conclusions from existing frameworks. This is consequential because the main contribution currently appears closer to a large-scale consolidation and systematization of prior ideas than to a clearly distinct methodological or scientific advance.
Questions for authors
1、Since both the construction of DivAlgo and the training/evaluation of CodeX-SFT rely substantially on judgments generated by proprietary LLMs, was this LLM-based annotation pipeline independently validated by human experts? More specifically, how do the authors verify that the learned similarity signal captures human expert judgments about the underlying algorithmic reasoning and solution strategy, rather than merely distilling the labeling preferences of the LLM judges? If human validation was conducted, please report the number of reviewed solution pairs, the number and expertise of the annotators, the annotation criteria, and inter-annotator agreement. 2、How exactly is the Vendi score implemented? Are the eigenvalues of the cosine Gram matrix normalized to sum to one before computing Equation (1), and how are small negative eigenvalues caused by numerical error handled? 3、Which human solution pool is used in the human-model diversity comparisons? Are Figures 2-4 and Tables 4-5 based on all accepted human submissions or on solutions retained after diversity-oriented filtering? 4、Are the main conclusions robust to alternative algorithmic-similarity metrics? Do the model rankings, generation-strategy rankings, and the reported 31% and 70.5% human-model diversity results remain qualitatively similar under CodeXEmbed, PDG, the LLM-judge ensemble, or BehaveSim where applicable? 5、Could the limited correctness test suite affect the measured diversity of model-generated solutions? Since model outputs are evaluated using 15 CodeContests+ test cases, whereas the human reference solutions were accepted by the original programming platforms, were the generated solutions re-evaluated using stronger or additional tests? If so, do the main diversity conclusions remain unchanged? 6、Which principal empirical conclusions are unique to DivAlgo relative to the closest prior algorithmic-diversity frameworks? In particular, does DivAlgo yield materially different model rankings, strategy rankings, or scientific conclusions, beyond extending prior work in scale and granularity?
4 comments
Dear Reviewer, we wanted to follow up on our response to your feedback. We would be grateful if you could take a look before the discussion period closes. We are happy to clarify any remaining points.
Thank you for your time and consideration
Dear Reviewer,
A friendly reminder to check the authors’ responses and the other reviewers’ comments. It would be helpful to hear your thoughts on the differing assessments and whether they affect your evaluation.
Thank you for your time and effort!
AC
I sincerely thank the authors for their detailed and thoughtful responses, as well as the additional experiments and clarifications. I appreciate the effort invested in addressing my comments, and several of my questions have been satisfactorily answered.
However, I remain particularly concerned about two central issues. First, the evaluation metric still lacks independent human expert validation, and the additional evidence does not fully establish the robustness of the main conclusions. Second, the novelty of DivAlgo over existing approaches remains insufficiently demonstrated, particularly regarding whether it enables substantially new scientific findings.
While the rebuttal has helped clarify several aspects of the work, these two concerns remain important to my overall assessment. Therefore, I maintain my original rating.
Dear reviewer, we have posted a response to your feedback and would be grateful if you could take a look when you have a moment. If our responses resolve your concerns, we would appreciate it if you could consider updating your final assessment. If any points remain open, please do let us know. We would be happy to clarify them further during the discussion period.
Thank you for your time and consideration.