acceptodds
Under review as a conference paper at ICLR 2027

: A systematic evaluation of algorithmic diversity for code generation models

Abstract

Code generation models are actively being used in professional and educational settings. However, the competence of large language models (LLMs) to generate algorithmically diverse code remains largely unknown. We propose the DivAlgo benchmark, a large-scale dataset constituting algorithmically diverse solutions to programming problems. We also introduce a lightweight similarity metric to conduct an evaluation at scale. Our experiments spanning 24 modern code generation models found that uncontrolled model-generated code is significantly less diverse than the human solutions (captures 31% of human diversity), and even the best diverse code generation strategy could only achieve 70% of the human diversity. Moreover, while it was possible to steer the models to generate solutions based on human strategies, they were unable to reproduce the diversity of the human solutions. These findings indicate the limitations of code generation models to capture the algorithmic diversity covered by humans, and overexposure to model-generated code can narrow the algorithmic vocabulary of programmers.

Primary area:
datasets and benchmarks
Keywords:
code generation, algorithmic diversity, evaluation benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RKUsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DivAlgo, a large-scale benchmark for systematically evaluating the algorithmic diversity of code-generation models. Constructed from CodeContests+, DivAlgo contains 7,623 programming problems, each associated with two representative human solutions and a larger pool of correct C++ submissions. Algorithmic differences are characterized at three levels: programming paradigm, problem-solving technique, and data structure, with solution pairs grouped into strong, medium, weak, and identical diversity categories. To support scalable evaluation, the authors distill CodeX-SFT, a lightweight embedding-based similarity metric trained using annotations from an ensemble of proprietary LLM judges, and combine it with the Vendi score to measure population-level diversity. They also introduce AlgoCov to assess coverage of representative human strategies. The benchmark construction and metric are validated through agreement analyses among multiple judges, comparisons with lightweight similarity measures, and evaluation on a small external human-curated dataset. Experiments covering 24 code-generation models, 1,000 test problems, and multiple diversity-oriented generation strategies show that uncontrolled generations recover only about 31% of human algorithmic diversity, while the strongest iterative strategy reaches approximately 70.5% on a matched subset. The results further suggest that models favor more popular human approaches and remain substantially below human coverage even when explicitly steered toward alternative strategies.

Critical strengths

1、The paper substantially expands the scale and granularity of algorithmic-diversity evaluation. DivAlgo contains 7,623 programming problems with human solution pools and representative solution pairs characterized at the levels of programming paradigm, problem-solving technique, and data structure. This is consequential because it enables a broader and more interpretable analysis than prior evaluations based on smaller task sets or coarser notions of similarity. The benchmark could therefore serve as a useful resource for studying dimensions of code-generation capability that are not captured by correctness-based evaluation alone. 2、The empirical evaluation is broad and systematically examines complementary notions of diversity. The paper evaluates 24 code-generation models across uncontrolled sampling, temperature variation, verbalized sampling, iterative constrained generation, heterogeneous model pooling, and explicit strategy steering, while distinguishing population-level diversity from coverage of representative human strategies. This breadth is consequential because the conclusions are not tied to a single model or generation method. The experiments consistently show that current models favor more common solution approaches, that diversity-oriented prompting is more effective than simply pooling heterogeneous models, and that a substantial gap from human algorithmic diversity remains.

Critical weaknesses

1、The paper’s central empirical claims rely on a diversity metric whose validation is not sufficiently independent of the supervision used to construct it. CodeX-SFT is trained on approximately 14,000 solution pairs labeled by proprietary LLM judges and is evaluated primarily against another LLM-ensemble majority on 700 pairs, with overlap in the judge families used for training and evaluation. Related automatic signals and LLM judgments are also used in constructing DivAlgo. The only external human-curated evaluation contains 30 pairs, with the analysis focusing mainly on 23 Type-3 and Type-4 pairs, which the authors acknowledge is too small for meaningful statistical analysis. This is consequential because the headline findings, including the reported 31% and 70.5% human-diversity results, depend directly on CodeX-SFT. Stronger independent human validation or cross-metric robustness evidence is therefore needed to establish that the conclusions reflect genuine algorithmic diversity rather than biases inherited from the automated judging pipeline. 2、The paper’s novelty beyond the closest prior work is not yet sufficiently established. Prior studies have already examined algorithmic diversity in model-generated code, the effects of temperature and heterogeneous-model sampling, diversity-oriented prompting, algorithm steering, and behavior-based measures of algorithmic similarity. DivAlgo clearly extends this line of work in scale, ontology granularity, and evaluation breadth, but the paper does not demonstrate that these additions lead to materially different model rankings, strategy rankings, or scientific conclusions from existing frameworks. This is consequential because the main contribution currently appears closer to a large-scale consolidation and systematization of prior ideas than to a clearly distinct methodological or scientific advance.

Questions for authors

1、Since both the construction of DivAlgo and the training/evaluation of CodeX-SFT rely substantially on judgments generated by proprietary LLMs, was this LLM-based annotation pipeline independently validated by human experts? More specifically, how do the authors verify that the learned similarity signal captures human expert judgments about the underlying algorithmic reasoning and solution strategy, rather than merely distilling the labeling preferences of the LLM judges? If human validation was conducted, please report the number of reviewed solution pairs, the number and expertise of the annotators, the annotation criteria, and inter-annotator agreement. 2、How exactly is the Vendi score implemented? Are the eigenvalues of the cosine Gram matrix normalized to sum to one before computing Equation (1), and how are small negative eigenvalues caused by numerical error handled? 3、Which human solution pool is used in the human-model diversity comparisons? Are Figures 2-4 and Tables 4-5 based on all accepted human submissions or on solutions retained after diversity-oriented filtering? 4、Are the main conclusions robust to alternative algorithmic-similarity metrics? Do the model rankings, generation-strategy rankings, and the reported 31% and 70.5% human-model diversity results remain qualitatively similar under CodeXEmbed, PDG, the LLM-judge ensemble, or BehaveSim where applicable? 5、Could the limited correctness test suite affect the measured diversity of model-generated solutions? Since model outputs are evaluated using 15 CodeContests+ test cases, whereas the human reference solutions were accepted by the original programming platforms, were the generated solutions re-evaluated using stronger or additional tests? If so, do the main diversity conclusions remain unchanged? 6、Which principal empirical conclusions are unique to DivAlgo relative to the closest prior algorithmic-diversity frameworks? In particular, does DivAlgo yield materially different model rankings, strategy rankings, or scientific conclusions, beyond extending prior work in scale and granularity?

4 comments

Dear reviewer, we have posted a response to your feedback and would be grateful if you could take a look when you have a moment. If our responses resolve your concerns, we would appreciate it if you could consider updating your final assessment. If any points remain open, please do let us know. We would be happy to clarify them further during the discussion period.

Thank you for your time and consideration.

Dear Reviewer, we wanted to follow up on our response to your feedback. We would be grateful if you could take a look before the discussion period closes. We are happy to clarify any remaining points.

Thank you for your time and consideration

Dear Reviewer,

A friendly reminder to check the authors’ responses and the other reviewers’ comments. It would be helpful to hear your thoughts on the differing assessments and whether they affect your evaluation.

Thank you for your time and effort!

AC

I sincerely thank the authors for their detailed and thoughtful responses, as well as the additional experiments and clarifications. I appreciate the effort invested in addressing my comments, and several of my questions have been satisfactorily answered.

However, I remain particularly concerned about two central issues. First, the evaluation metric still lacks independent human expert validation, and the additional evidence does not fully establish the robustness of the main conclusions. Second, the novelty of DivAlgo over existing approaches remains insufficiently demonstrated, particularly regarding whether it enables substantially new scientific findings.

While the rebuttal has helped clarify several aspects of the work, these two concerns remain important to my overall assessment. Therefore, I maintain my original rating.

Reviewer_4hQMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose DivAlgo, a benchmark for comparing the algorithmic diversity of human and model generated code. The paper builds a benchmark of 7623 problems, trains a similarity model, and tests 24 code models. Several experiments designed by the authors show that diverse prompting helps, but the scores remain below the human reference scores.

Critical strengths
  • The benchmark studies different algorithms for the same task, going beyond correctness and performance.
  • Comparing sampling, iterative prompting, explicit strategy hints, and some other methods help show whether models lack knowledge of certain algorithms or need better prompts to use them.
  • Sample-count matching and validation on human-curated program pairs provide useful checks on the diversity comparisons.
  • The authors provide detailed analyses of several models, supporting the transferability of the findings across models.
Critical weaknesses
  • The human pool is filtered to keep diverse solutions before comparison. Matching sample counts does not remove this selection advantage, so the gap measures performance against a curated pool rather than typical human output.
  • CodeX-SFT learns LLM judgments, with only a small human-curated check of actual algorithm differences. It remains unclear whether its Vendi score separates new algorithms from different implementations of the same algorithm.
  • AlgoCov rewards matches to two selected human strategies and ignores other valid approaches, as Section 5.3 acknowledges. A model can produce a new algorithm without receiving coverage credit, limiting the measure of creative breadth.
Questions for authors

Please respond to the concerns raised in Critical Weaknesses.

4 comments

Dear reviewer, we have posted a response to your feedback and would be grateful if you could take a look when you have a moment. If our responses resolve your concerns, we would appreciate it if you could consider updating your final assessment. If any points remain open, please do let us know. We would be happy to clarify them further during the discussion period.

Thank you for your time and consideration.

Dear Reviewer, we wanted to follow up on our response to your feedback. We would be grateful if you could take a look before the discussion period closes. We are happy to clarify any remaining points.

Thank you for your time and consideration

Thank you for the honest and thoughtful response, and for taking these concerns so seriously. I really appreciate the care you put into this work. So I will raise my score.

Thank you for considering our response and updating your score. We really appreciate it!

Reviewer_q2wWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents DivAlgo, a benchmark for systematically evaluating the algorithmic diversity of LLM-generated code. The authors construct a dataset of algorithmically distinct human solutions using code similarity metrics and LLM-based judgments, and introduce a lightweight embedding-based metric to quantify algorithmic diversity. Through experiments involving 24 code generation models and multiple generation strategies, the study finds that LLM-generated solutions exhibit substantially lower algorithmic diversity than human-written solutions, although diversity-oriented prompting can partially reduce this gap.

Critical strengths

The paper tackles an interesting and important research problem. Evaluating algorithmic diversity beyond functional correctness is a meaningful direction for understanding the capabilities and limitations of code generation models.

The study covers a large number of programming problems, 24 code generation models, and multiple generation strategies, providing a relatively comprehensive empirical investigation.

Critical weaknesses

The most severe concerns for the work lie in the methodology soundness:

  • Questionable validity of the primary diversity metric. The proposed VendiCodeX-SFT measures diversity in the learned embedding space, which does not necessarily correspond to actual algorithmic diversity. The reported AUC of 0.727 against LLM-judge consensus also raises concerns about measurement reliability. Consequently, the central quantitative findings may be sensitive to metric inaccuracies.

  • Potentially inadequate definition of algorithmic coverage. AlgoCov considers a generated solution to match a reference solution if their ontology labels share at least one element. However, sharing a programming paradigm, technique, or data structure does not necessarily imply using the same algorithm. This may lead to misleading coverage estimates in RQ3.

  • Although the authors evaluate CodeX-SFT on a small existing human-curated dataset, they do not directly validate the LLM-generated algorithmic diversity labels in DivAlgo against independent human annotations. Agreement among LLM judges does not necessarily establish the correctness of these labels.

  • The benchmark focuses exclusively on C++ competitive programming problems and selects only two representative human solutions per problem. It remains unclear whether the findings generalize to other programming languages, real-world software development, or a broader range of algorithmic strategies.

Questions for authors

Why is sharing a single ontology label sufficient to establish an algorithmic match in AlgoCov? How does the metric distinguish genuinely equivalent algorithms from solutions that merely share common techniques or data structures?

Can authors further justify the rationale and soundness of the algorithmic similarity metrics?

Could the authors conduct an independent human evaluation of a representative sample of DivAlgo solution pairs and report the agreement between human annotations and LLM judgments?

2 comments

Dear reviewer, we have posted a response to your feedback and would be grateful if you could take a look when you have a moment. If our responses resolve your concerns, we would appreciate it if you could consider updating your final assessment. If any points remain open, please do let us know. We would be happy to clarify them further during the discussion period.

Thank you for your time and consideration.

Dear Reviewer, we wanted to follow up on our response to your feedback. We would be grateful if you could take a look before the discussion period closes. We are happy to clarify any remaining points.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.