acceptodds
Under review as a conference paper at ICLR 2027

Strategic Competition and Convergence in LLM Leaderboards

Abstract

Leaderboards are a popular way to compare large language model (LLM) capabilities and can influence how model developers invest their resources. We ask when competition on head-to-head leaderboards leads developers to produce similar models, which we call monoculture, or models specialized in different capabilities. We study this question by formulating leaderboard competition as a game in which model developers invest resources across capabilities to maximize their win rate. We show that similar production functions can induce monoculture, while comparative advantage can sustain specialization. Turning to how competition evolves over time, we give conditions under which diminishing returns and model catch-up leads to crowding at the frontier. Using data from LM Arena and OpenRouter, we find evidence of both competitive catch-up and persistent differentiation: developers catch up where they lag while retaining category-specific advantages, alongside an increasingly crowded frontier and dispersed model usage. Together, our results characterize conditions under which competitive investment can generate convergence while preserving differences.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
leaderboards, evaluation, game theory, competition, monoculture

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_md2vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how competition on LLM leaderboards affects model capabilities and specialization. The authors develop a game-theoretic model to characterize when developers converge toward similar capability profiles or retain different strengths. They also analyze longitudinal data from LM Arena and OpenRouter. The results show that developers tend to improve more in areas where they lag behind competitors, while some differences across capability areas persist and performance gaps near the leaderboard frontier have narrowed over time.

Critical strengths
  1. The paper explores how shared evaluations may influence LLM development, which I believe few papers examine this directly so far. The distinction between convergence in overall performance and similarity in capability profiles is useful. The two are easy to conflate, and the distinction is helpful.

  2. The empirical analysis uses longitudinal data, and the catch-up result is checked in several ways, including alternative competitor definitions, release-event fixed effects, and an analysis of regression to the mean.

Critical weaknesses
  1. The definitions of monoculture and specialization are useful, but most of the analysis relies on standard game-theoretic tools and equilibrium conditions. The paper can state more clearly what it adds to existing work on multidimensional contests.

  2. The results are consistent with the proposed mechanisms, but they do not show that leaderboard competition causes the observed patterns. The catch-up analysis is careful. Even so, the relationship between the model's assumptions and how models are actually developed is still indirect.

Questions for authors
  1. Can you share directly what is the main theoretical insight beyond existing multidimensional contest models? Specially, what does the directional characterization in Theorem 2 allow us to understand that was not clear from previous work?

  2. Can the authors provide further evidence that the observed catch-up patterns reflect responses to leaderboard competition, rather than developers independently prioritizing weaker capabilities?

  3. How sensitive is the narrowing of leaderboard performance gaps to changes over time in which models and categories are included?

2 comments

Dear Reviewer md2v,

Thank you for your review, and for finding that our paper addresses an important and useful question. We answer some of your questions below.

Motivation and Model

The purpose of our work is to understand when monoculture arises in leaderboard competitions. Our model intentionally draws from prior work in competition. What distinguishes our work is a focus on multidimensional competition in leaderboards and how this impacts diversity, and in particular, whether this form of competition leads to monoculture or specialization. That is, our model is intentionally stylized, and the purpose is to better understand how evaluation works and how models compete. By having a stylized model, we make less assumptions in the hope that our results may apply more broadly. We relate these results to what occurs in practice to understand how observed behavior might correspond to our predicted theory.

Theoretical Results

Theorem 2 provides a necessary and sufficient characterization of which equilibrium capability directions can arise, rather than simply characterizing equilibrium investment allocations. In particular, it shows that monoculture can result even when developers have different production functions, as long as the resulting incentives lead to outputs in the same direction. This allows us to distinguish between similarity in output profiles from similarity in the underlying production functions, and allows us to characterize monoculture beyond the sufficient conditions in Lemmas 1 and 2.

At a high level, our key contributions are new insights into when monoculture and specialization arise in multidimensional competition, and how this relates to convergence in overall performance. Lemmas 1,2,3 and Theorem 2 each give different ways to characterize this; Lemma 1 and 2 characterize what happens under similar production functions, while Lemma 3 characterizes what happens under different ones, and Theorem 2 puts it together.

We additionally include new results in the paper. These results strengthen or complement the results from Lemma 3 and Section 5. We detail these new results below, and include the formal statements and proofs as a new Appendix in the paper.

Our first result extends Lemma 3 to consider how conditions on the marginal cost of producing capabilities can lead to specialization. That is, we show that sufficiently different marginal capability costs imply differentiated equilibrium outputs. When this separation cannot be explained solely by differences in overall capability levels, it also implies differentiation in capability profiles.

Our second result builds a new sufficient condition for when catch-up occurs, and is complementary to the existing Proposition 1 in Section 5. The key additional requirement is that a developer that trails in a capability retains marginal capacity to improve that capability at a rate related to the size of its gap. Intuitively, lagging capabilities retain greater marginal capacity for improvement, while competition creates incentives for developers to perform well in lagging capabilities. The new Proposition illustrates how diminishing returns and competitive incentives can jointly generate catch-up. The catch-up guarantee is stronger when lagging capabilities retain greater marginal production capacity, competitive evaluation places greater marginal value on improvement, and marginal investment costs grow more slowly.

Catch-Up Patterns

We thank the reviewer for their comments, and agree that distinguishing competitive responses from independent improvement is important. Our empirical claim is that leaderboard-relative gaps predict subsequent category-level improvement, consistent with the catch-up condition in our model, but we don't claim causal identification. Rather, we find that the relationship remains positive across alternative rival definitions, after controlling for overall model strength, and after release-event fixed effects. We will make the discussion clearer and distinguish what we claim about our catch-up procedure from causal evidence that leaderboards drive these choices.

Sensitivity to Changing Models

We agree that understanding the sensitivity to changing models is an important robustness check. As a first pass at this, we reproduce the numbers from Figure 6, but hold the set of categories fixed so it only contains categories present in all timesteps, and find that gaps are still narrowing. In particular, the gap to the 10th/20th/50th ranked model shrinks by at least 60% regardless of whether you hold the set of categories fixed or not.

Holding the set of models fixed addresses a different question: whether performance gaps narrow within an unchanged cohort, rather than whether competition and model entry lead to convergence. Since model entry and replacement are central to the dynamics we study, fixing the model set removes part of the phenomenon of interest.

Dear authors,

Thank you for the detailed response and for the new results in Appendix G.

My first concern is only partly addressed. The point that monoculture can arise even when production functions differ was already in the paper, and I still don't see how the results go beyond existing models such as Colonel Blotto or multi-battle contests. Proposition 4 is a nice addition, since catch-up was previously assumed. Still, both new results feel close to assuming their conclusions without rigid causual relation.

I appreciate that the catch-up claim is now framed as predictive, but they seem not to rule out a simpler explanation. That is, developers may just improve their weaker capabilities regardless of competitors. Separating a developer's own weakness from the leader's score might help. The argument for not fixing the model set makes sense to me, and the fixed-category check is helpful.

A note on number. The response says the gaps shrink by "at least 60%", while the paper reports 53% for the 10th-ranked model (Figure 5), and I couldn't find this analysis in the revision.

I'm sorry, but I've decided to keep my score.

Reviewer_W9x5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This study investigates how the popular arena-style benchmarks (i.e., LLMArena) would drive LLM developers in the direction of optimizing new models from the competition perspective, i.e., whether the LLMs have more similar or more specialized capability profiles. Based on the theory, not only the models but also the optimization strategies of developers would be reshaped by the evaluation.

Critical strengths
  • The investigation of the whole paper is really interesting and timely. I personally like this topic as it proposes a new view for more in-depth analysis of LLM evaluation.
  • Overall, the paper is well written.
Critical weaknesses

My main concern is the significant misalignment between the theory and the research tasks. Model training is a complex task, where not only the training investments shown in the paper but also the evaluation seem too trivial compared with the real-world situation. For instance, the training of LLMs cannot be directly and simply divided into several parts of training efforts. Sometimes the proportion of training cannot even be directly quantified. For the coding task, pre-training, mid-training, and post-training (SFT and agentic RL) will influence the final results, where a single dimension cannot reflect these dynamic and complex changes. Moreover, I personally disagree that one benchmark result determines the developers' training. On the one hand, Arena-like benchmarks, despite their effectiveness, are sometimes not the main target score LLM providers focus on. When reading the model card or technical report of new models, you would find that LM Arena or Chat Arena or others are not the determining strength. In contrast, the authors rely more on the results from different static benchmarks to demonstrate the performance of new models. Therefore, I do not believe that the competitive investment has been reshaped by these arena-style evaluations.

Regarding the tasks, Arena-style evaluations and previous static benchmarks are both important, but serve different evaluation dimensions. The tasks like math and code shown in your figure, in my opinion, are more suitable for a benchmark-style evaluation than any Elo-style evaluations.

Questions for authors

Please refer to the weaknesses.

2 comments

Dear Reviewer W9x5,

Thank you for your review, and for viewing our paper as interesting and timely, along with being well written. We provide responses to your questions below.

Paper Motivation and Alignment

Thank you for your inquiry into our model. We agree that model training is complex, with many stages, competing priorities, etc. A comprehensive model of LLM training would not only be very complex, but would also be difficult to instantiate as many factors that impact LLM training are private or hidden (e.g., how companies make decisions, differences in resources between companies, etc.).

Instead, the purpose of our work is to develop an intentionally stylized model. Doing so can capture some elements of model development to better understand specific phenomena. By having a stylized model, we make less assumptions in the hope that our results may apply more broadly. In our paper, the objective is to understand the relationship between strategic competition and diversity, and we both develop a theoretical model of this, and also analyze empirical trends with real data from OpenRouter/LLM Arena. We do not claim that model developers exactly respond to leaderboards or that improving models can be perfectly captured through production functions. Rather, we ask what factors might impact whether models become more or less diverse over time, and what role evaluation plays in this. We then design a stylized model around this idea to better understand the mechanisms at play.

We finally note that prior literature has designed similarly stylized models to understand evaluations; in both our work and theirs, the goal is to understand the forces underlying the phenomenon rather than perfectly capture reality [1]. These stylized models are intentionally simple, as otherwise, extracting the underlying mechanisms becomes more difficult.

[1] Chen, Yatong, Guanhua Zhang, and Moritz Hardt. "Leaderboard Incentives: Model Rankings under Strategic Post-Training." (ICML 2026)

Dear authors,

Thanks for your response. I have raised the score.

Reviewer_bfZZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper describes the LLM competition from the point of the game theory. It introduces a model of the market with developers and discusses the conditions for Nash Equilibria to occur, the scenarios of specialization and monocultures. The theoretical study is accompanied with some experimental study based on econometric model.

Critical strengths

None

Critical weaknesses
  • The paper seems a poor fit for ICLR. While it can be placed within the field of computational economics, this is not the main interest of ICLR (learning of representations). I suppose, a venue like IJCAI, AAAI, EC or related would be a better fit.

  • I have serious doubts about the whole setup of this model. From my perspective, apart from including ELO-style fragment, the rest looks like most of the "typical" studies on market structures. I felt like I could replace "LLMs" with any enterprise and the results would not be affected. And it is a serious problem, as studies on developers and computer-science/software-related environments should not assume production/costs functions as in the "traditional" markets (e.g., problems with increasing or non-diminishing returns to scale, negligible or zero costs, rise of gift economies...). The paper does not discuss any violations from the "mainstream" economics assumptions.

  • The results are expected -- the assumptions and the model are standard, so most of the paper boils down to a numerical task (that is expected to be successful, given the underlying standard way of modeling). Apart from a new "story", I do not see any contribution that could justify consideration of the acceptance at ICLR.

  • Part of the related studies is missing -- no discussion of the problems with network effects, standardization or other non-mainstream economics phenomena that have been described in the literature on software-related markets.

  • Problems with econometrics -- some important details are missing (why POLS? The full log instead of the small plots?)

Questions for authors

Could you elaborate on your econometric model? I do not know why you used Pooled OLS (have you checked if this is the correct model?). I miss the log from the model, the plots are not sufficient.

2 comments

Dear Reviewer bfZZ,

Thank you for your review; we provide answers to your questions below.

Modeling and Contributions

Thank you for your interest in our modeling and contributions. The purpose of our paper is to understand how leaderboard competition impacts competition between different LLM model developers. What distinguishes our work is a focus on multidimensional competition in leaderboards and how this impacts diversity, and in particular, whether this form of competition leads to monoculture or specialization. Our model is intentionally stylized, and the purpose is to better understand how evaluation works and how models compete. By having a stylized model, we make less assumptions in the hope that our results may apply more broadly. We relate these results to what occurs in practice to understand how observed behavior might correspond to our predicted theory.

Our framework does not require any particular production function or cost structure. Some results from our theory impose certain assumptions, such as diminishing returns and increasing cost functions, because without these, an equilibrium could fail to exist. We agree that other extensions, such as non-diminishing returns and low costs, are interesting, and we will clarify the scope of our results in an updated version of the paper.

Econometrics

We thank the reviewer for inquiring about our econometric model. Our regression model includes developer-category and category-time fixed effects, and we will include full regression tables in a final version of the paper. Our intention with this was to illustrate descriptive relationships rather than causal effects, and the plots demonstrate these trends in a visual manner.

New Theory

We additionally include new results in the paper. These results strengthen or complement the results from Lemma 3 and Section 5. We detail these new results below, and include the formal statements and proofs as a new Appendix in the paper.

Our first result extends Lemma 3 to consider how conditions on the marginal cost of producing capabilities can lead to specialization. That is, we show that sufficiently different marginal capability costs imply differentiated equilibrium outputs. When this separation cannot be explained solely by differences in overall capability levels, it also implies differentiation in capability profiles.

Our second result builds a new sufficient condition for when catch-up occurs, and is complementary to the existing Proposition 1 in Section 5. The key additional requirement is that a developer that trails in a capability retains marginal capacity to improve that capability at a rate related to the size of its gap. Intuitively, lagging capabilities retain greater marginal capacity for improvement, while competition creates incentives for developers to perform well in lagging capabilities. The new Proposition illustrates how diminishing returns and competitive incentives can jointly generate catch-up. The catch-up guarantee is stronger when lagging capabilities retain greater marginal production capacity, competitive evaluation places greater marginal value on improvement, and marginal investment costs grow more slowly.

Reviewer_bfZZOpenReview ↗

Response

Thank you for the nicely-written response that contains little information. Most of the weaknesses I mentioned are not addressed here; despite the length of the response. I am looking forward to the revised version of the paper, but based on this rebuttal I cannot increase my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.