- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how competition on LLM leaderboards affects model capabilities and specialization. The authors develop a game-theoretic model to characterize when developers converge toward similar capability profiles or retain different strengths. They also analyze longitudinal data from LM Arena and OpenRouter. The results show that developers tend to improve more in areas where they lag behind competitors, while some differences across capability areas persist and performance gaps near the leaderboard frontier have narrowed over time.
Critical strengths
-
The paper explores how shared evaluations may influence LLM development, which I believe few papers examine this directly so far. The distinction between convergence in overall performance and similarity in capability profiles is useful. The two are easy to conflate, and the distinction is helpful.
-
The empirical analysis uses longitudinal data, and the catch-up result is checked in several ways, including alternative competitor definitions, release-event fixed effects, and an analysis of regression to the mean.
Critical weaknesses
-
The definitions of monoculture and specialization are useful, but most of the analysis relies on standard game-theoretic tools and equilibrium conditions. The paper can state more clearly what it adds to existing work on multidimensional contests.
-
The results are consistent with the proposed mechanisms, but they do not show that leaderboard competition causes the observed patterns. The catch-up analysis is careful. Even so, the relationship between the model's assumptions and how models are actually developed is still indirect.
Questions for authors
-
Can you share directly what is the main theoretical insight beyond existing multidimensional contest models? Specially, what does the directional characterization in Theorem 2 allow us to understand that was not clear from previous work?
-
Can the authors provide further evidence that the observed catch-up patterns reflect responses to leaderboard competition, rather than developers independently prioritizing weaker capabilities?
-
How sensitive is the narrowing of leaderboard performance gaps to changes over time in which models and categories are included?
2 comments
Dear authors,
Thank you for the detailed response and for the new results in Appendix G.
My first concern is only partly addressed. The point that monoculture can arise even when production functions differ was already in the paper, and I still don't see how the results go beyond existing models such as Colonel Blotto or multi-battle contests. Proposition 4 is a nice addition, since catch-up was previously assumed. Still, both new results feel close to assuming their conclusions without rigid causual relation.
I appreciate that the catch-up claim is now framed as predictive, but they seem not to rule out a simpler explanation. That is, developers may just improve their weaker capabilities regardless of competitors. Separating a developer's own weakness from the leader's score might help. The argument for not fixing the model set makes sense to me, and the fixed-category check is helpful.
A note on number. The response says the gaps shrink by "at least 60%", while the paper reports 53% for the 10th-ranked model (Figure 5), and I couldn't find this analysis in the revision.
I'm sorry, but I've decided to keep my score.
Rebuttal
Dear Reviewer md2v,
Thank you for your review, and for finding that our paper addresses an important and useful question. We answer some of your questions below.
Motivation and Model
The purpose of our work is to understand when monoculture arises in leaderboard competitions. Our model intentionally draws from prior work in competition. What distinguishes our work is a focus on multidimensional competition in leaderboards and how this impacts diversity, and in particular, whether this form of competition leads to monoculture or specialization. That is, our model is intentionally stylized, and the purpose is to better understand how evaluation works and how models compete. By having a stylized model, we make less assumptions in the hope that our results may apply more broadly. We relate these results to what occurs in practice to understand how observed behavior might correspond to our predicted theory.
Theoretical Results
Theorem 2 provides a necessary and sufficient characterization of which equilibrium capability directions can arise, rather than simply characterizing equilibrium investment allocations. In particular, it shows that monoculture can result even when developers have different production functions, as long as the resulting incentives lead to outputs in the same direction. This allows us to distinguish between similarity in output profiles from similarity in the underlying production functions, and allows us to characterize monoculture beyond the sufficient conditions in Lemmas 1 and 2.
At a high level, our key contributions are new insights into when monoculture and specialization arise in multidimensional competition, and how this relates to convergence in overall performance. Lemmas 1,2,3 and Theorem 2 each give different ways to characterize this; Lemma 1 and 2 characterize what happens under similar production functions, while Lemma 3 characterizes what happens under different ones, and Theorem 2 puts it together.
We additionally include new results in the paper. These results strengthen or complement the results from Lemma 3 and Section 5. We detail these new results below, and include the formal statements and proofs as a new Appendix in the paper.
Our first result extends Lemma 3 to consider how conditions on the marginal cost of producing capabilities can lead to specialization. That is, we show that sufficiently different marginal capability costs imply differentiated equilibrium outputs. When this separation cannot be explained solely by differences in overall capability levels, it also implies differentiation in capability profiles.
Our second result builds a new sufficient condition for when catch-up occurs, and is complementary to the existing Proposition 1 in Section 5. The key additional requirement is that a developer that trails in a capability retains marginal capacity to improve that capability at a rate related to the size of its gap. Intuitively, lagging capabilities retain greater marginal capacity for improvement, while competition creates incentives for developers to perform well in lagging capabilities. The new Proposition illustrates how diminishing returns and competitive incentives can jointly generate catch-up. The catch-up guarantee is stronger when lagging capabilities retain greater marginal production capacity, competitive evaluation places greater marginal value on improvement, and marginal investment costs grow more slowly.
Catch-Up Patterns
We thank the reviewer for their comments, and agree that distinguishing competitive responses from independent improvement is important. Our empirical claim is that leaderboard-relative gaps predict subsequent category-level improvement, consistent with the catch-up condition in our model, but we don't claim causal identification. Rather, we find that the relationship remains positive across alternative rival definitions, after controlling for overall model strength, and after release-event fixed effects. We will make the discussion clearer and distinguish what we claim about our catch-up procedure from causal evidence that leaderboards drive these choices.
Sensitivity to Changing Models
We agree that understanding the sensitivity to changing models is an important robustness check. As a first pass at this, we reproduce the numbers from Figure 6, but hold the set of categories fixed so it only contains categories present in all timesteps, and find that gaps are still narrowing. In particular, the gap to the 10th/20th/50th ranked model shrinks by at least 60% regardless of whether you hold the set of categories fixed or not.
Holding the set of models fixed addresses a different question: whether performance gaps narrow within an unchanged cohort, rather than whether competition and model entry lead to convergence. Since model entry and replacement are central to the dynamics we study, fixing the model set removes part of the phenomenon of interest.