- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates whether adding more LLMs increases behavioral diversity in strategic reasoning. The authors evaluate 50 LLMs (26 model families) across 20 strategic games (signaling, coordination, Bayesian reasoning, and deception). They use text embeddings, Principal Component Analysis (PCA), and bottleneck autoencoders to measure behavioral similarity and dimensionality. The results show diminishing returns as more models are added, with 16 principal components explaining 90% of between-model variance and autoencoder reconstruction plateauing at 18-24 dimensions. The authors also evaluate three aggregation methods (majority voting, multi-agent discussion, and cross-provider critique), finding that majority voting improves average strategic scores (0.492 to 0.600), while the other two do not consistently improve performance.
Critical strengths
-
The paper provides a large-scale empirical analysis across 50 LLMs (26 model families) and 20 strategic-game conditions. The distinction between pooled response variation and between-model variation is useful, with PCA requiring 137 dimensions to explain 90% of pooled response variance but only 16 dimensions for between-model variance. The diminishing growth in dimensionality (76.2 at 5 models to 136.2 at 50 models) provides evidence that adding models contributes progressively less to the measured behavioral space. I find this useful for understanding diversity in multi-model systems.
-
Majority voting increases the average strategic score from 0.492 to 0.600, while cross-provider critique changes decisions in 40.2% of cases but improves scores in only 4.6%. I find this distinction between changing decisions and improving decision quality particularly interesting. The authors also conduct robustness checks using alternative embeddings, action-only representations, autoencoder architectures, and open-ended generation, which provide additional support for the observed geometric patterns.
Critical weaknesses
-
I find the motivation and significance somewhat limited given existing work on LLM behavioral homogeneity, correlated errors, and strategic reasoning [1, 2, 3]. The paper introduces a distinct geometric analysis using PCA and autoencoders to study how behavioral dimensionality changes as more models are added. However, it remains unclear what additional insight these findings provide beyond the already established observation that LLMs exhibit similar behaviors and correlated errors. I think the authors should demonstrate whether selecting geometrically diverse models improves collective strategic performance compared with selecting similar or randomly chosen models, which would establish the practical value of the proposed analysis.
-
The authors show that 16 principal components explain 90% of between-model variance, but this does not necessarily imply that models make similar strategic decisions or share the same errors. For example, two models may produce similar reasoning but choose different actions, or provide different explanations while making the same incorrect decision. While the action-only analysis supports the geometric findings, I think a direct analysis of the relationship between geometric distance, action disagreement, and error correlation is needed to establish whether the observed similarity reflects actual strategic behavior.
-
The experiments use a fixed generation temperature (0.7) and a standardized zero-shot prompt, without examining how different temperatures or prompting methods affect the measured behavioral diversity.
-
Prior work such as Social Agents [4] shows that demographic and psychographic persona conditioning can produce diverse behavioral predictions and outperform repeated calls to the same model without personas. This paper only examines diversity across different LLMs and does not investigate whether assigning distinct personas to the same model can produce additional strategic diversity. I think comparing different persona-conditioned agents, repeated calls without personas, and combinations of models from different providers under the same games and computational budget would be useful. If persona conditioning produces greater diversity or better strategic decisions, it would suggest that strategic diversity depends on how agents are constructed rather than simply how many models are combined, limiting the generality of the paper's conclusion.
[1] Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond). NeurIPS 2025. [2] Correlated Errors in Large Language Models. ICML 2025. [3] Playing Repeated Games with Large Language Models. Nature Human Behaviour, 2025. [4] Social Agents: Collective Intelligence Improves LLM Predictions. ICLR 2026.
Questions for authors
-
Could the authors clarify what additional insights the geometric dimensionality analysis provides beyond existing studies of LLM behavioral homogeneity and correlated errors [1, 2, 3]?
-
Have the authors examined whether geometric distance between models is related to action disagreement or error correlation?
-
Were different generation temperatures or prompt formulations evaluated? If so, could the authors clarify whether the dimensionality and saturation findings remain consistent across these settings?
-
Have the authors explored persona-conditioned agents in their strategic-game experiments?
2 comments
Persona-conditioned agents.
We follow Social Agents and consider 10 demographic/psychographic profiles (age, country, occupation, risk attitude, trust), prepended to system prompt. We evaluate of response embedding and action-only representation (see Appendix H) and strategic score (Section 5) using five models. We show the results (without personas with personas) below:
| Model | Embedding d90 | Action-only d90 | Strategic score |
|---|---|---|---|
| Claude Opus 5 | 18 → 25 | 4 → 5 | 1.00 → 0.77 |
| GPT-5.5 | 11 → 16 | 4 → 6 | 0.67 → 0.60 |
| GPT-5.6 | 13 → 17 | 4 → 5 | 0.53 → 0.60 |
| Qwen 3.8 Max | 15 → 22 | 5 → 5 | 0.53 → 0.57 |
| GLM-5.2 | 18 → 24 | 5 → 5 | 0.80 → 0.73 |
We observe that personas do not remove saturation. With personas balanced grows from 20.8 at one model to 58.0 at five. The marginal gain still falls with each added model. Therefore, our scaling result holds with and without personas. We will add this discussion in revision.
We thank the reviewer for detailed feedback.
Difference with model homogeneity and correlated error work.
We thank the reviewer for suggesting additional related work. The suggested works [1,2,3] evaluated whether models are similar (e.g., when answer open-ended questions, playing 2x2 games, agreeing on the same incorrect answers). This paper aims to address how many independent directions the differences between models span, and whether that count keeps growing as we add models. This is different than evaluating similarity among models. Suppose all models answer Stag Hunt in nearly the same way, so they all look highly similar. However, if each model adds its own consideration (one mentions reputation, another the chance that the partner makes a mistake, another fairness), then every new model adds something new. If instead models keep adding the same few considerations, new models add nothing. A similarity score can look the same in both cases. We count the distinct kinds of difference using the same embeddings, and find the count stops growing. We will clarify this distinction in our revision.
Does adding diverse models help the team.
We thank the reviewer for suggesting more practical tests. In Section 5, we test multi-agent discussion with different choices of models. A team of three GPT 5.2 copies and two teams of models from different providers both end with no clear gain in strategic score.
Does geometry reflect actual strategic behavior.
We perform additional experiments to evaluate (1) do models that are far apart in embedding space act differently, and (2) does the saturation hold for actions alone, with no text?
For question (1), we compare the distance between two models' average embeddings with:
To answer question (2), we show a similar saturation behavior when only actions are used for embedding in Appendix H. Combining these results together, the geometry is grounded in strategic choices. Adding models bring progressively less new behaviors. We will add these results to our revision.
Ablation on temperature and prompts.
We thank the reviewer for suggesting this experiment. For temperature ablation, we evaluate this using six models from six families that accept temperature setting (Qwen 3.8 Max, GLM-5.2, DeepSeek V3, Gemini 2.5 Flash, Llama 4 Maverick, Kimi K2). We vary temperature as 0.4, 0.7, and 1.0. Our results are reported below:
The results show that the marginal slope falls with every added model and the saturation still holds with varying temperature.
We consider four prompt styles: our prompt, no system prompt, "think step by step", and a one-sentence answer requirement. We evaluate on five models: Claude Opus 5, GPT-5.5, GPT-5.6, Qwen 3.8 Max, GLM-5.2.
We observe that saturation still holds under every prompt, but the level depends on the prompt. Combining the results together, we note that saturating shape is robust to temperature and prompt. We will add this ablation into our revision.