acceptodds
Under review as a conference paper at ICLR 2027

Does Reasoning Make Language Models Agree?

Abstract

Language models face two opposing concerns: monoculture, in which models make the same mistakes, and multiplicity, in which equally accurate models make conflicting decisions. The rise of reasoning models has complicated this debate: On the one hand, reasoning may increase model agreement if models anchor on the same salient features or follow similar reasoning paths. On the other hand, stochastic reasoning traces may introduce additional branch points leading to more diverse predictions. We resolve this tension by distinguishing reasoning as an intervention from reasoning as an observed behavior. We compare agreement under controlled changes in reasoning effort with agreement across observed reasoning lengths, measuring both against the agreement expected from model accuracy. Across nine models from two developers and three classification tasks, enabling reasoning increases agreement between models evaluated on the same individuals. Yet the individuals who elicit longer reasoning traces receive less consistent predictions across models. Both patterns persist after accounting for differences in accuracy. Asking models to reason more thus promotes agreement, while observing them reason longer signals disagreement.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reasoning language models, agreement, model multiplicity, algorithmic monoculture

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_33CKOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the impact of reasoning capability on homogenization/multiplicity in language model predictions, and deliberately distinguishes two questions: first, whether enabling reasoning as an intervention changes inter-model agreement; second, how actual generated reasoning length, as observed behavior, relates to inter-model agreement and individual recourse.

Critical strengths
  • The empirical analysis is relatively solid: multiple models, multiple tasks, and multiple resamples, and it attempts to use statistical techniques (e.g. robustness checks such as length binning, raw tokens, z-scores) to separate model performance from the relationship between inter-model agreement.
  • Distinguishing “reasoning as intervention” from “reasoning length as behavior” has conceptual value.
Critical weaknesses
  • The analyze setup is bloated but incomplete.
    • Causal interpretation and confounder control are insufficient: reasoning length is an observed variable, and long traces may merely be a proxy for “harder/noisier samples”; the accuracy-based null does not control for item difficulty.
    • The independent-error baseline is uninterpretable: it does not control for item difficulty or model-family correlation, so κ and excess agreement may be severely overestimated.
  • There are many analysis results, but they provide no practically meaningful guidance.
Questions for authors
  1. Appendix A.3 states: “We use temperature 1.0 with reasoning enabled and greedy decoding (temperature 0.0) with reasoning disabled.” Then when comparing reasoning effort = off/on, is it reasoning that causes the change in agreement or temperature decoding? Can you run a control with reasoning on + temp 0 and reasoning off + temp 1? If the models do not support this, at least report a temperature ablation or state that the conclusions are limited.
  2. As a suggestion to make this paper more complete, I think it is important to provide practically meaningful guidance beyond the paper’s conclusions. For example, in automatic evaluation tasks, what reasoning configuration should we adopt? How many models should be used for majority vote?
Reviewer_wU2uOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines how reasoning affects agreement across model. Across nine models on three socioeconomic prediction tasks, reasoning-enabled configurations produce greater accuracy-adjusted agreement and increase the proportion of individuals misclassified by all nine models. Within a fixed reasoning setting, however, individuals eliciting longer traces receive less consistent predictions. The authors connect these findings to shared errors and dependence on model choice, showing that greater agrement can concentrate mistakes while greater disagreement does not eliminate unanimous failures.

Critical strengths
  • The problem formulation is interesting. The authors carefully distinguishes interventions on reasoning settings from observational associations with trace length, avoiding the claim that longer traces themselves cause disagreement.
  • The experiments are extensive, where the authors experiment thousands of individuals per task, repeated generations, and checks using raw agreement, alternative binning, and different length measures support the descriptive findings.
Critical weaknesses
  • The central intervention does not isolate reasoning. Reasoning-enabled runs use temperature 1.0, while disabled runs use greedy decoding at temperature 0.0. The experiment therefore changes reasoning and sampling simultaneously. Matched-temperature controls are needed to attribute the agreement increase specifically to reasoning.
  • Classification thresholds are fitted separately for every model, reasoning setting, and resample. This is legitimate calibration, but agreement changes can reflect both model outputs and altered decision boundaries. Fixed-threshold sensitivity checks would clarify the source of the effect.
  • The baseline assumes errors are uniformly distributed across individuals. If the same individuals are difficult for every model, excess agreement can arise without an additional shared-error mechanism. The authors acknowledge this, but do not test robustness to a difficulty-aware baseline. Prior works [1] have shown this can materially change conclusions about excess agreement.
  • All three tasks involve binary socioeconomic predictions from two U.S. Census sources, and the nine models come from only two developer ecosystems. The findings do not yet establish how reasoning affects agreement on mathematics, coding, open-ended generation, or other substantially different tasks.

[1] Jo, N., Garg, N., & Raghavan, M. (2026). The Subjectivity of Monoculture. arXiv preprint arXiv:2602.24086.

Questions for authors

see weaknesses above

Reviewer_63fCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how reasoning affects agreement among language models predicting people's income and employment status. Across nine models and three tasks, enabling reasoning increases agreement but also shared errors, while longer reasoning on individual samples is associated with greater disagreement. The findings suggest that greater agreement does not necessarily imply greater reliability.

Critical strengths

The paper’s critical strengths are listed below:

  • Distinguishing reasoning effort from observed reasoning length. The paper distinguishes between changing reasoning settings and observing how much models reason on different inputs. This is a useful distinction, as the two analyses answer different questions and should not be interpreted in the same way.
  • Analysis of shared errors. The paper goes beyond pairwise agreement by examining how many models correctly classify each individual. It finds that both unanimous correct predictions and unanimous errors become more frequent when reasoning is enabled, highlighting that increased agreement does not necessarily improve reliability.
Critical weaknesses

The paper’s key weaknesses are listed below:

  • Violation of formatting requirements. The paper substantially exceeds the permitted text width, allowing the authors to present significantly more content than submissions that follow the formatting guidelines. I consider this unfair to other authors and a serious violation of the submission requirements. Therefore, I recommend rejection. Given the large number of submissions to ICLR 2027 and the heavy reviewing workload, I find it difficult to understand why this paper was not desk-rejected. The additional text made possible by violating the width limit gives the paper a substantial and unfair advantage over other submissions. In my view, accepting a paper with such a violation would raise serious questions about the fairness of the review process and the enforcement of formatting requirements.
  • Limited novelty and methodological weaknesses. The paper largely builds on Gorecki & Hardt (2025), reusing three of their prediction tasks, agreement measures, independent-error baselines, and recourse framework. Moreover, Liu (2026) already studied inter-model agreement and correlated errors among reasoning models. The main new contribution is therefore the comparison of inter-model agreement across different reasoning settings (e.g., with and without reasoning). However, this comparison is confounded by different decoding strategies (temperature 1.0 vs. greedy decoding), making it difficult to attribute the observed differences to reasoning itself. The other main finding, relating longer reasoning traces to greater disagreement, is also related to prior results by Hägele et al. (2026) on within-model inconsistency. Overall, I find the contribution too incremental, especially given these methodological limitations.

[1]: Gorecki, M., & Hardt, M. (2026). Monoculture or Multiplicity: Which Is It?. Advances in Neural Information Processing Systems, 38, 163422-163482.

[2]: Liu, N. (2026). LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning. arXiv preprint arXiv:2607.10139.

[3]: Hägele et al. (2026). The hot mess of AI: How does misalignment scale with model intelligence and task complexity? The Fourteenth International Conference on Learning Representations, 2026.

Questions for authors

The comparison between reasoning and non-reasoning uses different decoding strategies (temperature 1.0 vs. greedy decoding). Can you provide an experiment controlling for this difference, at least for the open-weight models, to establish that the observed increase in agreement is due to reasoning rather than decoding?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.