- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the impact of reasoning capability on homogenization/multiplicity in language model predictions, and deliberately distinguishes two questions: first, whether enabling reasoning as an intervention changes inter-model agreement; second, how actual generated reasoning length, as observed behavior, relates to inter-model agreement and individual recourse.
Critical strengths
- The empirical analysis is relatively solid: multiple models, multiple tasks, and multiple resamples, and it attempts to use statistical techniques (e.g. robustness checks such as length binning, raw tokens, z-scores) to separate model performance from the relationship between inter-model agreement.
- Distinguishing “reasoning as intervention” from “reasoning length as behavior” has conceptual value.
Critical weaknesses
- The analyze setup is bloated but incomplete.
- Causal interpretation and confounder control are insufficient: reasoning length is an observed variable, and long traces may merely be a proxy for “harder/noisier samples”; the accuracy-based null does not control for item difficulty.
- The independent-error baseline is uninterpretable: it does not control for item difficulty or model-family correlation, so κ and excess agreement may be severely overestimated.
- There are many analysis results, but they provide no practically meaningful guidance.
Questions for authors
- Appendix A.3 states: “We use temperature 1.0 with reasoning enabled and greedy decoding (temperature 0.0) with reasoning disabled.” Then when comparing reasoning effort = off/on, is it reasoning that causes the change in agreement or temperature decoding? Can you run a control with reasoning on + temp 0 and reasoning off + temp 1? If the models do not support this, at least report a temperature ablation or state that the conclusions are limited.
- As a suggestion to make this paper more complete, I think it is important to provide practically meaningful guidance beyond the paper’s conclusions. For example, in automatic evaluation tasks, what reasoning configuration should we adopt? How many models should be used for majority vote?