- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper examines whether different measures of output variation have different associations with reasoning accuracy. It compares within-response lexical richness, sentence-embedding dispersion, and equation-pattern variation using three 1.5B checkpoints on three benchmarks. The main analysis compares pooled correlations with correlations after removing benchmark-level means. An additional five-model evaluation reports accuracy, truncation, and answer-extraction failures, but does not independently replicate the diversity analysis.
Critical strengths
I think the benchmark-centered analysis gives a concrete illustration of how pooling different tasks can change the apparent relationship between an output statistic and accuracy. The authors also appropriately distinguish these observational associations from causal effects of RL training.
Critical weaknesses
-
The main analysis is too small. It contains three checkpoints, 30 questions per benchmark, and only nine model–benchmark observations. The expanded evaluation does not independently reproduce the diversity results because complete generations were not retained. I therefore find it difficult to tell whether the reported correlations are stable findings or artifacts of the selected models and questions.
-
I am not convinced that the proxies adequately measure the intended forms of output variation. TTR is sensitive to response length, while the equation statistic counts surface patterns rather than distinct reasoning strategies. With a 150-token generation limit, truncation is another plausible confound. The paper acknowledges these issues but does not show that its conclusions survive controls for them.
Questions for authors
The question I want to ask can be found in the Weaknesses. If it can be properly resolved, I would be willing to increase the score. However, given the current situation, I also have concerns that this might have a significant impact on the content of the paper and require the author to make a major revision.
No author response was provided during the rebuttal period. Therefore, the main concerns raised in my original review remain unresolved, and I maintain my score of 1.