acceptodds
Under review as a conference paper at ICLR 2027

Beyond Token Entropy: Measuring Distinct 2 Forms Of Output Variation In LLM Reasoning

13 views

Abstract

Reinforcement-learning-based post-training has made output diversity an increasingly important concept in LLM reasoning, but “diversity” can refer to several behaviorally distinct forms of variation. We study three observable proxies: lexical richness, semantic dispersion, and equation-pattern variation. In an original three-model pilot across StrategyQA, MMLU College Mathematics, and ARC-Challenge, the proxies exhibit markedly different pooled associations with reasoning accuracy. A benchmark-centered reanalysis shows that the lexical-richness association remains comparatively stable (r ≈ 0.784), semantic dispersion is near zero (r ≈ 0.019), and the initially strong negative equation-pattern association is substantially attenuated (r ≈ −0.405 versus pooled r = −0.834). We therefore interpret the study as evidence that output diversity is measurement- and benchmark15 dependent rather than as evidence that any diversity axis causally improves or harms reasoning. We additionally report an expanded five-model evaluation spanning 1.5B and 7B checkpoints on GSM8K, MMLU College Mathematics, and ARC-Challenge. The expanded study exposes substantial variation in accuracy, generation truncation, and answer-extraction reliability, reinforcing the need to separate behavioral measurements from inference-pipeline artifacts. Together, these results motivate more careful, axis-specific measurement and controlled training interventions before diversity-based prescriptions are made for RL algorithms.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, RLVR, LLM reasoning, output diversity, out-of-distribution generalization, diversity decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ppWiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines whether different measures of output variation have different associations with reasoning accuracy. It compares within-response lexical richness, sentence-embedding dispersion, and equation-pattern variation using three 1.5B checkpoints on three benchmarks. The main analysis compares pooled correlations with correlations after removing benchmark-level means. An additional five-model evaluation reports accuracy, truncation, and answer-extraction failures, but does not independently replicate the diversity analysis.

Critical strengths

I think the benchmark-centered analysis gives a concrete illustration of how pooling different tasks can change the apparent relationship between an output statistic and accuracy. The authors also appropriately distinguish these observational associations from causal effects of RL training.

Critical weaknesses
  1. The main analysis is too small. It contains three checkpoints, 30 questions per benchmark, and only nine model–benchmark observations. The expanded evaluation does not independently reproduce the diversity results because complete generations were not retained. I therefore find it difficult to tell whether the reported correlations are stable findings or artifacts of the selected models and questions.

  2. I am not convinced that the proxies adequately measure the intended forms of output variation. TTR is sensitive to response length, while the equation statistic counts surface patterns rather than distinct reasoning strategies. With a 150-token generation limit, truncation is another plausible confound. The paper acknowledges these issues but does not show that its conclusions survive controls for them.

Questions for authors

The question I want to ask can be found in the Weaknesses. If it can be properly resolved, I would be willing to increase the score. However, given the current situation, I also have concerns that this might have a significant impact on the content of the paper and require the author to make a major revision.

Reviewer_Vm7yOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

N/A.

Critical strengths

None.

Critical weaknesses

I recommend desk rejection because the submission does not follow the required ICLR 2027 format. The abstract spans the full text width instead of using the required narrower block, and the line numbering differs from the official template. The formatting instructions require the official style files and state that modifying them may be grounds for rejection.

The VAPO reference also omits the first author, Yu Yue. This gives incorrect credit for the cited work and needs correction.

Questions for authors

N/A.

Reviewer_DRvBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies three proxies: lexical richness, semantic dispersion, and equation-pattern variation, and compares their pooled associations with reasoning accuracy against benchmark-centered associations in a three-model pilot. An expanded five-model evaluation further reports heterogeneity in accuracy, truncation, and answer-extraction reliability, though it is not an independent replication of the diversity correlations. The paper concludes that observed diversity–accuracy relationships are proxy- and benchmark-dependent and should be interpreted observationally rather than causally.

Critical strengths
  1. The paper makes a useful measurement point: “diversity” is not a single scalar, and pooled correlations can conflate between-benchmark differences with within-benchmark model variation.

  2. The paper is transparent about evaluation artifacts and easy to follow.

Critical weaknesses
  1. The empirical basis is too thin for a main-conference paper. The diversity analysis rests on only three models, three benchmarks, and nine non-independent aggregate cells, and the expanded five-model study is not an independent replication because raw generations were not retained.

  2. The proxies are weak and the conclusions remain only cautionary. TTR is length-sensitive, Sentence-BERT full-response distance is encoder-dependent, and the regex equation count is surface-level; the paper therefore does not establish a robust diversity–accuracy relationship beyond “measurement matters

Questions for authors
  1. What is the single falsifiable claim this paper establishes beyond “diversity measurement matters”? Please state it explicitly and identify the evidence that would falsify it.

  2. Can you retain and release the complete raw generations for the expanded five-model evaluation, and recompute the diversity–accuracy associations as an independent replication? Without this, the expanded study does not test the paper’s central correlations.

  3. The current experimental scale is too small to support the paper’s conclusions: three models, three benchmarks, and nine aggregate cells, with only 25 questions per benchmark in the expanded run. Can you substantially expand the model, benchmark, and question coverage and show that the benchmark-centered associations remain stable?

1 comment

No author response was provided during the rebuttal period. Therefore, the main concerns raised in my original review remain unresolved, and I maintain my score of 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.