- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the concentration of correct responses on fewer solution strategies as RL improves language-model accuracy. Based on a simplified GRPO model, the authors analyze the relationship between correct-solution diversity and accuracy and characterize the conditions under which an exponential approximation holds. The paper also reanalyzes training logs to compare token entropy with correct-solution diversity and fits 41 published entropy–accuracy curves.
Critical strengths
- The authors attempt to provide mathematical relations between entropy and accuracy.
- The paper compares token entropy with the number of groups of correct responses, showing that the two measures may disagree.
Critical weaknesses
- As the authors acknowledge, the core dynamics, concentration mechanism, and intervention come from prior work. The substantive contribution of the new entropy analysis needs strengthening.
- The paper characterizes largely expected early concentration but leaves the more interesting later diversity recovery unexplained, limiting its explanatory power for actual training.
Questions for authors
- The theory assumes that the probabilities of different solution types can be updated through independent parameters. The authors should clarify whether the resulting competition mechanism also holds in language models with shared parameters and provide supporting evidence.
- The authors should consolidate the repeated training trajectories in Figures 1–4 and explain how the observations relate to the theoretical results in one place, clarifying the main thread of the empirical analysis.
3 comments
Follow-up: a measured answer to Q1 (shared parameters)
Thank you for pressing on parameter sharing. We add additional experiment, we ran a synthetic GRPO experiment where we can read every mode probability directly. Each prompt has 16 modes, four correct. Training uses 16 sampled rollouts per prompt with standardized advantages. At the checkpoint where mean training accuracy reached 0.60 (from 0.25), we compared the predicted and observed changes in each correct mode’s log-share.
| Policy | Calibration slope (target 1) | Initial leader gained share |
|---|---|---|
| Independent logits, SGD | 0.99 | 96% |
| Shared network, input dimension 4096 | 1.02 | 75% |
| Shared network, input dimension 256 | 0.66 | 59% |
| Shared network, input dimension 16 | 0.28 | 52% |
These are means over five seeds, with 200 prompts measured per seed. The shared networks used Adam, and the ordering across dimensions also held with SGD. With independent logits, Proposition 7.1 predicted correct-mode Shannon entropy with an RMSE of 0.05 nats, compared with 0.45 for predicting no change.
The clearest limit appeared in a separate shared-network run during the first pass over 16k prompts. Correct-mode collision rose from 0.38 to 0.44. The theory predicted 0.61, the mode-level calibration slope was 0.04, and the initial leader gained share on 41% of prompts. So aggregate concentration can occur even when the theory predicts the individual mode changes poorly.
This sharpens how we interpret the 14B evidence. In our fixed-prompt panel, the initial leader gained share on 81% of prompts, which supports preferential amplification in that run. How much of that amplification came from updates on other prompts remains open. On your question about late recovery, the 14B increase in family count followed a batch-size change, so its cause also remains open. We’ll include the synthetic test and code and make both limits explicit in the revision.
We hope this experiment helps answer your question about competition between correct solutions when parameters are shared. If there’s a specific comparison you’d find helpful, we’d be happy to look into it before the discussion period ends.
I appreciate the authors' efforts in the rebuttal and their frankness regarding the limitations. My concerns have been partially addressed, and I will raise my score to 3. I encourage the authors to incorporate the proposed changes into the revision.
Thank you for the review. Your two questions shaped the main changes planned for the final version.
W1. Contribution. We agree that the collision mechanism, concentration and intervention come from prior work, and we will not present concentration as a new result. We do not agree that the entropy analysis adds little. Concentration tells you where y ends up. It does not tell you the accuracy–entropy curve, when that curve looks exponential or logistic, or that both forms must fail in the terminal regime, where Hy∼C∗pM/(M+1)∣logp∣. Those are the new results (table in general response A), and the panel now tests the curve directly (general response D). The family/sequence/token decomposition then shows why a token-entropy reading can mislead.
W2. "Expected" early concentration, unexplained recovery. Early concentration is the mechanism of Rad et al. (2026b), and we agree it is expected. On the recovery (details in general response C):
Q1. Shared parameters. Not automatically. General response C gives the condition: a linearized kernel formula under which uniform sharing changes only the time scale, exchangeable similarity preserves the competition, and heterogeneous similarity changes it. That is a condition, not evidence.
The evidence from the logs is indirect. From the same first batch, GRPO raises the correct-family collision, while an intervention built to reverse the independent-logit field lowers it. Family counts are typically 4–6 times higher on 14B after step 150 (at least 3.2 times to the end of the run), and about 3–5 times higher at the end of the 7B runs. The bonus acts on the statistic we measure, and there is one run per arm, so this is consistent with the condition but does not test it. The logs also show that the concentration happens on prompts not yet trained on, so in the network it is carried across prompts by shared parameters (general response C).
We tested the form vj∝yj directly on the fixed-prompt panel (general response D). The slope of family log-share change on family share is 6.1 (95% CI [4.9, 7.3]), and the initial leader gains share in 81% of prompts. So in this network, on this run, the competition has the sign the independent-logit model gives.
Q2. Consolidating Figs. 1–4. Agreed. We will use one empirical figure with one story:
The prediction → observation table from general response B will sit next to it.
We hope these comments address your concerns.