acceptodds
Under review as a conference paper at ICLR 2027

Winner-Take-All: Where Exponential Entropy Collapse Comes From in Reinforcement Learning for LLMs

Abstract

Reinforcement learning with verifiable rewards (RLVR) can improve accuracy while concentrating probability on fewer correct solutions. We study this process through a finite-mode, clean-verifier mean-field model of group relative policy optimization (GRPO), an algorithm used for RLVR. The correct-mode dynamics increase collision probability and decrease both Shannon and Rényi-2 entropy; a unique initial leader eventually captures all correct mass. We derive exact entropy–mass relations along a scalar trajectory and distinguish this mode-level concentration from token entropy through an entropy decomposition over solution families and token sequences. The distinction matters empirically: in a GRPO training run, correct-family diversity falls sharply early in training and only partially recovers, while token entropy subsequently rises above its initial level. Comparing unmodified GRPO with an anti-collision variant, family-count ratios remain between 4.4 and 5.9 over the tested training cutoffs, whereas token-entropy ratios cross unity. The scalar solution also characterizes when accuracy–entropy curves admit shifted-exponential or logistic approximations: each corresponds to a locally constant slope in a different transformed error coordinate. Fits to training curves across model families and sizes on math and coding tasks support both finite-window descriptions. These results motivate measuring correct-family collision alongside token entropy when assessing solution diversity in RLVR.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, GRPO, entropy collapse, solution diversity, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7tQ6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the concentration of correct responses on fewer solution strategies as RL improves language-model accuracy. Based on a simplified GRPO model, the authors analyze the relationship between correct-solution diversity and accuracy and characterize the conditions under which an exponential approximation holds. The paper also reanalyzes training logs to compare token entropy with correct-solution diversity and fits 41 published entropy–accuracy curves.

Critical strengths
  • The authors attempt to provide mathematical relations between entropy and accuracy.
  • The paper compares token entropy with the number of groups of correct responses, showing that the two measures may disagree.
Critical weaknesses
  • As the authors acknowledge, the core dynamics, concentration mechanism, and intervention come from prior work. The substantive contribution of the new entropy analysis needs strengthening.
  • The paper characterizes largely expected early concentration but leaves the more interesting later diversity recovery unexplained, limiting its explanatory power for actual training.
Questions for authors
  • The theory assumes that the probabilities of different solution types can be updated through independent parameters. The authors should clarify whether the resulting competition mechanism also holds in language models with shared parameters and provide supporting evidence.
  • The authors should consolidate the repeated training trajectories in Figures 1–4 and explain how the observations relate to the theoretical results in one place, clarifying the main thread of the empirical analysis.
3 comments

Thank you for the review. Your two questions shaped the main changes planned for the final version.

W1. Contribution. We agree that the collision mechanism, concentration and intervention come from prior work, and we will not present concentration as a new result. We do not agree that the entropy analysis adds little. Concentration tells you where yy ends up. It does not tell you the accuracy–entropy curve, when that curve looks exponential or logistic, or that both forms must fail in the terminal regime, where Hy∼C∗pM/(M+1)∣log⁡p∣H_y\sim C_*p^{M/(M+1)}|\log p|. Those are the new results (table in general response A), and the panel now tests the curve directly (general response D). The family/sequence/token decomposition then shows why a token-entropy reading can mislead.

W2. "Expected" early concentration, unexplained recovery. Early concentration is the mechanism of Rad et al. (2026b), and we agree it is expected. On the recovery (details in general response C):

  • Token-entropy recovery is not evidence of regained diversity. In the 14B GRPO run token entropy rises from ≈0.15 to ≈1.1 nats between steps 60 and 100 while the family count stays at ≈1.3.
  • Family diversity stays collapsed at ≈1.33 for about 120 steps after training accuracy levels off, at constant configuration. That is consistent with the monotone model. It rises only after the run was resumed with half the batch size. So in our data the recovery is confounded with a configuration change, and we cannot attribute it to GRPO dynamics. We will mark the change on the figures and list as a limitation that the existing runs cannot show whether recovery occurs at a constant batch size.

Q1. Shared parameters. Not automatically. General response C gives the condition: a linearized kernel formula under which uniform sharing changes only the time scale, exchangeable similarity preserves the competition, and heterogeneous similarity changes it. That is a condition, not evidence.

The evidence from the logs is indirect. From the same first batch, GRPO raises the correct-family collision, while an intervention built to reverse the independent-logit field lowers it. Family counts are typically 4–6 times higher on 14B after step 150 (at least 3.2 times to the end of the run), and about 3–5 times higher at the end of the 7B runs. The bonus acts on the statistic we measure, and there is one run per arm, so this is consistent with the condition but does not test it. The logs also show that the concentration happens on prompts not yet trained on, so in the network it is carried across prompts by shared parameters (general response C).

We tested the form vj∝yjv_j\propto y_j directly on the fixed-prompt panel (general response D). The slope of family log-share change on family share is 6.1 (95% CI [4.9, 7.3]), and the initial leader gains share in 81% of prompts. So in this network, on this run, the competition has the sign the independent-logit model gives.

Q2. Consolidating Figs. 1–4. Agreed. We will use one empirical figure with one story:

  1. early concentration (consistent with the mechanism);
  2. later divergence of token entropy and family diversity (consistent with the decomposition);
  3. the intervention comparison (field reversal);
  4. the limit of the model's scope (the late recovery, with the batch-size change marked).

The prediction → observation table from general response B will sit next to it.

We hope these comments address your concerns.

AuthorsOpenReview ↗

Follow-up: a measured answer to Q1 (shared parameters)

Thank you for pressing on parameter sharing. We add additional experiment, we ran a synthetic GRPO experiment where we can read every mode probability directly. Each prompt has 16 modes, four correct. Training uses 16 sampled rollouts per prompt with standardized advantages. At the checkpoint where mean training accuracy reached 0.60 (from 0.25), we compared the predicted and observed changes in each correct mode’s log-share.

PolicyCalibration slope (target 1)Initial leader gained share
Independent logits, SGD0.9996%
Shared network, input dimension 40961.0275%
Shared network, input dimension 2560.6659%
Shared network, input dimension 160.2852%

These are means over five seeds, with 200 prompts measured per seed. The shared networks used Adam, and the ordering across dimensions also held with SGD. With independent logits, Proposition 7.1 predicted correct-mode Shannon entropy with an RMSE of 0.05 nats, compared with 0.45 for predicting no change.

The clearest limit appeared in a separate shared-network run during the first pass over 16k prompts. Correct-mode collision rose from 0.38 to 0.44. The theory predicted 0.61, the mode-level calibration slope was 0.04, and the initial leader gained share on 41% of prompts. So aggregate concentration can occur even when the theory predicts the individual mode changes poorly.

This sharpens how we interpret the 14B evidence. In our fixed-prompt panel, the initial leader gained share on 81% of prompts, which supports preferential amplification in that run. How much of that amplification came from updates on other prompts remains open. On your question about late recovery, the 14B increase in family count followed a batch-size change, so its cause also remains open. We’ll include the synthetic test and code and make both limits explicit in the revision.

We hope this experiment helps answer your question about competition between correct solutions when parameters are shared. If there’s a specific comparison you’d find helpful, we’d be happy to look into it before the discussion period ends.

I appreciate the authors' efforts in the rebuttal and their frankness regarding the limitations. My concerns have been partially addressed, and I will raise my score to 3. I encourage the authors to incorporate the proposed changes into the revision.

Reviewer_BbpaOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates entropy collapse in RLVR through a simplified GRPO mean-field model. The authors theoretically characterize the winner-take-all dynamics of correct solution modes and distinguish solution-level diversity from token-level entropy. They further derive accuracy–entropy relationships and identify conditions for exponential and logistic approximations. Empirical analyses suggest that token entropy alone may not reliably reflect correct-solution diversity during RL training.

Critical strengths
  1. The paper provides rigorous derivations of Shannon and Rényi-2 entropy dynamics, along with theoretical characterizations of accuracy–entropy relationships.

  2. The experiments reveal that token entropy and correct-solution diversity can exhibit different trends during RL training, offering useful insights into diversity monitoring.

Critical weaknesses
  1. My main concern is the disconnect between the theoretical analysis and empirical evidence. The experiments only demonstrate early-stage entropy collapse without directly validating the predicted collision dynamics. Moreover, the theoretical accuracy–entropy relationship is derived using correct-mode entropy, whereas the empirical evaluation relies on aggregate token entropy. These discrepancies raise concerns about whether the theoretical findings accurately characterize real LLM training dynamics.

  2. The core collision dynamics and scalar parameterization are inherited from Rad et al. (2026b), while the family-diversity measurements and anti-collision experiments largely build on Rad et al. (2026a). It remains unclear whether the additional entropy analysis provides sufficiently novel insights beyond these existing works.

I would be happy to raise my score if the authors can adequately address these concerns.

Questions for authors

None

3 comments

Thank you for the review and for the offer to reconsider. We take W2 first, since it frames W1.

W2. Novelty. (We have addressed this question in the General response (1/4) as well)

The collision dynamics, concentration of yy, the scalar parameterization and the anti-collision method are due to Rad et al. (2026a,b), and we will attribute them explicitly (table in general response A). What is new is the entropy–accuracy geometry those dynamics induce:

  • the exact parametric curve (Prop. 7.1);
  • separate conditions for the two finite-window laws, exponential when dlog⁡p/dHyd\log p/dH_y is near-constant and logistic when d logit p/dHyd\,\mathrm{logit}\,p/dH_y is, with errors bounded by slope variation and window width (Cor. 7.2);
  • a terminal law, Hy∼C∗pM/(M+1)∣log⁡p∣H_y\sim C_*p^{M/(M+1)}|\log p|, so neither is an exact global relation.

These do not follow from concentration alone. The curve needs the coupled correct/incorrect clocks, and the terminal exponent depends on the number of incorrect modes. They hold within the mean-field model, for correct-mode entropy. Whether they also explain the law of Cui et al. (2025) on aggregate token entropy is a hypothesis we do not establish.

W1. Theory vs. experiments. We agree, and separate two claims. (a) Token entropy and correct-family diagnostics can disagree: supported. (b) Neural training follows the collision field: in the original experiments, supported only through early concentration on one backbone. We now test it directly (below). The fits on aggregate token entropy compare curve shapes and do not validate the mechanism. We will say so.

The per-step logs give four results that are relevant to (b). None is a direct test (details in general response B and C):

  • Direction. From the same first batch, GRPO takes the collision from 0.52 to 0.94 by step 50. The anti-collision bonus, built to reverse the within-correct field, lowers it.
  • Sample size. The number of correct samples per prompt is similar in the two arms, and a worst-case multinomial bias correction leaves the rise intact (≤0.47 → ≥0.88).
  • Persistence. The collapsed state lasts about 120 steps at constant configuration.
  • Family entropy vs. accuracy, for our own 14B GRPO run (7-step means):
StepTraining accuracy−log⁡sˉ2-\log\bar s_2 (nats)Token entropy (nats)
10.260.650.70
100.350.360.32
200.410.340.22
300.490.310.11
400.540.140.11
500.590.060.11
700.640.060.18

We read this table cautiously. Token entropy reaches its floor by step ≈30, while −log⁡sˉ2-\log\bar s_2 keeps falling until step ≈50, so the two entropies separate. But −log⁡sˉ2-\log\bar s_2 is a batch-level Rényi-2 summary, not the per-prompt Shannon HyH_y of Sec. 7. Its implied slope dlog⁡p/dHd\log p/dH is also not constant (≈0.5 for steps 1–10, ≈5 for steps 10–30, ≈0.6 for steps 30–40). So it does not show a single finite-window law, and we do not claim it does.

The direct tests need per-prompt family counts matched over time, so we ran a fixed-prompt panel on a new run (general response D). Two results:

  • Mechanism. Family log-shares increase with family share: slope 6.1 (95% CI [4.9, 7.3], sign-flip p ≤ 0.0002). A positive covariance alone would only restate that collision rose, so we test the form by regression, with the baseline taken from a held-out half of the rollouts.
  • This paper's curve. From each prompt's step-0 shares, Prop. 7.1 predicts its entropy at the later error rate with no fitted parameter. Predicted and observed entropy changes agree with calibration slope 0.92 (95% CI [0.74, 1.10]; 1 is exact agreement).

We hope the new experiment, together with the clearer scope, addresses your main concern.

Thank you for the detailed response. The additional experiments have adequately addressed my concerns regarding the gap between the theoretical analysis and novelty.

I believe this work provides valuable theoretical insights into the entropy dynamics of RLVR, particularly by clarifying the relationships among token entropy, sequence entropy, and correct-solution family entropy. These findings offer useful guidance for monitoring and diagnosing LLM behavior during practical RL training.

Given the theoretical contributions, strengthened empirical evidence, and practical relevance of these insights, I will raise my rating to Weak Accept.

Thank you for taking another careful look at the paper. Your question pushed us to test the theory’s predictions more directly, and we’re glad the new experiments addressed your concerns. Our broader hope is that a clearer, effective theory of RLVR can help the community understand why some training heuristics work and use that understanding to design better algorithms. We really appreciate your support.

Reviewer_eyRWOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies diversity loss in RLVR through a finite-mode mean-field model of GRPO. It characterizes the concentration of correct solutions, derives entropy–accuracy relations and their finite-window approximations, and distinguishes correct-family diversity from sequence and token entropy. Empirical analyses show that token entropy and observed solution-family diversity can rank training configurations differently.

Critical strengths
  1. The distinction between token entropy and meaningful solution diversity is important and thoughtfully developed. The paper challenges the common practice of interpreting higher token entropy as evidence of broader reasoning strategies.
  2. The theoretical discussion provides a clear account of how equally rewarded correct solutions can nevertheless undergo winner-take-all competition. The entropy decomposition and analysis of finite-window versus terminal behavior offer useful conceptual insights.
  3. The paper is generally careful about the scope of its conclusions, acknowledging that neural training can depart from the idealized dynamics and that no single entropy–accuracy curve should be treated as universal.
Critical weaknesses
  1. The practical relevance of the collision model remains insufficiently established. Fixed support, independently adjustable mode logits, and population-level updates omit important aspects of neural optimization. The observed late-stage diversity recovery is also not captured by the monotone concentration dynamics. Although acknowledged, this gap calls for controlled experiments identifying when the mechanism is predictive and which omitted factors explain its breakdown.

  2. Section 7 includes a chronological held-out evaluation, but its predictive and mechanistic implications remain limited. The fits use aggregate logged entropy rather than the correct-mode entropy analyzed theoretically, with parameters estimated separately for each trajectory. Consequently, good curve fits do not directly validate the proposed dynamics. The paper should clarify whether held-out predictions require observed entropy at future checkpoints and provide stronger evidence concerning fitting-window sensitivity and the predicted local slopes.

  3. The empirical collision and family-count measurements are confounded by the number of correct samples. With only 16 rollouts per prompt, changes in accuracy can substantially alter this number. The plug-in collision estimator is upward biased at small sample sizes, while observed family counts also depend on sampling coverage. Appendix A.1 acknowledges these issues but cannot apply prompt-level corrections using the available summaries. Matched-correct-sample comparisons and corrected collision estimates would strengthen the central empirical claims.

  4. Embedding clusters are not necessarily equivalent to distinct reasoning strategies. Different phrasings of one strategy may form separate clusters, while genuinely different strategies may be merged. The reported embedding and clustering-threshold checks are useful, but they establish configuration robustness rather than semantic validity. Given the paper’s emphasis on meaningful diversity beyond token-level variation, a small human-annotated evaluation of solution families would be particularly valuable.

  5. The incremental contribution relative to Rad et al. (2026a,b) could be delineated more clearly. The collision dynamics and scalar parameterization come from prior work, as do the anti-collision method, family diagnostics, and empirical training panels. The paper should more explicitly distinguish its new theoretical results from consequences of the existing framework and explain what additional testable predictions or practical insights these results enable.

Questions for authors

ref weakness

4 comments

Thank you for the careful review. We answer W1–W5 below. Shared points are in the general response.

W1. Practical relevance, when the mechanism is predictive, late recovery. We agree that fixed support, independent logits and population updates leave out important parts of neural optimization. We still think the model is useful as a null model, because it makes a testable statement: family log-shares move in proportion to family share (vj∝yjv_j\propto y_j). What we can say now:

  • Where it is predictive (from the logs). On Qwen3-14B-Base, concentration is fast (sˉ2\bar s_2 from 0.52 to 0.94 by step 50) and persists for about 120 steps at constant configuration.
  • Breakdown. The late recovery in the 14B run coincides with a resume at half the batch size. The existing runs cannot tell which omitted factor causes it. We will list this as a limitation (general response C).

We ran the controlled experiment you suggested: the fixed-prompt panel (general response D). On 150 fixed prompts, family log-shares increase with family share. The slope is 6.1 (95% CI [4.9, 7.3]), and the initial leader gains share in 81% of prompts.

W2. Section 7 evaluation.

  • Forecasting. The held-out evaluation conditions on the observed entropy at each held-out checkpoint. It tests the accuracy–entropy mapping chronologically and does not forecast accuracy from early training. We will describe it that way.
  • Aggregate vs. correct-mode entropy, separate fits. Agreed. With aggregate entropy and free offsets and saturation levels, the fits are descriptive surrogates and do not validate the dynamics. The panel tests the relation in per-prompt HyH_y: the entropy change predicted by Prop. 7.1 matches the observed one with calibration slope 0.92 (general response D).
  • Window sensitivity and slopes. For the 41 curves we will report held-out RMSE for fit fractions 40–80% and the distribution of fitted slopes c^\hat c of b−aecHb-ae^{cH}. Sec. 7.3 will present both laws strictly as finite-window descriptions within the model.

W3. Dependence on the number of correct samples. This is an important point, and the logs answer part of it.

The arms have similar n. Family statistics use prompts with at least two correct rollouts. The logs record the mean incorrect fraction p^\hat p on those prompts, so the mean number of correct samples follows exactly as nˉ=G(1−p^)\bar n=G(1-\hat p). It is higher than batch accuracy alone suggests, because prompts with fewer than two correct rollouts are excluded. On Qwen3-14B it is 10.1 / 12.1 / 12.3 under GRPO and 9.6 / 11.9 / 12.3 under the anti-collision bonus, for steps 1–68 / 69–300 / 301–end. Both arms start from the same first batch.

The bias works against the early concentration. E[s^2]=s2+(1−s2)/n\mathbb E[\hat s_2]=s_2+(1-s_2)/n. nˉ\bar n rises from 6.5 to 11.8 while sˉ2\bar s_2 rises from 0.52 to 0.94. A worst-case correction over all per-prompt allocations consistent with the logged means still gives a rise from ≤0.47 to ≥0.88. Over the same nˉ\bar n path, K^\hat K falls from 3.25 to 1.29 under GRPO and rises from 3.28 to 4.24 under the bonus.

Per-prompt check. The bound assumes a fixed family partition, and density-based clustering can merge clusters as nn grows. So we measured it on the fixed-prompt panel (general response D). The unbiased pair-count estimator s~2=∑jnj(nj−1)/[n(n−1)]\tilde s_2=\sum_j n_j(n_j-1)/[n(n-1)] rises from 0.41 to 0.92 between steps 0 and 50. The family count at a matched 8 correct samples falls from 3.6 to 1.3. Family-count ratios between arms remain sample-dependent, and we will stop describing them as population-support ratios.

W4. Clusters vs. strategies. Agreed. Our ablations show robustness to embedding and threshold choices, not semantic validity. We will write "embedding-defined families" throughout.

One more limitation we will state: the sentence embedder reads only the first ≈2048 word pieces of a response, and mean response length grows from ≈1.2k to ≈6k tokens in the concentration window. So the families reflect the opening of each solution, and part of the measured concentration could be a shared opening. On the panel we also embedded whole responses. The collision still rises, from 0.38 to 0.90 (general response D).

We ran the human check on the panel. Two annotators labeled the main mathematical/algorithmic approach of 480 full responses (20 prompts at three checkpoints), blind to the checkpoint. They agree with each other at κ\kappa = 0.72 and with the clusters at ARI 0.61. The clusters separate 21% of the pairs both annotators call the same approach, and join 6% of the pairs both call different. Under the human labels the collision rises from 0.48 to 0.89, so the concentration holds. Note that (4.1) holds for any fixed partition, and Thm 3.1 for any partition whose modes have independent logits. So this check tests the measurement, not the derivations.

W5. Increment beyond prior work. Please see general response A. In short, the new part is the accuracy–entropy geometry the dynamics induce: separate conditions and error bounds for the exponential and logistic finite-window laws, and a terminal law that limits both. The decomposition showing why token entropy need not track family concentration is also new. We will reorganize the contribution statement around these.

Thank you for your detailed and candid response, which addresses most of my concerns. The paper still has some limitations in the practical utility of its theory and the scope of its originality, so I cannot give it a score of 4; however, overall it would be a solid ICLR paper. Accordingly, I will maintain my positive scores and increase my confidence.

Thank you for reading the rebuttal so carefully. We’re glad the additional analysis addressed most of your concerns. Your questions helped us be more precise about where the theory is useful in practice and what this paper adds to our earlier work.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response (1/4)

General response (1/4): Contribution

We're very grateful to the reviewers. Three points were raised and we address them here:

(A) the contribution,

(B) theory versus measurements,

(C) shared parameters and the late recovery.

Reviewer specific points are in the individual replies. We've also re-analysed the per-step logs for every training run and have a new run for the fixed prompt panel. Results are in B, C and D. All of the changes below are incorporated into the final version.


A. Contribution. We view this work as a natural extension of the framework developed in Rad et al. (2026a,b), representing a third step in this line of research toward understanding and improving RLVR from a mean-field perspective. We believe this framework provides a powerful tool for studying RLVR dynamics, which are otherwise difficult to capture theoretically. We therefore position our work as a further application of this framework, with the accuracy–entropy trade-off being one interesting problem it allows us to address

Inherited: the collision dynamics, the winner take all concentration of yy, the scalar parameterisation, the anti-collision intervention and the training runs. We will be very clear about all of those.

New: what those dynamics mean for entropy and for the accuracy-entropy relations. The fact that yy concentrates doesn't tell you anything about the shape of the accuracy-entropy curve. That comes from the coupled correct/incorrect clocks (7.2). The curve is the model's version of the empirical law of Cui et al. (2025). We derive it for the correct mode entropy within the mean field model. Whether it is also the origin of the law which we see in the aggregate token entropy is a hypothesis. We'll state that as our hypothesis in the abstract and in the introduction.

ComponentSource
Collision ODE (2.3)–(2.4), reciprocal parameterization, concentration of yy onto the initial leaderRad et al. (2026b)
Anti-collision bonus, embedding-cluster protocol, GRPO/anti-collision runsRad et al. (2026a)
Exact accuracy–entropy curve (Prop. 7.1), hazard identities (7.5)–(7.7)New
Finite-window conditions: exponential when dlog⁡p/dHyd\log p/dH_y is near-constant, logistic when d logit p/dHyd\,\mathrm{logit}\,p/dH_y is. Error bounded by slope variation × window width (Cor. 7.2, App. G)New
Terminal law Hy∼C∗pM/(M+1)∣log⁡p∣H_y\sim C_*p^{M/(M+1)}\lvert\log p\rvert (App. D). Neither form is an exact global lawNew
Shannon-entropy monotonicity in both blocks, with opposite signs in the incorrect block. Limit of the full mode entropy (Thm 3.1b,c)New
Family / sequence / token entropy decomposition and prefix-tree lift (Sec. 4, App. F)New

We don't think these can be read off the prior work. Two examples: the exponential form requires a near-constant dlog⁡p/dHyd\log p/dH_y and this clearly breaks down as p→0p\to0 in the terminal law. The terminal exponent M/(M+1)M/(M+1) is a function of the number of incorrect modes, not the number of correct ones. There's also a practical consequence in that token entropy does not capture family diversity (4.1)-(4.2) and so the two can well give different rankings to a configuration as is evident from Fig. 3.

We'll move the accuracy-entropy analysis to the front of the paper to convey our message and result, earlier.

AuthorsOpenReview ↗

General response (2/4)

General response (2/4): theory vs. measurements

B. The paper mixed two empirical claims:

  • (a) Token entropy and correct-family diagnostics can disagree, even in how they rank configurations.
  • (b) Neural training follows the collision field.

The original experiments support (a). For (b) their only support was early concentration on Qwen3-14B-Base experiments. The 41-curve comparison (Sec. 7.3) uses aggregate entropy. It compares curve shapes and does not validate the collision field. We will say so.

What the per-step logs show (values are 7-step means):

  • Sample size. Family statistics use prompts with at least two correct rollouts. On those, the mean number of correct samples is similar in the two 14B arms (within 6%). Under a worst-case multinomial bias correction, GRPO's collision still rises from ≤0.47 to ≥0.88 (details in the reply to eyRW, W3).
  • Direction. Both 14B runs start from the same first batch. GRPO takes sˉ2\bar s_2 from 0.52 to 0.94 by step 50 (the paper quotes these values at step 68). The anti-collision bonus takes it from 0.51 to 0.42 by step 68. Caveat: the bonus acts on the statistic we measure, so this shows the direction of the intervention, not the mechanism.
  • Persistence. K^\hat K then stays at ≈1.33 for about 120 steps at constant configuration (see C).
  • Family entropy vs. accuracy. On 14B GRPO, −log⁡sˉ2-\log\bar s_2 falls from 0.65 to 0.06 nats by step 50 while training accuracy rises from 0.26 to 0.59 (table in the reply to Bbpa). Token entropy reaches its floor by step ≈30, and −log⁡sˉ2-\log\bar s_2 keeps falling until step ≈50. Two caveats: this is a batch-level Rényi-2 summary, not the per-prompt Shannon HyH_y of Sec. 7, and its slope against log⁡p\log p is not constant. It is descriptive.

The logs cannot test (b) directly. For any logit velocities vjv_j, s˙2=2Covy(yj,vj)\dot s_2=2\mathrm{Cov}_y(y_j,v_j) and H˙y=−Covy(vj,log⁡yj)\dot H_y=-\mathrm{Cov}_y(v_j,\log y_j). These are identities, so a positive covariance only says that collision rose. The content of the model is the form vj∝yjv_j\propto y_j. Testing it needs per-prompt family counts matched over time, and the original runs kept only batch statistics.

So we ran a fixed-prompt panel: the same 150 prompts sampled at steps 0, 20 and 50 of a new GRPO run, with families matched across checkpoints. Full results are in D. In short:

  • the slope of family log-share change on family share is 6.1 (95% CI [4.9, 7.3]);
  • the entropy change predicted by Prop. 7.1, with no fitted parameter, matches the observed one with calibration slope 0.92 (1 is exact agreement);
  • the unbiased incorrect-family collision falls from 0.31 to 0.19.
Prediction (mean-field model)ObservationStatus
Correct-family collision rises under GRPO (Thm 3.1a)14B, by step 50: sˉ2\bar s_2 0.52 → 0.94, K^\hat K 3.25 → 1.29. Anti-collision run from the same start: 0.51 → 0.42Consistent on 14B
Concentration persists (Thm 3.1c)K^≈1.33\hat K\approx1.33 over steps 60–180. Rises after a resume with halved batch size (C)Consistent at constant configuration. Later rise confounded
vj∝yjv_j\propto y_j (Eq. 3.4)Fixed-prompt panel: slope 6.1 (CI [4.9, 7.3]). Leader gains share in 81% of prompts (D)Consistent on one run
Accuracy–entropy curve in HyH_y (Prop. 7.1, Cor. 7.2)Fixed-prompt panel: calibration slope 0.92 (CI [0.74, 1.10]) against the parameter-free curve (D)Consistent on one run
Token entropy and family diversity can rank configurations differently (4.1–4.2)14B GRPO: token entropy above its initial level while K^\hat K is below half its initial value (steps ≈87–264). Between arms after step 150: token-entropy ratio 0.23–1.90, family-count ratio ≥3.2 (4.4–5.9 up to step 400)Observed
Reversing the field lowers collision (Sec. 6)Matched-step difference in sˉ2\bar s_2 of −0.16 to −0.69 in all five anti-collision runs (three on 7B, two on 14B), one run per armConsistent
Incorrect-block collision falls (Thm 3.1a)Fixed-prompt panel, unbiased estimator: 0.31 → 0.19 (D). The logged aggregate, which App. A.1 sets aside, roseConsistent on one run
Terminal exponent M/(M+1)M/(M+1)Terminal regime not reached. Checked on exact trajectories (Fig. 15)Theory only
AuthorsOpenReview ↗

General response (3/4)

General response (3/4): shared parameters and the late recovery

C. Shared parameters. The independent-logit theorem does not automatically extend to a network with shared parameters. With shared parameters in the linearized regime (θ˙=ηKJ(π)A\dot\theta=\eta KJ(\pi)A, K⪰0K\succeq0 a mode-level tangent kernel):

ddtlog⁡πiπj=η∑ℓ(Kiℓ−Kjℓ) πℓAℓ.\frac{d}{dt}\log\frac{\pi_i}{\pi_j}=\eta\sum_\ell (K_{i\ell}-K_{j\ell})\,\pi_\ell A_\ell .

  • K=IK=I is the independent-logit case.
  • A uniform shared component cI+b11⊤cI+b\mathbf 1\mathbf 1^\top changes only the time scale.
  • Exchangeable correct–correct similarity with uniform correct–incorrect coupling preserves the competition.
  • Heterogeneous similarity changes it, in the direction the formula gives.

Real networks change their kernel during training, so this is a condition, not evidence. The evidence comes from the fixed-prompt panel (D): family log-shares move with family share, with slope 6.1 (95% CI [4.9, 7.3]).

One fact from the logs matters here. On Qwen3-14B the concentration is complete by step ≈50, inside the first pass over the 17.4k training prompts (67 steps at batch size 256). So the family statistics in that window come from prompts the policy had not yet trained on. The per-prompt model describes one prompt in isolation. In the network, the per-prompt family distributions concentrate together, through shared parameters. The logs therefore show concentration at the population level. They do not show the per-prompt dynamics, which is what the panel measures. We will state this in the paper.

Late recovery. Two different things:

  • Token-entropy recovery is not evidence of regained diversity. By (4.1)–(4.2), token entropy can rise through within-family variation and incorrect sequences. In the 14B GRPO run it rises from ≈0.15 to ≈1.1 nats between steps 60 and 100 while K^\hat K stays at ≈1.3.
  • Family-diversity recovery is confounded with a configuration change. Training accuracy levels off near 0.6 from step ≈60 (mean 0.61 over steps 60–180). K^\hat K stays flat at ≈1.33 through step 180 at batch size 256 (slope +0.015 per 100 steps). This persistence is consistent with the monotone model. It is not a test, since accuracy is also flat in that window. K^\hat K rises only after the run was resumed from the step-180 checkpoint with batch size 128: +0.24 per 100 steps over steps 181–300, with most of the rise after step 250. No batch-256 segment runs past step 194. We cannot separate the effect of the smaller batch from a recovery that would have happened anyway.

We will no longer present the late recovery as a property of GRPO dynamics. The batch-size change is in App. A.1 but was not marked on the figures. We will mark it (step 181 for GRPO, step 281 for anti-collision) and label the recovery as confounded. Claim (a) does not depend on this. From step 281 both arms use the same batch size, and the token-entropy ratio still ranges from 0.23 to 1.90 while the family-count ratio stays between 3.2 and 5.2. Whether recovery occurs at a constant batch size is open, and we will list it as a limitation.

AuthorsOpenReview ↗

General response (4/4)

General response (4/4): results of the fixed-prompt panel (D)

This comment gives the full results of the fixed-prompt panel. We report every test we ran.

Setup. A new GRPO run on Qwen3-14B-Base with the original settings and a constant batch size of 256. Its logged correct-family collision rises from 0.52 to 0.94 by step 50, as in the original run. We chose 150 training prompts at random before the run and sampled 48 rollouts per prompt from the model at steps 0, 20 and 50. By step 50, 111 of these prompts had been in one training batch and 39 in none. Rollouts are clustered jointly across the three checkpoints, so a family keeps its identity over time. Each test splits a prompt's step-0 rollouts in two halves. One half defines the families and their shares. The other is the baseline, compared with an equal-size sample from the later checkpoint. If shares did not move, every statistic below has mean zero. Log-shares use an add-half correction, so families that vanish stay in. Intervals are 95% bootstrap intervals over prompts. p-values are sign-flip tests over prompts.

1. Form vj∝yjv_j\propto y_j (Eq. 3.4). We regress the change in family log-share on the family's initial share, within prompt. From step 0 to 50 the slope is 6.1 (95% CI [4.9, 7.3], sign-flip p ≤ 0.0002) on the 98 prompts with at least two families. On steps 0–20 alone it is 3.4 [2.5, 4.3]. The initial leader gains share in 81% of prompts. For prompts already seen in training the slope is 6.4 [5.0, 7.8]; for prompts not yet seen it is 5.3 [3.2, 7.4]. A positive slope shows that log-shares increase with share. It does not by itself establish exact proportionality.

2. Accuracy–entropy relation in HyH_y (Prop. 7.1). From a prompt's step-0 correct- and incorrect-family shares and its error rate, Prop. 7.1 gives its correct-family distribution at any later error rate, with no fitted parameter. We compare the entropy change it predicts with the observed one. The calibration slope (1 is exact agreement, 0 is no relation) is 0.92 (95% CI [0.74, 1.10]) at step 50 on 91 prompts, and 0.88 [0.61, 1.15] at step 20. The mean entropy change is −0.78 nats observed and −0.71 predicted. The RMSE is 0.21 nats for the curve, 0.34 for a rule that assumes full collapse onto one family, and 0.83 for no change. On these prompts the curve predicts a collision of 0.89 at step 50, and the observed value is 0.92. Within prompts, dlog⁡p/dHyd\log p/dH_y is 0.8 (95% CI [0.5, 1.1]), with entropy computed at a fixed 8 correct samples.

3. Collision at matched sample size (Thm 3.1a).

StepAccuracyCorrect-family collisionFamilies at 8 correct samplesIncorrect-family collision
00.270.413.60.31
200.420.662.40.25
500.600.921.30.19

Collisions use the unbiased pair-count estimator ∑jnj(nj−1)/[n(n−1)]\sum_j n_j(n_j-1)/[n(n-1)]. The correct block concentrates and the incorrect block spreads, as Thm 3.1a states. The paired change in incorrect-family collision from step 0 to 50 is −0.12 (95% CI [−0.17, −0.07]).

4. Robustness.

  • Families defined from step-0 rollouts only: slope 5.4 [4.0, 6.8].
  • Whole responses embedded (256-word-piece windows, averaged): slope 5.6 [4.3, 6.9]; collision 0.38 → 0.90.
  • Human labels: two annotators (graduate students not involved in the paper) labeled the main approach of 480 responses, blind to the checkpoint. They agree at κ\kappa = 0.72 and with the clusters at ARI 0.61. Under their labels the collision rises from 0.48 to 0.89.

Limits. This is one run on one model(Qwen3-14B-Base), over the concentration window only. Most panel prompts were trained on at most once. So the panel shows how family shares move across the population of prompts, carried by shared parameters.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.