- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies probability reports scored against a shared stochastic outcome under GRPO reward normalization. It characterizes the distorted population target for strictly proper scores, derives exact finite-group updates and nonconservative examples, and establishes target-preserving alternatives. Controlled Qwen2.5-3B experiments connect these results to neural gradients and scoring regret.
Critical strengths
-
A useful theoretical advance beyond the known normalization problem. Score-general impossibility, exact candidate-dependent finite-group fields, and the nonconservative-field examples establish substantially more than the previously reported Brier/log overconfidence effect.
-
An unusually careful theory-to-experiment connection. Exact enumeration, neural-gradient checks, plain-SGD experiments, and complementary update-budget controls make the claimed mechanism credible.
Critical weaknesses
-
Limited generalizability of the experimental setting. The main neural experiments deliberately select probability ranges and report tokens that expose the predicted distortion. Although Appendix F.7 includes additional ten-report experiments and robustness checks, the evidence remains concentrated on selected tasks. It is unclear how frequently or severely the distortion occurs with more representative probability distributions and realistic reporting tasks.
-
Restricted scope of the theoretical setting. The main results assume that all probability reports within a group are scored against the same stochastic outcome. This is an important and relevant setting, but the conclusions do not directly extend to standard GRPO training with deterministic verifiable rewards or more general reasoning tasks. The broader practical implications therefore remain uncertain.
-
The connection between theoretical target distortion and neural training failure is not fully established. The theory predicts changes in preferred probability reports, but the neural experiments sometimes exhibit a more pronounced phenomenon: the model nearly collapses to reporting the same probability (0.25) across different prompts (Appendix F.4, Table 7). It remains unclear how much of the observed performance degradation is directly explained by the predicted target distortion, versus optimization dynamics, shared-parameter interactions, or policy collapse. A more quantitative analysis separating these mechanisms would strengthen the paper.
-
Limited algorithmic novelty. While the theoretical contributions are substantial, the proposed correction of centering without standard-deviation normalization is closely related to existing methods such as RLOO and Dr. GRPO. The paper provides a stronger theoretical justification for these approaches rather than introducing a fundamentally new optimization algorithm.
Questions for authors
-
Can the authors further distinguish the theoretically predicted target distortion from the observed policy collapse in neural training? In particular, how much of the final performance degradation can be attributed to changes in the preferred reports versus optimization dynamics or shared-parameter interactions?
-
What conclusion should a practitioner draw when choosing between centering alone and the independent common scale? Can the existing results identify when prompt reweighting helps or harms the aggregate proper-score objective?
2 comments
W3 & Q1: “Can the authors further distinguish the theoretically predicted target distortion from the observed policy collapse in neural training? In particular, how much of the final performance degradation can be attributed to changes in the preferred reports versus optimization dynamics or shared-parameter interactions?”
Yes. The added controls show a final-regret benefit from replacing the distorted expected direction, and a separate selected-regret benefit from removing report-group and outcome sampling noise. The shared-parameter analysis explains how individual reporting incentives can coexist with declining discrimination.
Across six paired SGD runs, the measured reductions are:
| Controlled change | Checkpoint | Regret reduction |
|---|---|---|
| Replace expected direction, retaining sampling noise | Final | 0.03695 |
| Integrate sampling noise under the original direction | Validation-selected | 0.01779 |
| Integrate sampling noise under the replaced direction | Validation-selected | 0.02625 |
At a common policy, direction replacement matches each prompt's expected logit-update norm and preserves the conditional sampling-noise distribution. Its final-regret reduction has a 95% paired-seed interval of [0.00096, 0.07908]; the validation-selected direction comparison does not establish a difference.
For the collapse, the local gradient decomposition at initialization gives an own-prompt contribution of +2.350 and a cross-prompt contribution of −3.409 to the high-minus-low report gap, even under the proper-score direction. Prompt interactions can therefore oppose discrimination. The direction-replacement runs improve final scoring loss while the report gap remains close to zero. At the original collapsed policies, the exact field also still increases reports on all high-truth prompts, so 0.25 is not their pointwise target.
For your “how much” question, the reductions above quantify the controlled direction and noise effects. They are effects within this intervention protocol, not causal percentages of the original training gap. We will organize the mechanism discussion around these measured loss effects and the local analysis of discrimination.
Q2: “What conclusion should a practitioner draw when choosing between centering alone and the independent common scale? Can the existing results identify when prompt reweighting helps or harms the aggregate proper-score objective?”
Use centering to preserve the aggregate expected proper-score direction. Consider independent common scaling when prompt-scale control is useful and improves validation performance. For a sufficiently small raw parameter step, whether the reweighted update helps or harms the raw objective is determined by its alignment with that objective's aggregate gradient.
At fixed group size, centering applies the same positive factor to every prompt's expected gradient. Common scaling applies different factors, so conflicting prompt gradients can reverse the aggregate direction after reweighting.
The comparisons show why the scale choice should be validated on the task. Common scaling improves regret on the earlier Civil boundary test, while its final regret on ChaosNLI exceeds centering by 0.02043. We recommend separately tuning the methods at matched sampling and update budgets, and will place this guidance next to the two guarantees.
W4: “Limited algorithmic novelty.”
Our contribution is the score-general impossibility result, the exact finite-group update, and the conditions for preserving the reporting target. The impossibility result shows why changing the proper score cannot universally remove the distortion; the finite-group analysis accounts for the dependence between a report and its own normalization denominator.
Centering's relationship to Dr. GRPO and RLOO will remain explicit. Its centered advantages equal times the corresponding RLOO advantages. The preservation results also distinguish retaining each prompt's reporting target from preserving the aggregate neural gradient.
Thank you for recognizing the theoretical contribution and the connection between exact updates and neural gradients. We have added experiments and controls to address your questions about generalizability, training failure, and the choice of correction.
W1: “Limited generalizability of the experimental setting.”
We have added naturally sampled Civil data without the original truth-range filter and found target changes on 20.21% of test prompts on average. We also added ChaosNLI with SmolLM2-1.7B and free-form explanation-and-confidence responses, where centering improves test regret in all six paired training seeds.
The Civil study removes the truth-range filter and probability-bin balancing from RL training and evaluation, and uses six paired seeds and a 4,096-prompt test set. To measure “how frequently or severely the distortion occurs,” we compare the raw and population-normalized optimal-report sets at the selected policies. These sets are disjoint for the fraction reported above; the mean absolute effective-truth shift is 0.02039.
For generated reports, the validation-selected test comparisons are:
Decimal-generation final regret also favors centering by 0.03799. The free-form comparison uses six independent training runs per method, paired by seed, with learning rates fixed from separate tuning runs and checkpoints selected on validation data. These tests extend the benefit to generated responses and another task and model family.
For natural Civil performance, both rows below use validation-selected checkpoints on the same unfiltered test set. The middle-probability group contains 962 prompts and favors centering in all six seeds; 2,912 of the 4,096 test prompts have truth below 0.1.
The target-change frequency answers your question about whether distortion occurs on representative data. The generation experiments establish benefits beyond categorical reporting, while the full Civil comparison shows why final performance also needs to be measured on the intended distribution.
W2: “Restricted scope of the theoretical setting.”
The shared-outcome setting covers probability forecasts about external events, and finite-group distortion can also occur with independent outcomes. Bereket and Leskovec's CRISPR study supplies a concrete shared-outcome pipeline: several probability forecasts are scored against the same observed gene-perturbation label.
We added an independent-outcome counterexample with Bernoulli(0.6) outcomes, Brier reports 0.6 and 0.8, group size 2, and stabilizer 0.03. Normalization favors 0.8, while centering favors the truthful report 0.6. Thus, independent outcomes alone do not preserve the target when each sample enters its own normalization statistics.
For the “deterministic verifiable rewards” and “more general reasoning tasks” you mention, the key distinction is whether the scored outcome depends on the generated answer. A deterministic verifier alone does not determine that relationship. In RLCR, correctness depends on the answer; our forecasts concern an exogenous event. The generated-report experiments broaden the response format within that forecasting setting. We will put this pipeline mapping next to the assumptions.