acceptodds
Under review as a conference paper at ICLR 2027

Proper Scoring under Group Reward Normalization: Target Distortion and Exact Finite-Group Updates

Abstract

GRPO trains language models from several answers to one prompt and standardizes their rewards to stabilize learning. For probability answers scored by one shared random outcome, prior work found overconfidence with two proper scores, which favor honest reports, in a large-group approximation, but left open whether this failure is general or persists in finite groups. We identify the key mechanism: the shared outcome determines both the rewards and their normalization scale, which can turn a small win and a large loss into equally strong updates. We precisely characterize the favored probability with group statistics fixed and prove that every such binary reward fails in some setting; even the best available answer can reverse. We derive the exact update when each sampled answer also changes its own denominator. Centering without division preserves the mean gradient direction at fixed group size; an independent scale shared across outcomes preserves each prompt's target. Qwen2.5-3B experiments connect exact updates to neural gradients and show higher scoring loss with standardization than with centering alone under plain SGD and complementary budget controls.

Primary area:
reinforcement learning
Keywords:
group relative policy optimization, reward normalization, proper scoring rules, probabilistic calibration, finite-group analysis, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_367jOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies probability reports scored against a shared stochastic outcome under GRPO reward normalization. It characterizes the distorted population target for strictly proper scores, derives exact finite-group updates and nonconservative examples, and establishes target-preserving alternatives. Controlled Qwen2.5-3B experiments connect these results to neural gradients and scoring regret.

Critical strengths
  1. A useful theoretical advance beyond the known normalization problem. Score-general impossibility, exact candidate-dependent finite-group fields, and the nonconservative-field examples establish substantially more than the previously reported Brier/log overconfidence effect.

  2. An unusually careful theory-to-experiment connection. Exact enumeration, neural-gradient checks, plain-SGD experiments, and complementary update-budget controls make the claimed mechanism credible.

Critical weaknesses
  1. Limited generalizability of the experimental setting. The main neural experiments deliberately select probability ranges and report tokens that expose the predicted distortion. Although Appendix F.7 includes additional ten-report experiments and robustness checks, the evidence remains concentrated on selected tasks. It is unclear how frequently or severely the distortion occurs with more representative probability distributions and realistic reporting tasks.

  2. Restricted scope of the theoretical setting. The main results assume that all probability reports within a group are scored against the same stochastic outcome. This is an important and relevant setting, but the conclusions do not directly extend to standard GRPO training with deterministic verifiable rewards or more general reasoning tasks. The broader practical implications therefore remain uncertain.

  3. The connection between theoretical target distortion and neural training failure is not fully established. The theory predicts changes in preferred probability reports, but the neural experiments sometimes exhibit a more pronounced phenomenon: the model nearly collapses to reporting the same probability (0.25) across different prompts (Appendix F.4, Table 7). It remains unclear how much of the observed performance degradation is directly explained by the predicted target distortion, versus optimization dynamics, shared-parameter interactions, or policy collapse. A more quantitative analysis separating these mechanisms would strengthen the paper.

  4. Limited algorithmic novelty. While the theoretical contributions are substantial, the proposed correction of centering without standard-deviation normalization is closely related to existing methods such as RLOO and Dr. GRPO. The paper provides a stronger theoretical justification for these approaches rather than introducing a fundamentally new optimization algorithm.

Questions for authors
  1. Can the authors further distinguish the theoretically predicted target distortion from the observed policy collapse in neural training? In particular, how much of the final performance degradation can be attributed to changes in the preferred reports versus optimization dynamics or shared-parameter interactions?

  2. What conclusion should a practitioner draw when choosing between centering alone and the independent common scale? Can the existing results identify when prompt reweighting helps or harms the aggregate proper-score objective?

2 comments

Thank you for recognizing the theoretical contribution and the connection between exact updates and neural gradients. We have added experiments and controls to address your questions about generalizability, training failure, and the choice of correction.

W1: “Limited generalizability of the experimental setting.”

We have added naturally sampled Civil data without the original truth-range filter and found target changes on 20.21% of test prompts on average. We also added ChaosNLI with SmolLM2-1.7B and free-form explanation-and-confidence responses, where centering improves test regret in all six paired training seeds.

The Civil study removes the truth-range filter and probability-bin balancing from RL training and evaluation, and uses six paired seeds and a 4,096-prompt test set. To measure “how frequently or severely the distortion occurs,” we compare the raw and population-normalized optimal-report sets at the selected policies. These sets are disjoint for the fraction reported above; the mean absolute effective-truth shift is 0.02039.

For generated reports, the validation-selected test comparisons are:

Report formatCentering regretOutcome-wise regretPaired reduction
Full-vocabulary decimals0.035220.065200.02998
Free-form explanation + confidence0.040460.066640.02618

Decimal-generation final regret also favors centering by 0.03799. The free-form comparison uses six independent training runs per method, paired by seed, with learning rates fixed from separate tuning runs and checkpoints selected on validation data. These tests extend the benefit to generated responses and another task and model family.

For natural Civil performance, both rows below use validation-selected checkpoints on the same unfiltered test set. The middle-probability group contains 962 prompts and favors centering in all six seeds; 2,912 of the 4,096 test prompts have truth below 0.1.

Civil test populationCentering regretOutcome-wise regret
Full test set0.034110.02775
Truth between 0.1 and 0.50.019130.03768

The target-change frequency answers your question about whether distortion occurs on representative data. The generation experiments establish benefits beyond categorical reporting, while the full Civil comparison shows why final performance also needs to be measured on the intended distribution.

W2: “Restricted scope of the theoretical setting.”

The shared-outcome setting covers probability forecasts about external events, and finite-group distortion can also occur with independent outcomes. Bereket and Leskovec's CRISPR study supplies a concrete shared-outcome pipeline: several probability forecasts are scored against the same observed gene-perturbation label.

We added an independent-outcome counterexample with Bernoulli(0.6) outcomes, Brier reports 0.6 and 0.8, group size 2, and stabilizer 0.03. Normalization favors 0.8, while centering favors the truthful report 0.6. Thus, independent outcomes alone do not preserve the target when each sample enters its own normalization statistics.

For the “deterministic verifiable rewards” and “more general reasoning tasks” you mention, the key distinction is whether the scored outcome depends on the generated answer. A deterministic verifier alone does not determine that relationship. In RLCR, correctness depends on the answer; our forecasts concern an exogenous event. The generated-report experiments broaden the response format within that forecasting setting. We will put this pipeline mapping next to the assumptions.

W3 & Q1: “Can the authors further distinguish the theoretically predicted target distortion from the observed policy collapse in neural training? In particular, how much of the final performance degradation can be attributed to changes in the preferred reports versus optimization dynamics or shared-parameter interactions?”

Yes. The added controls show a final-regret benefit from replacing the distorted expected direction, and a separate selected-regret benefit from removing report-group and outcome sampling noise. The shared-parameter analysis explains how individual reporting incentives can coexist with declining discrimination.

Across six paired SGD runs, the measured reductions are:

Controlled changeCheckpointRegret reduction
Replace expected direction, retaining sampling noiseFinal0.03695
Integrate sampling noise under the original directionValidation-selected0.01779
Integrate sampling noise under the replaced directionValidation-selected0.02625

At a common policy, direction replacement matches each prompt's expected logit-update norm and preserves the conditional sampling-noise distribution. Its final-regret reduction has a 95% paired-seed interval of [0.00096, 0.07908]; the validation-selected direction comparison does not establish a difference.

For the collapse, the local gradient decomposition at initialization gives an own-prompt contribution of +2.350 and a cross-prompt contribution of −3.409 to the high-minus-low report gap, even under the proper-score direction. Prompt interactions can therefore oppose discrimination. The direction-replacement runs improve final scoring loss while the report gap remains close to zero. At the original collapsed policies, the exact field also still increases reports on all high-truth prompts, so 0.25 is not their pointwise target.

For your “how much” question, the reductions above quantify the controlled direction and noise effects. They are effects within this intervention protocol, not causal percentages of the original training gap. We will organize the mechanism discussion around these measured loss effects and the local analysis of discrimination.

Q2: “What conclusion should a practitioner draw when choosing between centering alone and the independent common scale? Can the existing results identify when prompt reweighting helps or harms the aggregate proper-score objective?”

Use centering to preserve the aggregate expected proper-score direction. Consider independent common scaling when prompt-scale control is useful and improves validation performance. For a sufficiently small raw parameter step, whether the reweighted update helps or harms the raw objective is determined by its alignment with that objective's aggregate gradient.

At fixed group size, centering applies the same positive factor to every prompt's expected gradient. Common scaling applies different factors, so conflicting prompt gradients can reverse the aggregate direction after reweighting.

The comparisons show why the scale choice should be validated on the task. Common scaling improves regret on the earlier Civil boundary test, while its final regret on ChaosNLI exceeds centering by 0.02043. We recommend separately tuning the methods at matched sampling and update budgets, and will place this guidance next to the two guarantees.

W4: “Limited algorithmic novelty.”

Our contribution is the score-general impossibility result, the exact finite-group update, and the conditions for preserving the reporting target. The impossibility result shows why changing the proper score cannot universally remove the distortion; the finite-group analysis accounts for the dependence between a report and its own normalization denominator.

Centering's relationship to Dr. GRPO and RLOO will remain explicit. Its centered advantages equal (n−1)/n(n-1)/n times the corresponding RLOO advantages. The preservation results also distinguish retaining each prompt's reporting target from preserving the aggregate neural gradient.

Reviewer_JUNQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies GRPO-style reward standardization when several probability reports for one prompt are scored by a strictly proper rule against a single shared outcome. Since this outcome also sets the normalization scale, outcome-dependent division shifts the elicited target away from the true probability.

The authors characterize the shifted target, prove that no strictly proper binary score avoids it for every policy, and derive the exact finite-group update. Centering without division, or a reference scale shared across outcomes, preserves the target.

Experiments with Qwen2.5-3B show higher Brier regret for outcome-wise standardization than for centering alone, under plain SGD and budget-matched AdamW.

Critical strengths

The paper identifies a clear mechanism: outcome-dependent reward normalization can alter incentives even when the underlying score is strictly proper. The 60%/80% Brier example in Figure 1 makes this intuitive. The analysis substantially generalizes Bereket and Leskovec (2025), which established overconfidence for Brier and log scores under a large-group approximation. The impossibility result covers every finite-valued strictly proper binary score, and the paper provides a closed-form characterization of the distortion, genericity results, and regret bounds. The exact finite-group analysis also captures the dependence between each report’s reward and its own normalization scale. Theorem 4.5 gives a necessary and sufficient condition for target preservation, together with an implementable correction that does not require pp.

The experiments include useful controls: an exact-versus-sampled gradient check, plain SGD as a reference, AdamW comparisons matching gradient norms and training budgets, and paired seeds with bootstrap intervals. The experiment using independent common scaling further isolates the role of outcome-dependent division.

Critical weaknesses

My main concern is the practical scope of the shared-outcome setting. The analysis assumes one random outcome shared by the whole group, with an exogenous truth pp, as in forecasting tasks. In calibration-oriented approaches such as RLCR-style training (Damani et al., 2026), each sampled answer’s confidence is instead scored against that answer’s own correctness. Appendix E.1 shows that independent outcomes with a common population scale preserve the target, but the finite-group case, where each sample enters its own normalization statistics, is only briefly discussed. A clearer mapping between the paper’s assumptions and concrete training pipelines would help establish its practical relevance.

The training results also leave the connection between target distortion and observed failure unresolved. In the SGD study, outcome-wise scaling does place more mass on the majority-extreme report at the validation-selected checkpoints (0.120 vs. 0.011 without division), in line with the theory. Yet outcome-wise policies show less outward bias than policies trained without division at both checkpoints, and by step 320 they collapse to a nearly constant report of 0.25 across prompts, including the 130 high-truth test prompts (Table 7). The regret gap therefore appears more closely associated with a loss of prompt discrimination than with the predicted outward shift. Although Appendix F.4 acknowledges this loss of variation, it remains unclear how the identified target distortion contributes to it. The effective truth in Section 3 is pointwise because the scales depend on the policy, and the exact finite-group field is solved only on two-report faces. For M≥3M\ge3 and n≥3n\ge3, its direction can be score-dependent, and long-run behavior is not characterized. Moreover, under frozen statistics, Tν(p)→pT_\nu(p)\to p as σy/τ→0\sigma_y/\tau\to0, leaving open how much distortion persists near converged policies with τ>0\tau>0 beyond the two-report face.

The experiments demonstrate the mechanism in a controlled setting, but provide limited evidence about its magnitude and generality in practical training. At the model’s own policies, the mean effective-truth shift is 0.0166, and the optimal report changes for 16.2% of validation prompts; as noted above, it is unclear how much of the regret gap these shifts explain. The task uses four report tokens and truths filtered to [0.15,0.21]∪[0.79,0.85][0.15,0.21]\cup[0.79,0.85], so that the Bayes report differs from the majority extreme. The study also uses a single model, rank-4 LoRA, one dataset, and short runs. Beyond the ten-token study in Appendix F.7, evidence on finer report grids or free-form verbalized confidence is limited. Ratio clipping, KL regularization, and multiple inner epochs are analyzed in Appendices E.3–E.4, but stronger empirical evidence with these standard GRPO components would help establish practical relevance.

The practical case for the reference-scale correction remains unclear. For centering without division, the main contribution is the target-preservation guarantee: the update coincides with Dr. GRPO and equals RLOO up to a factor of (n−1)/n(n-1)/n, and the gradient proportionality already appears in Bereket and Leskovec, as the paper acknowledges. The reference-scale correction is new, and its two-fold cross-fitting variant needs no extra samples (Appendix F.7), but it does not consistently outperform plain centering (Appendices F.5.2 and F.7). The paper should clarify when this additional machinery offers a practical advantage.

A short takeaway for practitioners, including the practical consequence of Proposition 4.2, would also improve accessibility.

Overall, the paper offers a clear mechanism and a substantial theoretical generalization of prior work. Its practical significance would be more convincing with a clearer account of which training pipelines satisfy its assumptions and how the identified distortion contributes to the failure observed in training. I currently lean toward rejection, but would raise my score if both points are addressed.

Questions for authors
  1. Which LLM training pipelines used in practice fall into the shared-outcome regime? In finite groups where each sample has its own outcome but still enters its own normalization statistics, is the target preserved?

  2. How does the policy-dependent target distortion relate to the near-constant 0.25 collapse and lower outward bias in Table 7? Can its contribution to the regret gap be distinguished from other optimization effects?

  3. How large is the distortion near converged policies with τ>0\tau>0, and is the restricted Bayes report a stable fixed point of the exact field beyond the two-report face?

  4. Do the conclusions hold with standard GRPO components (ratio clipping with several inner epochs, KL regularization) and with free-form verbalized confidence?

  5. When should practitioners prefer the reference-scale correction over plain centering?

3 comments

Thank you for your careful review and constructive suggestions. In the responses below, we explain more clearly which training pipelines satisfy the shared-outcome assumptions, how the identified distortion contributes to the failure observed in training, and the other issues you raised. We hope that these explanations and the additional evidence address your concerns.

Q1: “Which LLM training pipelines used in practice fall into the shared-outcome regime? In finite groups where each sample has its own outcome but still enters its own normalization statistics, is the target preserved?”

Forecasting pipelines that score several reports about one external event against the same observed label satisfy the shared-outcome assumption. The probability-reporting component of Bereket and Leskovec's CRISPR study is a concrete example: multiple forecasts for a gene perturbation are scored against its observed label, which does not depend on the report. Our Civil and ChaosNLI experiments use this structure, drawing one Bernoulli outcome from each prompt's annotation fraction and scoring all reports against it.

For the independent-outcome finite-group question, the target need not be preserved. We added a counterexample with Brier reports {0.6,0.8}\{0.6,0.8\}, independent Bernoulli(0.6) outcomes, group size two, and stabilizer 0.03. If rr is the probability of reporting 0.8, the expected upper-report logit update under standardization is

94969r(1−r)>0,\frac{94}{969}r(1-r)>0,

which favors 0.8 even though the truthful report 0.6 is available. Centering gives −r(1−r)/50-r(1-r)/50, favoring 0.6. The sample's participation in its own normalization statistics can therefore change the target even when outcomes are independent. We will include the complete enumeration proof alongside the population-scale result.

The distinction from RLCR is the one you highlighted. RLCR scores each answer's confidence against that answer's own correctness, so the outcome depends on the generated action. The forecasting assumption and the independent-outcome counterexample concern exogenous outcomes; the pipeline mapping will make this distinction explicit.

Q2: “How does the policy-dependent target distortion relate to the near-constant 0.25 collapse and lower outward bias in Table 7? Can its contribution to the regret gap be distinguished from other optimization effects?”

Yes. In the added controlled SGD study, replacing the distorted expected direction lowers final regret. The collapse analysis also shows how shared parameters can reduce prompt discrimination despite correct individual update directions.

The direction intervention lowers final regret by 0.03695, with a 95% paired-seed interval of [0.00096, 0.07908]. At a common policy, it matches each prompt's expected logit-update norm and preserves the conditional sampling-noise distribution. The validation-selected comparison does not establish a difference.

For the “near-constant 0.25 collapse,” the exact finite-group update at the original final policies increases mean reports on all 130 high-truth prompts in every one of the six seeds. Their pointwise updates still favor higher reports. We then examined how these directions combine in the shared network. At initialization, under the proper-score direction, the own-prompt contribution to the high-minus-low report gap is +2.350, while the cross-prompt contribution is −3.409. This local calculation shows how prompt interactions can oppose discrimination. In the direction-replacement runs, the final scoring loss improves while the high-minus-low report gap remains 0.00064.

The “lower outward bias” follows from the same reporting pattern: low truths are overpredicted and high truths are underpredicted as reports approach a common value. The quantitative contribution we can isolate is the direction intervention's effect on loss in this controlled study; it is not a percentage decomposition of the original training gap. We will use the intervention and discrimination measurements together to explain Table 7.

Q3: “How large is the distortion near converged policies with τ > 0, and is the restricted Bayes report a stable fixed point of the exact field beyond the two-report face?”

Near a vertex, the absolute update vanishes with the rare-report probability, but the restricted Bayes report need not be locally stable even with a positive stabilizer. We derived the local expansion for any finite report vocabulary. Near a vertex concentrated on report ii, a rare report jj has update

gj=Cjiπj+O(πjϵ),ϵ=∑j≠iπj.g_j=C_{ji}\pi_j+O(\pi_j\epsilon), \qquad \epsilon=\sum_{j\ne i}\pi_j.

The sign of CjiC_{ji} determines whether that report can invade. A group with one rare report becomes less likely near the vertex, but its conditional standard deviation remains determined by the score difference between the reports. A positive stabilizer therefore does not automatically remove the directional distortion.

For Brier reports (0.2,0.6,0.8)(0.2,0.6,0.8), truth p=0.6p=0.6, group size 16, and τ=0.03\tau=0.03, the invasion coefficient for 0.8 is approximately 0.06928. The truthful vertex is locally unstable. Strictly negative invasion coefficients for all competing reports give local stability under the categorical expected dynamics. We will include the full expansion and proof in the appendix.

For the practical consequence of Proposition 4.2, refreshed updates in its setting are not Euclidean logit gradients of one fixed scalar objective. Convergence must therefore be analyzed through the update dynamics; nonconservativity alone establishes neither convergence nor divergence.

Q4: “Do the conclusions hold with standard GRPO components (ratio clipping with several inner epochs, KL regularization) and with free-form verbalized confidence?”

Yes. Centering's advantage holds in both added generation studies with four inner epochs, ratio clipping at 0.2, and KL regularization at 0.02. For both full-vocabulary decimals and free-form explanations followed by confidence, all six paired training seeds favor centering at validation-selected checkpoints.

Report formatCentering test regretOutcome-wise test regretPaired reduction
Full-vocabulary decimals on ChaosNLI0.035220.065200.02998
Free-form explanation + confidence0.040460.066640.02618

The free-form study uses six independent training runs per method, paired by seed. Learning rates are fixed using separate tuning runs; each trained model's checkpoint is selected on validation data and evaluated on held-out test data. Invalid outputs remain included in regret.

The final-checkpoint reduction is 0.03799 for decimal generation. For free-form responses, final regret is 0.04304 under centering and 0.10940 under outcome-wise normalization, with a paired reduction of 0.06637. The benefit thus appears at both the validation-selected and final checkpoints.

Q5: “When should practitioners prefer the reference-scale correction over plain centering?”

Prefer reference scaling when prompt-scale differences are a practical problem and its reweighting improves validation performance. Use centering when preserving the aggregate expected proper-score direction is the priority.

Reference scaling retains each prompt's reporting target without requiring the unknown truth pp, and cross-fitting can reuse existing rollouts. Its extra effect is to change prompt weights. That can be useful for scale control, but the benefit depends on how prompt gradients combine. In our comparisons, common scaling improves regret on the earlier ten-report Civil boundary test, while centering has lower final regret on ChaosNLI. We recommend comparing separately tuned methods on the intended distribution at matched sampling and update budgets.

Centering's relationship to Dr. GRPO and RLOO will remain explicit. The score-general impossibility result and exact finite-group analysis explain why the normalization choice matters, while the preservation conditions identify which choices retain the reporting target.

Thank you for the response.

The new theoretical results strengthen the paper, but the distortion seems to explain only part of the observed failure, as the collapse persists without it. The responses to Reviewer Pcy5 also show lower overall regret for outcome-wise scaling than for centering on the unfiltered Civil data.

I will maintain my score.

Reviewer_Pcy5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how GRPO reward normalization changes the incentive of proper scoring rules: dividing its rewards by an outcome-dependent standard deviation may change the relative influence of outcomes and even reverse the preferred report. The authors characterize this distortion with frozen population statistics and derive exact finite-group expected updates. They study 2 corrections: entering without division and independent scaling shared across outcomes. They experiment on Qwen2.5-3B and connect the theoretical updates to neural gradients, demonstrating higher scoring regret under standardization.

Critical strengths
  1. S1. The impossibility result is not scoring rule specific and clarifies why changing the proper score alone cannot universally prevent normalization-induced target distortion. The exact finite-group analysis shows that distortion persists under finite sampling.
  2. S2. The experiments connect exact updates to sampled LoRA gradients and compare 2 optimization recipes: plain SGD and gradient-normalized AdamW, while holding sampling and update budgets consistent. This is helpful for showing that the performance difference cannot be accounted for by step-size or training-budget artifacts.
Critical weaknesses
  1. W1. Insufficient experimental evidence / limited scope. All “neural experiments” are conducted on Qwen2.5-3B-Instruct with rank-four LoRA on a single dataset: Civil Comments. Notably, this is a simple task of categorical selection among single-token probability reports, and not representative of generative language or reasoning tasks. Experimental diversity in task, dataset, model family, model size, ablations are severely lacking. The learning experiments also restrict the report vocab to four values and deliberately select truth interval settings where the best probability report differs from that favoured by the majority voting. In all, the contrived nature and limited scope of experimental setups raise serious concerns about the applicability of the proposal on real data and tasks across diverse models.

  2. W2. Observed training failure does not support overconfidence interpretation. The authors report that all six finalized SGD policies become near constant at 0.25 across prompts. This demonstrates that the model collapses to similar predictions across prompts and fails to disambiguate them, and underestimates high-probability events. Higher regret implies worse predictions but nothing establishes overconfidence as the underlying failure model. Model collapse does not automatically imply overconfidence and the discussion should be more discerning.

  3. W3. Poor presentation. The motivation is legible but the paper is filled with excessive notation and lacking in coherent organization and presentation of results:

  • Section 3 introduces the restricted bases set, normalized payoffs, finite-vocabulary maximizer, effective truth transformation without properly motivating why these terms are defined; how they relate to and differ from each other.
  • Section 4 uses unusual terminology like “field” to refer to the expected update vector at the current policy. An intuitive explanation before the formal statement would better contextualize Eq. 12.
  • The discussion moves from exact updates to the non-existence of a scalar potential, to finite group thresholds and root structure, target-preserving corrections and neural gradients. These results all have different implications but they are not adequately discussed and merely itemized without explanation of how each finding contributes to the central argument and why each is necessary.
  • Eq. 20 is the chain-rule from expected logic updates to parameter gradients. It perhaps does not warrant being highlighted as an “exact bridge” and standalone equation.
  • Section 5’s interpretation of the list of results is fragmented. The paper presents gradient cosines, effective-truth shifts, update angles, regret decompositions, collapse and budget controls without synthesising these separate observations into coherent claims.
  1. W4. Independent scaling does not preserve the combined neural gradient. Prompt-dependent scales do preserve each prompt’s expected update direction but could reweigh or even reverse the aggregate gradient through shared parameters. This is an important practical limitation and should be discussed prominently and not deferred to Appendix D.2.1.

  2. W5. Proposal still exhibits critical convergence and stability flaws. Preserving the expected proper-score gradient direction does not imply convergence to truthful reports under finite steps, adaptive optimization or neural parametrization. Centering without division makes the objective sensitive again to reward magnitudes, and make results fragile wrt learning rates and gradient variance. Independent scaling introduces new tradeoffs: nearly homogeneous reference groups can produce small denominators (large or noisy updates), which would increase sensitivity to reference-group size and stabilization. The stability tradeoffs and potential convergence issues of this method are two limitations yet insufficiently discussed.

Questions for authors
  • Q1. Could the authors evaluate the better calibration and absence of overconfidence claims more directly? Consider supplementing proper-score regret with reliability diagrams/ECE and signed reporting separated between low and high probability prompts. This would help distinguish between different failure modes—overconfidence, under confidence and collapse (towards a constant prediction); please additionally clarify if calibration is currently assessed for sampled reports or the policy’s mean report.
3 comments

Thank you for your detailed review. We have added experiments and analyses to address your concerns about experimental scope, calibration, and training stability.

W1: “Insufficient experimental evidence / limited scope.”

To address the lack of “task, dataset, model family, model size, ablations,” we have added naturally sampled Civil data and ChaosNLI with SmolLM2-1.7B. Centering improves test regret for both full-vocabulary decimal generation and free-form explanations followed by confidence, with all six paired training seeds favoring centering in each comparison. The benefit therefore extends beyond categorical selection among four single-token reports.

The added studies span 3B and 1.7B models, rank-4 and rank-8 LoRA, and ten-report selection as well as generated responses. The generation experiments include ratio clipping, repeated inner updates, and KL regularization. At validation-selected checkpoints, mean test regret across the six runs is:

Report formatCentering regretOutcome-wise regretPaired reduction
Full-vocabulary decimals0.035220.065200.02998
Free-form explanation + confidence0.040460.066640.02618

The free-form comparison uses six independent training runs per method, paired by seed, with learning rates fixed using separate tuning runs. Checkpoints are selected on validation data and evaluated on held-out test data; invalid outputs remain included in scoring. Decimal-generation regret also favors centering at the final checkpoint, by 0.03799. For ablations, we have separately varied update direction, sampling noise, reference-group size, and stabilization; the learning-rate and reference-scale findings are discussed under W5.

For the “truth interval settings,” the added Civil RL training and evaluation use natural sampling, without the original truth-range filter or probability-bin balancing. The study includes 2,048 training prompts, 1,024 validation prompts, 4,096 test prompts, and six paired training seeds. Population normalization changes the optimal available report on 20.21% of test prompts on average at the selected outcome-wise policies; the mean absolute effective-truth shift is 0.02039. The target change therefore also occurs outside the original probability intervals.

For training performance on natural Civil, the overall and intermediate-probability comparisons are below. Of the test prompts, 2,912 have truth below 0.1; the intermediate group contains 962 prompts, with centering better in every seed. Both rows use validation-selected checkpoints from the same unfiltered test set.

Civil test populationCentering regretOutcome-wise regret
Full test set0.034110.02775
Truth between 0.1 and 0.50.019130.03768

The target diagnostic measures which report the update favors at a given policy; the table measures the trained network's performance. On the question of whether the benefit requires the original four-token, selected-interval setting, the generated-report comparisons above provide direct evidence that it does not.

W2: “Observed training failure does not support overconfidence interpretation.”

The “near constant at 0.25 across prompts” behavior is a loss of prompt discrimination. It overpredicts low-probability events and underpredicts high-probability events, as the original six final outcome-wise policies show:

Prompt groupMean truthMean report
Low probability0.181150.25000
High probability0.817820.25005

We will describe the observed failure as loss of prompt discrimination. The target-distortion result concerns which report the update favors; here, the mean reports show the additional loss of distinction between low- and high-probability prompts.

Q1: “Could the authors evaluate the better calibration and absence of overconfidence claims more directly?”

Yes. We added separate calibration analyses for the policy's mean report and sampled reports. Both show lower binned calibration error under centering in the original four-report experiment; the signed errors above identify the combination of low-probability overprediction and high-probability underprediction.

Using ten bins and annotation fractions as the calibration targets, the differences at validation-selected checkpoints are:

Quantity evaluatedOutcome-wise minus centering calibration error
Policy mean report0.07483
Sampled report0.06478

The mean-report analysis evaluates one policy mean per prompt. The sampled-report analysis weights individual reports by their policy probabilities. In the camera-ready version, we will include reliability diagrams for both quantities at the selected and final checkpoints, with bin counts and signed errors.

W3: “Poor presentation.”

We will reorganize Sections 3–5 around the question motivating the paper: how reward normalization changes probability-reporting incentives, how that change enters finite-group updates, and what it does during neural training. We will introduce the notation through these questions and explain each result's role before giving its formal statement.

For the notation in Section 3, the restricted Bayes set identifies the best report available in the vocabulary. The normalized payoff asks whether scaling changes that choice. Effective truth expresses the changed outcome weights as another probability. We will introduce these definitions in that order, so the reader can follow the comparison between the original and normalized reporting targets.

For “field” in Section 4, we will first say that it is the expected update vector at the current policy, before Eq. 12. The exact update accounts for each sample's contribution to its own denominator. The thresholds identify when its direction changes; the potential analysis asks whether refreshed updates are Euclidean logit gradients of a fixed scalar objective; the preservation results identify modifications that retain the proper-score target. For Eq. 20, we will make clear that the chain rule is standard and that the substantive input is the exact expected logit update mapped to neural parameters.

For the “fragmented” interpretation in Section 5, we will group gradient cosines and update angles as checks of the predicted update, effective-truth shifts as measurements of target change, and regret and discrimination as training outcomes. Step-size and budget controls will accompany the outcomes whose optimization explanation they test. We will use this organization in the camera-ready version so that each observation has a stated purpose in the argument.

W4: “Independent scaling does not preserve the combined neural gradient.”

Centering preserves the combined expected proper-score gradient direction. Independent common scaling preserves each prompt's target and expected direction, while changing the weights of prompts. For fresh on-policy updates at fixed group size nn, centering applies the same factor (n−1)/n(n-1)/n to every prompt's expected gradient, which is why its aggregate guarantee holds.

The reweighting issue you raise applies to reference scaling. Equally weighted gradients 0.09 and −0.12 average to −0.015; scaling them by 2 and 1 changes the average to +0.03. Reference scaling therefore serves a different purpose from centering: it controls prompt scales while retaining individual reporting targets. We recommend centering when preserving the aggregate direction is the priority, and will put this distinction and the example next to the preservation result in the main text.

W5: “Proposal still exhibits critical convergence and stability flaws.”

Centering retains its mean-report MSE advantage at all three learning rates tested in the original ten-report study. It also avoids the small reference denominators you mention. Reference scaling requires choosing the reference size and stabilizer together, because both affect the update magnitude and noise.

The learning-rate sweep uses six paired training seeds per rate. On the methods' common observed KL range, the outcome-wise-minus-centering excess MSE is at least 0.02077. Thus, the advantage in this study persists across the tested learning rates and at comparable amounts of policy change. In the broader training comparisons, we also select learning rates separately for the methods.

For reference scaling, we added measurements at fixed neural policies with reference-group sizes 4, 8, 16, and 32 and stabilizers 0.01, 0.03, and 0.1. Larger references and stabilizers reduce absolute gradient variance in this grid. We also measure noise relative to the mean update: that ratio is not monotone, since the mean can shrink as well. Choosing the reference size and stabilizer therefore requires checking both quantities.

For the “convergence” concern, the theorem guarantees the expected proper-score direction under its sampling assumptions. The finite-step controls show why step size also matters for the realized neural response, and the learning-rate sweep tests performance across the chosen rates. We will present these controls alongside the theorem, so its target-preservation guarantee and the evidence for neural training are clear.

I appreciate the authors' added generation experiments and clarifications about overconfidence, but centering yielded higher regret on the full, naturally sampled Civil Comments test set (0.03411 vs. 0.02775 under outcome-wise normalization). These mixed results leave me uncertain about how broadly benefits of centering generalise; the expected-gradient guarantee still does not settle whether neural training converges to truthful reports. I maintain my score of 1 (clear rejection).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.