acceptodds
Under review as a conference paper at ICLR 2027

Correct Reward, Costly Rule: Gradient Coherence Decides What Group-Relative RL Locks In

Abstract

When a verifiable reward is exact but the policy's observation is insufficient, the reward-maximising policy must follow a feature that is right only on average, and its cost falls entirely on the samples that feature misleads. We study how group-relative RL reaches this outcome, using next-POI ranking as a controlled testbed. First, GRPO converges to the constrained-optimal rule: it locks onto the most readable feature, lifts aggregate accuracy, pushes the misled subgroup below chance, and keeps applying the rule where the feature is uninformative. Second, the cost is asymmetric because aligned gradients cohere while misaligned ones cancel, not because either side is larger. Third, an exact advantage–position identity shows that this coherence is created by centring the advantage, not inherited from the model's representations. Fourth, the resulting criterion predicts which feature is locked in before training, even against the base model's preference. Finally, re-anchoring the advantage moves the cost exactly as the diagnosis implies, and bounds what any reweighting of sampled rollouts can recover.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards (RLVR), gradient coherence, GRPO / group-relative policy optimization, shortcut learning, training dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_zDmXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates a potential failure mode of Group Relative Policy Optimization (GRPO), where optimizing an accurate and verifiable reward can improve aggregate performance while systematically degrading performance on certain subgroups. Using next-point-of-interest (POI) recommendation as a controlled testbed, the authors demonstrate that GRPO tends to converge toward a dominant feature-based ranking rule, benefiting samples aligned with this rule while penalizing misaligned samples. The paper attributes this asymmetric behavior to gradient coherence rather than gradient magnitude, showing that gradients from aligned samples reinforce each other while those from misaligned samples tend to cancel out. To explain this phenomenon, the authors derive an advantage–position identity and investigate how group-relative advantage normalization influences gradient geometry. They further propose a gradient-coherence-based criterion for predicting feature lock-in and introduce Heuristic-Relative Advantage (HRA) as an intervention to redistribute subgroup performance. Experiments across different feature constructions, cities, and model backbones provide empirical support for the proposed analysis.

Critical strengths
  1. The paper investigates an important and relatively underexplored failure mode of group-relative reinforcement learning. Unlike conventional reward hacking, which often involves misspecified rewards, this work highlights that subgroup performance degradation can occur even when the reward is accurate. The distinction between aggregate reward improvement and subgroup-level deterioration is meaningful, particularly for recommendation systems and other applications with heterogeneous user preferences.
  2. The paper provides a systematic analysis connecting gradient coherence, advantage normalization, and feature lock-in. Rather than merely reporting performance degradation, the authors investigate gradient directions, aggregation efficiency, and the relationship between target rank and gradient similarity. The advantage–position identity and the empirical gradient measurements offer a useful perspective for understanding how subgroup contributions interact during GRPO optimization.
Critical weaknesses
  1. The generalizability of the proposed findings is limited by the highly controlled experimental setting. The empirical investigation is primarily conducted on next-POI recommendation with eight candidates, a position-based reward, and a small set of explicitly constructed features, such as historical visit count and category. This setting provides a convenient environment for analyzing ranking behavior, but it differs substantially from more general RLVR applications, particularly mathematical reasoning, code generation, and complex decision-making tasks. The proposed gradient-coherence mechanism depends on the relationship between candidate rank, output position, and gradient direction, which may not naturally extend to tasks without such explicit positional structures. Although the authors include experiments across two cities and two model backbones, these experiments still share essentially the same task formulation. Consequently, the evidence is insufficient to establish gradient coherence as a general explanation for feature lock-in in group-relative reinforcement learning.

  2. The theoretical contribution does not fully establish the claimed relationship between gradient coherence and feature lock-in. Theorem 1 characterizes the optimal ranking strategy under a strictly decreasing target-rank distribution, but this result follows directly from the rearrangement inequality and is independent of GRPO or gradient-based optimization. Therefore, it establishes the optimality of the ranking rule rather than explaining why GRPO specifically learns that rule. Similarly, Proposition 1 provides an exact algebraic decomposition of the group-relative gradient under simplified assumptions, but it does not establish that gradient coherence is a necessary or sufficient condition for feature lock-in. The argument connecting advantage centering, gradient alignment, and the eventual dominant feature still relies heavily on empirical observations. A more rigorous characterization of the conditions under which gradient coherence determines the optimization outcome would substantially strengthen the theoretical contribution.

  3. The causal role of group-relative advantage centering is not sufficiently isolated from other optimization mechanisms. A central claim is that the observed gradient coherence arises from centered advantages rather than the model's learned representations. The authors support this claim by comparing advantage-weighted gradients with unweighted gradients on identical rollouts. However, removing advantage weighting fundamentally changes the optimization signal, and the disappearance of the observed gradient structure does not necessarily establish that group-relative centering is the unique cause of the phenomenon. In particular, action-independent baselines are a standard component of policy-gradient methods, and subgroup gradient interference may also arise under alternative reinforcement learning objectives. Comparisons with REINFORCE, PPO using alternative baselines, or other advantage estimators would help determine whether the reported behavior is specific to GRPO or reflects a more general property of policy-gradient optimization.

  4. The experimental evidence for the predictive capability of gradient coherence remains insufficient. One of the paper's central contributions is a criterion that predicts which feature will dominate training based on gradient coherence measured before optimization. However, the strongest prospective validation is primarily conducted in the Tokyo Construction B setting, where the coherence criterion favors category over count. Moreover, the category rule also has a higher achievable performance ceiling than the count rule, making it difficult to completely disentangle the effects of gradient coherence and objective strength. The current evidence does not establish whether the proposed criterion consistently predicts feature selection across a broader range of competing features, correlation strengths, reward distributions, and model initializations. Furthermore, the training experiments are conducted with a single seed, which limits confidence in the robustness of the reported feature-selection dynamics. Additional controlled experiments in which feature informativeness and gradient coherence vary independently would provide stronger evidence for the predictive claims.

  5. The proposed HRA intervention provides limited practical improvements and does not fully resolve the identified failure mode. Although Heuristic-Relative Advantage is primarily introduced as a diagnostic intervention, its experimental effectiveness remains limited. In Table 4, HRA only partially recovers performance on the misaligned subgroups, while reducing performance on the aligned subgroups and slightly decreasing overall accuracy compared with GRPO. The misaligned subgroups still perform below the base model. Moreover, HRA relies on a predefined heuristic and its expected reward, which may be difficult to obtain in more complex applications where the dominant shortcut or reference strategy is unknown. The method also introduces additional design choices, including advantage clipping and fallback thresholds, whose effects are not thoroughly isolated. Therefore, while the intervention supports the qualitative observation that subgroup performance can be redistributed, it provides limited evidence for a generally applicable solution to the identified problem.

Questions for authors

See Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer zDmX (1/2)

Thank you for the systematic reading, and for recognising that this failure mode differs from reward hacking because the reward is accurate. Your questions about what the theory does and does not show, and about isolating centring, led to the clearest changes in the revision. We also worked on readability: the revision adds a running example, numerical examples in §3.3, §3.4 and §4.1, a conclusion-first §3.4 and a glossary (A.30).

1. Generality and positional structure (W1)

A second task and a second reward both reproduce lock-in, and the identity does not depend on positional structure. Our claims are limited to tasks with a candidate-choice structure.

  • Multiple-choice ranking (MMLU-Pro, SuperGPQA; General Response C): the cue predicted before training is locked in all three seeds, and accuracy on items the cue misleads falls by 9–15 points. A binary reward also reproduces the lock and its cost (A.29).
  • In general form, a prompt's centred gradient is the difference between the gradients of its better and worse rollouts; position rewards are one case. On the multiple-choice task, the aligned/misaligned gradient asymmetry appears after training in all three seeds (A.27).
  • As you suggest, the introduction now states the scope alongside §5, with reasoning tasks listed as untested.

2. Theorem 1 and Proposition 1 (W2)

We agree, and the revised §3.4 states what each part does.

  • Theorem 1 says where reward maximisation ends for policies acting on count order, independently of the algorithm (A.2). Within this class, the misaligned subgroup's low accuracy follows from the available information, not from a misspecified reward; the predictor controls (A.5) find no better policy using exact counts, gaps, ties or other prompt fields.
  • Proposition 1, with the control in General Response B, says where GRPO's gradient structure comes from: centring leaves the covariance between position and gradient direction, and plugging in the measured position distributions reproduces the measured decay, also across rollout seeds (split-half Spearman 0.89–0.94, A.16).
  • That coherence predicts the locked rule is an empirical regularity, not a theorem, tested by predictions written before training (General Response C) and by interventions on the centre (§4).

3. Isolating centring; specificity to GRPO (W3)

With reward weighting kept, removing only the group-mean subtraction removes the rank structure (+0.017 → +0.001; General Response B). Lock-in itself is not specific to GRPO; what is specific is where its gradient structure comes from.

We have also revised the claim you cite: centring does not create the structure but exposes position-dependent differences already present in the model's outputs (abstract, §3.4). Any training that approximately maximises expected reward ends at the same rule (A.2); what is specific to GRPO is the source of the gradient structure and its predictability.

  • REINFORCE without a baseline: no structure. With a constant baseline: structure, about 88% of it from differences in difficulty between prompts. Training with either baseline locks the same rule as GRPO (follow-up below).
  • PPO: GRPO uses PPO's objective with the group mean in place of a value network; with one update per batch the clip is inactive, and the update is REINFORCE with a group-mean baseline. We have not trained PPO with a value network; in this one-step task it would estimate the same quantity as the group mean, but whether it yields the same gradient structure is untested.
  • DAPO (A.20) and IAC (A.21) are already included.

4. Evidence for the predictive power of coherence (W4)

The confound is real in the Tokyo experiment. In the new test, informativeness is equal by construction, and the prediction based on coherence measured before training held in all three seeds, against the base model's preference (General Response C).

Coverage.

  • two tasks (POI recommendation, multiple-choice ranking) and four cues (count, category, option position, option length);
  • two cue strengths (51.5% and 40%; the weaker locks less strongly);
  • seven runs in settings where base coherence was measured (TKY-B 3, multiple-choice 51.5% 3, 40% 1), all as predicted, four of them with exactly equal accuracy;
  • NYC-B reproduces with Qwen (3 seeds) and Llama-3.1-8B; three base models overall (Qwen2.5-14B, Qwen2.5-7B, Llama-3.1-8B).

Not yet covered: varying payoff and coherence over a wider range, including payoff favouring the less coherent rule, and testing the coherence prediction on several base models for the same task.

AuthorsOpenReview ↗

Response to Reviewer zDmX (2/2)

5. HRA's limited gain; clip and fallback (W5)

HRA is a controlled probe of the diagnosis (§4.1, §5). Partial recovery, a slight overall drop and accuracy still below base are what §4.3 implies: recombining the gradients of sampled rollouts can shift the cost between subgroups but not remove it.

HRA anchors on a known rule's expected reward because it is a probe; for unknown shortcuts, the introduction gives three checks that need no closed-form baseline. Ten configurations lie within 0.11 points of one trade-off line whose slope can be computed in advance, and HRA moves along it (General Response D). Replaying HRA on archived GRPO rollouts, the ±10 clip is never reached, and 98.6% of fallback groups have identical rewards (fallback rates in Table 24); these carry no gradient under either method, and the fallback only prevents a non-zero r−rHr - r_H from being divided by ϵ=10−4\epsilon = 10^{-4}.

We hope these answers resolve your concerns and would be grateful if you would consider updating your score.

AuthorsOpenReview ↗

Follow-up on W3: REINFORCE without a baseline and with a batch-mean baseline

Both REINFORCE variants from item 3 lock the rule predicted before training on TKY-B (200 steps; two seeds without a baseline, one with the constant, batch-mean baseline). They pass the same prediction test as GRPO (+0.044 to +0.050; all intervals exclude 0) and lower W=0 HR@1 below the base model by 1.36–1.50 points while raising W=1 HR@1. This answers your question whether the behaviour is specific to GRPO: the lock and its cost do not require the group-mean subtraction. What the group mean adds is a per-prompt gradient structure in which the two rules' difference in coherence is clearly measurable before training (General Response B), and this is what makes the lock predictable. This matches the revised claim in item 3: centring exposes the structure rather than creating it. Details are in General Response, addendum F.

Reviewer_AaetOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The work demonstrates that for tasks (e.g., recommendation ranking) where verifiable rewards are exact but candidate observations cannot fully determine individual outcomes, GRPO converges to a rigid, constrained-optimal shortcut. The policy rapidly locks onto the most readable average heuristic (such as popularity or category matching), increasing aggregate benchmark accuracy while driving accuracy on misaligned tail subgroups far below random chance. The policy is also applied to regions where the shortcut feature carries no target signal.

Through full-parameter gradient audits, the authors show that this divergence is driven by gradient coherence rather than gradient magnitude. An exact advantage-position identity proves that this directional asymmetry stems directly from GRPO’s group-mean centering. As the shortcut sharpens into a near-deterministic policy, rollout rewards homogenize into zero-variance "silent groups" that generate no gradient, cutting off corrective feedback and freezing the rule in place.

To break this dynamic, the authors propose Heuristic-Relative Advantage (HRA), which centers rewards on an explicit external baseline, eliminating the unearned reinforcement of the shortcut and substantially recovering tail-subgroup performance.

Critical strengths
  • The authors mathematically and empirically demonstrate that even under exact rewards, for underspecified inputs, GRPO training can lock the policy towards rigid average-case heuristics. They show that as the shortcut rule sharpens and becomes near-deterministic, rollouts for a prompt become homogenous causing advantage values to zero out, cutting off any corrective gradients and freezing the policy in place. This creates silent-groups on tail prompts, even for exact rewards.

  • Through full-parameter gradient audits, the paper demonstrates that feature selection in GRPO is driven by gradient coherence rather than gradient magnitude. Through experiments the paper shows that gradient coherence during training can lock-in a policy that can even overrule strong base-model behavioral prior.

  • Through careful negative controls the authors eliminate several alternative explanations including impacts of position bias and label variance. Experiments are conducted on Llama and Qwen models, for both GRPO and DAPO.

Critical weaknesses
  • The paper frames its findings as a broad critique of RLVR and GRPO, yet the empirical validation is entirely restricted to an isolated next-POI recommendation ranking task over eight candidates. This testbed features a very specific structure: an exact linear position reward and easily separable scalar attributes (like visit count or category). It remains unproven whether these conclusions generalize to standard RLVR benchmarks (like multi-step mathematical reasoning or code generation), where rewards are typically sparse and binary, and shortcuts are complex, non-linear, and distributed across long reasoning chains.

  • As the authors note in the limitations, all primary training runs use only a single seed. Because the paper's core claims involve training dynamics, gradient variance, and policy freezing, evaluating only one run per configuration leaves open the question of how sensitive these lock-in dynamics, error concentrations, and silent-group proportions are to random initialization and sampling noise.

  • The proposed Heuristic-Relative Advantage (HRA) is useful as a diagnostic probe, however it might be difficult to use as a practical learning algorithm. It requires practitioners to anticipate the model's shortcut beforehand and compute its expected reward in closed form without generating rollouts which is rarely possible in open-ended LLM tasks. Moreover, even HRA only seem to rescue near-miss candidates at rank 2 and slightly hurt overall accuracy.

Questions for authors
  • The Advantage-Position Identity and the resulting covariance decomposition fundamentally assume a linear position-graded reward. However, mainstream RLVR tasks (such as mathematical reasoning, code execution, or standard multiple-choice QA) operate on sparse, binary rewards. How does this mathematical framework translate to purely binary outcome rewards where there is no continuous rank-position structure? Does gradient coherence still drive shortcut lock-in when rewards cannot be linearly decomposed across candidate positions?

  • The authors note that all primary training runs use a single seed. Given that the paper makes specific quantitative claims did the authors run any multi-seed checks on smaller subsets or validation runs? How sensitive are the speed of lock-in and the proportion of silent groups to random rollout and optimization seeds?

  • Equation 4 requires evaluating the heuristic's expected reward rH(q)r_H(q) in closed form without rollout generation. In broader RLVR settings where potential shortcuts cannot be computed analytically, how can HRA be applied? If rH(q)r_H(q) must instead be estimated empirically via sampling, does the sampling variance of the baseline disrupt the group-level cancellation in Equation 5 and reduce training stability?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Aaet

Thank you for recognising the core finding and the negative controls, and for the direct feedback on scope and readability. Your Q1 mentions standard multiple-choice QA; we added a multiple-choice ranking task on MMLU-Pro and SuperGPQA (A.27, with a position-graded reward), binary-reward training on the POI task (A.29) and three seeds (A.26), and improved readability (a running example, numerical examples in §3.3, §3.4 and §4.1, a conclusion-first §3.4, and a glossary, A.30). One clarification to the summary: the gradient analyses use the LoRA parameters that training updates, some on random coordinate subsets validated against the full vectors (A.13–A.14). We have also refined our own wording on the identity (revised abstract and §3.4): centring exposes position-dependent differences already present in the model's outputs, rather than creating them on its own.

1. Binary reward; mathematics and code (W1, Q1)

The identity does not require a linear reward, and lock-in with its misaligned cost reappears under a binary reward and on multiple-choice questions. We agree that the original paper read too broadly: the introduction and §5 now restrict the claims to choice from a bounded candidate set, and multi-step mathematics and code remain untested.

Theory. With 5 rollouts of which 2 are correct, centring gives weights +0.6 and −0.4, so the prompt's gradient is a constant times (mean gradient of correct − mean gradient of incorrect rollouts), and the shared component cancels. The position reward is a special case (general form in A.15).

Training (A.29). With a binary reward (1 for rank 1; we also tested the top 3), both misaligned subgroups still fall (5.0% → 3.5%, 12.1% → 8.7%) and the aligned ones rise. The lock is somewhat weaker (most-visited first: 87.0% vs. 89.5%) and the cost shrinks in proportion, on the same trade-off line (Figure 3).

Gradients under a binary reward. Re-scoring archived rollouts, after lock-in a prompt's 5 rollouts almost always agree, so only 26 of 1,248 groups carry gradient, too few to repeat the test of §3.4; the evidence for the binary case is behavioural.

Multiple-choice (General Response C). On MMLU-Pro and SuperGPQA with planted position and length cues of equal accuracy (still a position-graded reward), training locked the cue predicted before training in all three seeds. Averaged over seeds, ranking D first rises in all 27 source-by-subject strata and accuracy on items whose answer is not at D falls in 26 (computer science and engineering, 347 items: +46 and −11 points), including items answered correctly before training. Post-training gradients show the same asymmetry.

2. Sensitivity to the seed (W2, Q2)

All three are stable across seeds (General Response A).

  • Speed: by step 200, two seeds have completed 79.5% and 99.6% of their 400-step rise in rule following; the third, not trained to step 400, 89.3% of the others' mean rise.
  • Error concentration: among misaligned errors, picks of the most-visited candidate rise from 86.8% at base to 93.7%, 94.7% and 94.2%.
  • Silent fraction: 46.0%, 47.5% and 46.8% at step 200 (base 9.5%); 51.2% and 49.8% at step 400 in the two seeds trained that far.
  • Subgroup accuracy: within 1.5 points across seeds.

3. HRA is impractical; a sampled baseline (W3, Q3)

As you note in W3, HRA is a diagnostic probe rather than a training algorithm (§4.1, §5). Its rank-2 recovery and slight overall drop match a weaker lock: ten configurations lie within 0.11 points of one trade-off line (General Response D).

For practitioners the paper offers three checks (end of the introduction): evaluate by subgroup; treat a rising fraction of identical-reward groups as a possible sign of lock-in (seen in the POI task, not in the multiple-choice task); and measure, before training, which cue the reward will lock. The rank-2 concentration is not specific to HRA: when the model leaves the top candidate it mostly picks the second, so any weaker lock recovers those prompts first.

Sampled rH(q)r_H(q) and Eq. 5 (A.24). The contrast term gg does not contain rH(q)r_H(q), so the cancellation is unaffected. Sampling error enters only the level term: rˉ−rH(q)\bar r - r_H(q), roughly 0 when the model follows the heuristic, gains a random multiple of wˉ\bar w unrelated to rank. Replaying with noise of a kk-sample estimate weakens the rank structure by 11% at k=1k=1 (23% under a pessimistic noise level) and by under 1% at k=32k=32, and never creates structure. Training stability with a sampled baseline has not been tested.

We hope these answers resolve your concerns and would be grateful if you would consider updating your score.

AuthorsOpenReview ↗

Short summary before the discussion closes

Thank you again for your review. Before the discussion closes, here are the direct answers to your three questions:

  • Q1: The general form of the identity needs no linear reward (A.15), and lock-in with its misaligned cost reappears under a binary reward (A.29) and on a multiple-choice ranking task built from MMLU-Pro and SuperGPQA (General Response C); the claims are now scoped to choice from a bounded candidate set (§1, §5).
  • Q2: The speed of lock-in, the silent fraction and the concentration of errors replicate across three seeds (item 2).
  • Q3: Because a sampled baseline takes the same value for all rollouts of a prompt, it preserves the cancellation in Eq. 5 and adds noise only to the level term; in replay, this weakens the rank structure by 11% with one sample and by less than 1% with 32 samples (item 3).

Addendum E of the General Response adds that, at matched aggregate accuracy on NYC Vanilla, HRA closely resembles an earlier GRPO checkpoint, consistent with your reading of HRA as a probe rather than a method. On presentation, the revision opens with a running example, states the conclusion of §3.4 first, adds numerical examples in §3.3, §3.4 and §4.1, and adds a glossary (A.30). We would be glad to answer any remaining questions.

Thank you for the response. I will maintain my score.

AuthorsOpenReview ↗

Follow-up on remaining concerns

Thanks for reading our response and letting us know where you stand. To help us understand what weighs most in your assessment, could you share whether a specific soundness or presentation concern remains after the 3-seed runs, binary-reward and multiple-choice tests, sampled-baseline check, and readability revisions, or whether your reservation mainly concerns scope beyond bounded candidate choice (such as multi-step reasoning and code)? Any brief pointer would be very helpful.

Reviewer_eJXEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how GRPO can improve aggregate accuracy while reducing accuracy on samples that contradict a learned rule when observations are limited, even with an exact reward. Using next POI ranking experiments, the authors relate this behavior to gradient coherence and use an advantage–position identity to explain the role of group mean centering. They also test whether coherence predicts the lock target and introduce heuristic relative advantage (HRA) to examine how changing the advantage center affects subgroup performance.

Critical strengths
  • The paper addresses an important issue: improvements in aggregate performance can hide substantial losses for particular groups of samples. The controlled experiments make this tradeoff clear and show that similar behavior occurs with both count and category rules.

  • The gradient analysis distinguishes the size of individual gradients from how well their directions align. Comparisons using the same rollouts and different output labels, together with experiments in another city and with another model, strengthen the empirical observations.

Critical weaknesses
  • The advantage position identity establishes that centering cancels shared gradient components, but its role in determining which rule is learned is less clearly established by the experiments. The main control removes reward dependence as well as centering, and it remains unclear whether the subgroup changes under HRA differ from those obtained by simply training GRPO less.

  • Although HRA produces similar directional changes across settings, the use of one training seed per configuration leaves its sensitivity to training variation unclear. The feature prediction experiment also favors category in both coherence and rule ceiling, so the additional predictive value of coherence is not isolated.

Questions for authors
  • Could the authors complement Proposition 1 by comparing centered and uncentered reward weights on the same rollouts, using identical reward normalization? Specifically, how does the decay in gradient cosine similarity with target count rank distance in Figure 2 change when only the mean subtraction is removed?

  • Across independent training seeds, does HRA produce a different pattern of subgroup recovery from earlier GRPO checkpoints or GRPO trained with a smaller learning rate at comparable aggregate HR@1? Since HRA is intended as an interventional test, comparing recovery by target rank and the corresponding rule follow rates would help distinguish its effect from a difference in how strongly the rule has been learned.

  • Does coherence still predict the lock target when the count and category rules have matched predictive strength, or when predictive strength favors the less coherent rule? Table 3 gives category both higher coherence and a higher rule ceiling (0.680 vs. 0.622), so a comparison that also accounts for the rules' expected training rewards would help separate these explanations.

5 comments
AuthorsOpenReview ↗

Response to Reviewer eJXE (1/2)

Thank you for recognising the importance of the issue and that the gradient analysis separates the size of individual gradients from how well their directions align. Each of your three questions names a control. We ran the controls for Q1 and for Q3 (matched predictive strength). For Q2, the three-seed results and the by-rank comparison with other weaker-lock runs are below, and the comparison with earlier GRPO checkpoints at matched aggregate accuracy is in our follow-up.

1. What does centring contribute to which rule is learned? (W1)

Centring makes the difference between the two rules clearly measurable, but in our experiments it does not change which rule is more coherent.

Coherence of the two rules' gradients on the Tokyo base model (TKY-B):

Rollout weightCategory − countRatio
Raw reward (no mean subtraction)+0.0071.2×
Group mean subtracted (GRPO)+0.017 to +0.0193.6–5.0×

(The second row's range covers the training parse and a stricter parse, A.4; both difference intervals exclude 0.) Without the mean subtraction, both rules' gradients are dominated by the component all rollouts share, so their coherence is high and similar; removing it widens the gap. Category is higher in both cases. The revised abstract and §3.4 therefore say that centring exposes this difference, which is what lets coherence measured under GRPO's gradient predict GRPO's lock before training. Our prediction, written before training, was that training without the mean subtraction again locks category; it does (follow-up below).

2. Same rollouts, same normalisation, only the mean subtraction removed (Q1)

The decay disappears. With the standard-deviation normalisation kept in both cases and only the mean subtraction removed, the structure of Figure 2 falls from +0.017 [+0.010, +0.023] to +0.001 (interval contains 0) (General Response B, A.25).

3. Does HRA differ from training GRPO less? (W1, Q2)

It depends on the task: on NYC Vanilla, where visit count is the only cue, HRA does not differ from a weaker lock by the measures you suggest, as §4.3 implies: recombining the gradients of sampled rollouts can shift the cost but not remove it. On TKY-B, where two cues compete, HRA's behaviour is not what less training would produce.

On TKY-B (one seed; Tables 6 and 23), the count-anchored HRA raises count-rule following to 0.818, above both the base model (0.668) and GRPO (0.775); the category anchor reaches 0.802. This lies outside the range between the base model and the GRPO endpoint. On W=0 prompts whose target has count-rank 1, HRA reaches HR@1 of 0.448, against 0.409 at base and 0.375 under GRPO, and the two anchors direct the recovery to different ranks (count anchor: rank 1; category anchor: rank 2).

On NYC Vanilla, by target rank (A.28):

  • Rule following: HRA follows the count rule 1–2 points less than GRPO at every target rank, uniformly.
  • Recovery: concentrated at rank 2 (+1.2 points) and near zero at ranks ≥ 3. When the model leaves the top candidate it mostly picks the second, so any weaker lock recovers rank 2 first.
  • Against other weaker-lock runs: rank-2 recovery per unit of lock weakening is 0.74 for HRA and 0.73 and 0.80 for the binary and top-3 rewards (intervals of the differences contain 0); all lie on one trade-off line whose slope can be computed in advance (General Response D).

The comparison you ask for, against an earlier GRPO checkpoint with the same aggregate HR@1 as HRA, is in our follow-up below.

4. One training seed per configuration (W2)

There are now three seeds (General Response A, A.26). In the main runs no subgroup differs by more than 1.5 points across seeds. HRA raises both misaligned subgroups in all three seeds (+0.6 and +1.8 points), lowers both aligned subgroups in all three, and every seed lies on the trade-off line. The rule-prediction tests (NYC-B, TKY-B and the multiple-choice task) also have three seeds each, and each locks the same rule in every seed.

AuthorsOpenReview ↗

Response to Reviewer eJXE (2/2)

5. Does coherence still predict the lock when predictive strength is matched? (W2, Q3)

Yes: with predictive strength and expected reward exactly equal, coherence measured before training predicted the locked rule (General Response C, A.27). Your criticism of Table 3 is right: the Tokyo experiment alone cannot separate the two explanations.

As you suggest, we computed the expected training reward in Tokyo. For a policy that ranks the rule's choice first and the rest at random, on W=1 training prompts category earns 0.954 and count 0.922; on the 37,989 W=1 prompts where the two rules make unique, different choices, category is correct on 79.6% and count on 12.0%. (Over all training prompts the order reverses, 0.787 vs. 0.829, but a policy can use category where W=1 and count elsewhere; A.18.) TKY-B thus shows that training goes against the base model's preference, but it cannot isolate payoff; NYC-B likewise.

In the multiple-choice task, the position cue (option D) and the length cue (longest option) are equally accurate overall (51.5% each) and on the items where they disagree (34.9% each); their expected training rewards are therefore also equal. The base model mildly prefers length; coherence favours position; we wrote down "position" before training. All three seeds lock D (25% → 62.5%–72%, while ranking the longest first rises by about 6 points), and weaker 40% cues give the same result. A higher-reward policy that combines both cues would itself pick D often (63.7% of items), so we checked the items on which it picks the longest option instead: trained models still pick D on 36%–76% of them and agree with its first choice on only 45%–50% of items. Training locked the position rule, not the combined policy.

The case where predictive strength favours the less coherent rule is not covered. §3.5 and Table 3 are rewritten to state the Tokyo limitation, and the new test is in §3.5 (Table 4) and A.27.

We hope these answers resolve your concerns and would be grateful if you would consider updating your score.

AuthorsOpenReview ↗

Follow-up on Q2: HRA versus GRPO at matched aggregate accuracy

We now provide the requested comparison with earlier GRPO checkpoints at approximately matched aggregate accuracy.

HRA's recovery by rank and its rule following fall between those of GRPO at steps 125 and 150, the two checkpoints that bracket its aggregate accuracy. No GRPO checkpoint matches the aggregate HR@1 of HRA (0.4885, trained for 200 steps) exactly; the checkpoints at steps 125 (0.4830) and 150 (0.4918) bracket it, so we compare HRA with both (seed 42, full test set, user-level paired bootstrap).

Target count-rank (N)HR@1: step 125 / HRA / step 150Count-rule following: step 125 / HRA / step 150
1 (19,364)0.865 / 0.882 / 0.8890.889 / 0.906 / 0.913
2 (7,472)0.132 / 0.116 / 0.1150.855 / 0.874 / 0.876
3 (4,253)0.010 / 0.008 / 0.0070.863 / 0.886 / 0.888
≥4 (5,745)0.001 / 0.001 / 0.0010.868 / 0.885 / 0.888

HRA lies between the two checkpoints on 58 of the 60 comparisons specified before examining this comparison: HR@1 at each of the 8 exact target ranks, and HR@1 and count-rule following overall, on all non-popular prompts and in each of the four quadrants, each computed pooled, at W=0 and at W=1. The two exceptions are HR@1 at target rank 4 in the W=0 and W=1 layers, where HRA differs from both checkpoints by a single prompt (W=0: 1 vs. 2 correct out of 1,938; W=1: 2 vs. 1 out of 797; every interval for the differences includes 0). The by-rank rule following in the table, which was not part of the pre-specified set, also lies between the checkpoints.

This is consistent with the by-rank analysis in item 3 and with General Response D: in recovery by rank and in rule following, HRA after 200 steps closely resembles GRPO at an earlier point in training. The comparison uses one seed, and we did not train GRPO with a smaller learning rate.

Reviewer_nWPdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how GRPO learns simple feature-following rules when rewards are correct but the input does not fully identify the correct answer. Using next-POI ranking, it reports that overall accuracy can improve while performance on samples that contradict the learned rule gets worse. The authors connect this behavior to the coherence of subgroup gradients and derive an identity expressing the centered update through position-dependent gradient contrasts. They test feature selection in a second city and use a heuristic-relative advantage as an intervention. The reported intervention partly restores performance on affected subgroups while giving up some gains elsewhere.

Critical strengths
  1. The subgroup results are worth reporting. In NYC Vanilla, overall HR@1 increases from 0.4591 to 0.4953 while both non-popular quadrants get worse. The category constructions, Tokyo experiments, and Llama replication show that this is not confined to one feature or one model. Looking only at the aggregate score would miss most of the story.

  2. I appreciated the effort to examine the gradients rather than stop at the accuracy tables. The norm comparisons, same-rollout controls, and pre-training Tokyo test all help narrow the explanation. The HRA results are also reported with their costs to other subgroups, including the failed prediction about which anchor should work better.

Critical weaknesses
  1. I do not see why Theorem 1 describes the optimum of the actual task. The rearrangement argument works for a fixed permutation of distinct count ranks. The model, however, sees numerical counts and could use their magnitudes, gaps, or tie patterns to choose different rules on different prompts. Removing POI names in Appendix A.5 does not rule this out. Appendix A.6 also allows tied ranks, which do not form the permutation assumed in the proof. The reward needs clarification too: Appendix A.4 says the temporal bonus is top-1 gated, while Appendix A.16 says top-3 with additional mode conditions. Until these points are resolved, the claim that the subgroup loss is an unavoidable cost of the task optimum is too strong.

  2. The experiments do not isolate centering as the cause of the effect. Setting every advantage to 1 removes reward weighting as well as centering. Proposition 1 shows that a shared gradient component cancels, but the remaining directions still depend on the model and its outputs. HRA is a useful intervention, although its denominator, fallback, and clipping make the result harder to interpret. Appendix A.24 ignores the denominator's correlation with the samples in the expected-update argument. Also, summing gradients over prompts does not explain why an input-conditional model must learn an unconditional rule. I would need cleaner controls before accepting this causal account. Multiple HRA training seeds are important here: rollout resampling and the separate DAPO run do not establish that the HRA effect is stable across runs.

Questions for authors
  1. What justifies restricting the Vanilla policy to a global count-rank permutation? Does a predictor using count magnitudes, gaps, and tie patterns improve held-out performance over the count-only rule? Please also state how Theorem 1 handles tied ranks.

  2. Is the temporal bonus applied when the target is first, or when it is in the top three? Please provide the exact training reward and either establish that it preserves the claimed ordering or restrict the theorem's interpretation to the position-only reward.

  3. Does the claimed centering effect persist in matched comparisons that retain reward weighting while varying the baseline and normalization separately? Multiple training seeds would be important for the HRA comparison, since the reported intervention effects are smaller than the main lock-in effect.

3 comments
AuthorsOpenReview ↗

Response to Reviewer nWPd (1/2)

Thank you for checking the text against the appendix so closely. Your comments found a real error in A.4 and three statements that went beyond the evidence (the policy class covered by Theorem 1, the role of the model in Proposition 1, and the step from summed gradients to an input-blind rule); the revision corrects all four. We also appreciate your note that the HRA results are reported with their costs, including the failed anchor prediction.

1. Why only rank-based policies? How are ties handled? (W1, Q1)

The theorem covers policies that act on count order. The predictor control and the proof for ties that we added both support the conclusion, and §3.1 now states it in these terms.

Predictors trained to pick the correct venue (gradient-boosted trees shown; logistic regression gives the same):

MethodTest W=0Test W=1
Most-visited candidate (Table 8 ceiling)0.4720.577
Predictor: exact counts, gaps, ties0.4700.575
+ other prompt fields and two user statistics not shown in the prompt0.4700.578

None improves on count order (A.5; as A.5 also notes, the prompt shows only each candidate's label, category and global visit count, with no venue names). Accordingly, the main text no longer calls the cost unavoidable: it is the price of the best policy we found within the information visible in the prompt (§5). On ties (proof in A.2): orders within a tie block are equivalent, and blocks should be ordered by per-candidate target probability, which on the training data decreases with count-rank.

2. The exact training reward (W1, Q2)

The bonus is given for the top 3. A.16 was correct and the original A.4 was wrong: the term is a visit-count preference match, not a time-window bonus. A.4 has been rewritten, and enumeration shows that descending count remains the unique optimum under the full reward.

Rfull=Rposition+Rbonus+RformatR_{\text{full}} = R_{\text{position}} + R_{\text{bonus}} + R_{\text{format}}:

  • RpositionR_{\text{position}}: from 1 at rank 1 to 0 at rank 8, linear (Eq. 1); −0.5 if the correct candidate is absent.
  • Rbonus=0.2exp⁡(−∣vc−mu∣/(2(su+1)))R_{\text{bonus}} = 0.2\exp(-|v_c - m_u|/(2(s_u+1))) if (i) the behaviour mode is deliberate or social, (ii) at least 3 of the user's 5 most recent check-ins share it (mum_u, sus_u: median and SD of those venues' global visit counts), and (iii) the correct candidate is in the top three; otherwise 0. vcv_c is the correct venue's global visit count.
  • RformatR_{\text{format}}: 0.1 for an explicit "X > Y" ordering.

Theorem 1 is stated for the position reward. Per prompt, the bonus is non-negative and constant over ranks 1–3, so ranking the target first is best (checked on 2,000 prompts). At the policy level, we enumerated all 8!=40,3208! = 40{,}320 maps from count-ranks to positions on the training set (ties broken at random): descending count is the unique maximiser and also maximises the position and bonus terms separately (A.4).

3. Isolating centring; the model's role in Proposition 1 (W2, Q3)

We agree with both points. With reward weighting kept, removing only the group-mean subtraction removes the structure (+0.017 → +0.001, interval contains 0), and the normalisation has no effect (General Response B, A.25).

The remaining directions do depend on the model's outputs, and the original abstract's "not inherited from the model's representations" was too strong. The model's outputs map count-ranks to positions, and positions to gradient directions (§3.4); without centring, the component shared by all rollouts buries these differences, and subtracting the mean exposes them. The abstract and §3.4 are rewritten accordingly.

4. HRA's denominator, fallback and clip; independence in A.24 (W2)

These safeguards act almost only on groups whose rewards are all identical.

  • Denominator: as in GRPO; only the centre differs.
  • Fallback: one rollout off by a single rank already gives SD ≈ 0.057 > 0.05, so the threshold catches groups without disagreement. Replaying the HRA rule on archived GRPO rollouts, 98.6% of fallback groups have identical rewards. Under HRA their numerator r−rHr - r_H is generally non-zero and, without the fallback, would be divided by ϵ=10−4\epsilon = 10^{-4}.
  • Clip: never reached in the replay (largest advantage: 8.5).
  • A.24: dividing by a positive SD preserves sign, so the non-positive level term needs no independence. On replayed data the numerator–denominator correlation is about −0.10; with it, signs and the ordering of subgroups are unchanged.

5. Summing gradients does not explain an input-blind rule (W2)

We agree; the statement has been removed from §3.4 and Figure 1. The revision explains spillover by the input's inability to tell when the rule fails (end of §3.2).

W depends on the correct candidate's category, which the model does not know, so a W=0 prompt whose history contains another candidate's category looks like a W=1 prompt.

AuthorsOpenReview ↗

Response to Reviewer nWPd (2/2)

6. HRA seeds (W2, Q3)

We now have three seeds, and HRA's two-way shift has the same sign in all of them (General Response A, A.26).

Against same-seed GRPO: misaligned +0.6 and +1.8 points, aligned −2.4 and −3.4, silent fraction about 47% → 26%. HRA's across-seed SD is 0.1–0.6 points per subgroup, and all three seeds lie on the trade-off line of General Response D.

We hope these answers resolve your concerns and would be grateful if you would consider updating your score.

AuthorsOpenReview ↗

Short summary before the discussion closes

Thanks again for your review.

Before the discussion closes, here are the direct answers to your three questions:

  • Q1: no predictor using exact counts, gaps, ties or other prompt fields improves top-1 accuracy over count order, and ties are covered by the proof in A.2 (item 1);
  • Q2: the exact reward is stated in item 2, and enumeration of all 40,320 orderings shows that descending count remains the unique optimum under it;
  • Q3: with reward weighting kept on the same rollouts, removing only the group-mean subtraction removes the rank structure, and the normalisation leaves it unchanged (General Response B).

HRA now has three seeds (General Response A), and training without the group-mean subtraction locks the same rule (General Response, addendum F). We would be glad to answer any remaining questions.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response (1/3)

We thank all four reviewers for their detailed and concrete comments. The reviewers recognised the phenomenon the paper reports (under an exact reward, aggregate accuracy rises while a misaligned subgroup systematically degrades) and that it differs from reward hacking, since the reward is correct; the gradient-direction analysis; and the controls across features, cities and models. The main questions concern whether the key controls are clean, whether single-seed results are robust, how far the conclusions extend, and the readability of some sections. This part summarises our answers; Part 2 reports four new sets of results (A–D), all included in the revised PDF (in blue). Each reviewer's reply covers the remaining specific points.

Terms.

  • Aligned prompts: among 8 candidates, the correct answer is the most-visited one (in general, the one the locked rule picks); the others are misaligned.
  • Lock-in: after training, the model almost always ranks the most-visited candidate first.
  • Coherence: whether the gradients of many prompts point the same way.
  • Centring: GRPO subtracts the mean of a prompt's 5 rollout rewards, which removes the gradient component the rollouts share.
Shared questionRaised byAnswer
One seed per configuration: are the results robust?All fourYes (A, A.26). The main runs and HRA now have 3 seeds, with no subgroup differing by more than 1.5 points across seeds; the rule-prediction tests at the main cue strength also hold in all 3 seeds.
Was centring isolated from reward weighting?nWPd, eJXE, zDmXNow isolated (B, A.25). With reward weighting kept, removing only the group-mean subtraction removes the rank structure; the normalisation has no effect. Training without the mean subtraction locks the same rule (addendum F).
In Tokyo, the locked rule also pays more.eJXE, zDmXAgreed; §3.5 now says so. New test (C, A.27): two rules of equal accuracy, the base model preferring the other one; the coherence prediction written before training holds in 3 of 3 seeds.
Does HRA differ from a weaker lock? Why only partial recovery?eJXE, Aaet, zDmXHRA moves the cost in the predicted directions; and on NYC Vanilla the movement matches a weaker lock (D, A.28). The shift holds in all three seeds, and IAC moves the cost the opposite way. As §4.3 argues, recombining the gradients of sampled rollouts can shift the cost between subgroups but not remove it.
Does it extend beyond POI and a linear reward?Aaet, zDmXYes: lock-in and its cost reappear on multiple-choice ranking (C) and under a binary reward (A.29); the general form of the identity does not need a linear reward (A.15).
Training reward; ties and value-based policies; HRA's fallbacknWPd (and zDmX)A.4 was wrong and is rewritten, with an exhaustive check over 40,320 orderings; a predictor control and a proof covering ties are added; the fallback and clip act almost only on groups with identical rewards.

Corrections and scope.

  • The original A.4 misdescribed the bonus term (A.16 was right); we thank Reviewer nWPd for catching this. In the original §3.5, the Tokyo category rule also had the higher payoff, so that experiment could not separate coherence from payoff (we thank Reviewers eJXE and zDmX for pointing this out); §3.5 is narrowed, and experiment C holds the payoff equal.
  • A second task. The multiple-choice experiment (C) supplies what the POI task could not: two cues whose payoff is equal by construction, so that payoff cannot explain which one is locked; a computable optimal policy that combines both cues, against which the trained model can be checked; and items the base model already answers correctly, which show that the lock makes the model fail on questions it knew. It also moves the phenomenon from venue recommendation to general-domain questions (MMLU-Pro, SuperGPQA).
  • Scope. Our conclusions concern choice from a bounded candidate set with a cue that predicts the answer only on average (ranking, recommendation, multiple-choice evaluation), as the introduction and §5 now state. They have not been tested on open-ended generation such as multi-step reasoning or code generation.

Other revisions. For readability (raised by Reviewers Aaet and zDmX): a running example opens the introduction; §3.4 states its conclusion first; the roles of the theorem (where reward maximisation ends among count-order policies), the identity and controls (where the gradient structure comes from) and the prediction tests (empirical checks) are explicit; §3.3, §3.4 and §4.1 each gain a short numerical example; a glossary is added (A.30). Related work is condensed in the main text, with the full discussion in A.1. For the final version: three seeds for the binary reward, and release of code, data scripts and model outputs. Training without the mean subtraction is reported in addendum F.

AuthorsOpenReview ↗

General Response (2/3)

Part 2: new results

A. Three training seeds (A.26)

Across three seeds, every subgroup moves the same way, no subgroup differs by more than 1.5 points, and HRA's two-way shift has the same sign in every seed.

Subgroup (HR@1)BaseGRPO (3 seeds)HRA (3 seeds)HRA − GRPO (same seed)
Misaligned, first visit0.0490.025 ± 0.0020.031 ± 0.001+0.6 pts, up in all 3
Misaligned, revisit0.1210.061 ± 0.0040.079 ± 0.003+1.8 pts, up in all 3
Aligned, first visit0.7770.875 ± 0.0050.850 ± 0.004−2.4 pts, down in all 3
Aligned, revisit0.8040.918 ± 0.0050.884 ± 0.006−3.4 pts, down in all 3
Overall0.4590.497 ± 0.0010.486 ± 0.002−1.1 pts, down in all 3

(Mean ± population SD over seeds.) The seed-to-seed variation is far smaller than the lock-in itself. Speed of lock-in, error concentration and the silent fraction also replicate (details in our reply to Reviewer Aaet). NYC-B and TKY-B, retrained with three seeds, lock category in all six runs; payoff and coherence agree in those constructions, so these runs show stability, and C addresses the payoff confound.

B. Removing only the mean subtraction (A.25)

With reward weighting kept, removing only the group-mean subtraction makes the rank structure of Figure 2 disappear (+0.017 → +0.001); dividing by the standard deviation changes nothing.

On the same outputs, we vary the baseline and the normalisation separately. "Structure" is the measure of Figure 2: gradient similarity of prompt pairs with close target count-ranks minus that of pairs with distant ones.

Rollout weightClassical counterpartStructure [95% interval]
Raw rewardREINFORCE, no baseline+0.001 [−0.003, +0.006]
Reward − mean of the prompt's 5 rolloutsGRPO+0.017 [+0.010, +0.023]
Reward − mean over all promptsREINFORCE, constant baseline+0.042 [+0.034, +0.051]
All set to 1original control0.000 [−0.005, +0.004]
  • Without a baseline the structure is absent; subtracting the group mean exposes the within-prompt contrast of Proposition 1. Dividing by the within-group standard deviation rescales each prompt's gradient and leaves the structure unchanged.
  • The batch-mean baseline also gives a positive value, but from differences in difficulty between prompts: difficulty tracks count-rank, and shuffling the prompt-level mean rewards removes about 88% of it.

Training without the mean subtraction locks the same rule (addendum F).

C. An equal-payoff prediction test on multiple-choice ranking (§3.5, Table 4, A.27)

With two cues of exactly equal accuracy, training locked the one that was more coherent before training, against the base model's preference and as predicted before training, in all three seeds; the misaligned cost and the spillover reappear.

Design. Eight-option ranking items from MMLU-Pro and SuperGPQA (27 subject categories), with a reward graded by the position of the correct option. The answer is option D on 51.5% of items (position cue) and the longest option on 51.5% (length cue); the two rules are equally accurate overall and where they disagree (897 correct for each of the 2,573 training items on which they name different options). Before training, the base model mildly prefers length while coherence favours position; we wrote down the prediction "position D" before training began.

Results.

  • Prediction holds in every seed. Ranking D first rises from 25% to 62.5%–72%; ranking the longest option first rises by about 6 points. With both cues weakened to 40%, position is again locked (one seed).
  • The cost and the spillover reappear. Aggregate accuracy rises by about 17 points while accuracy on items whose answer is not at D falls by 9–15 points, including on items answered correctly before training, and falls in 26 of 27 source-by-subject strata. On a test set where D is the answer only at chance rate, aggregate accuracy falls below base. After training, the gradients of items whose answer is at D add up while the others cancel, at similar norms.
  • Not a combined policy. Because the two cues are correlated, a higher-reward policy that uses both would itself pick D on 63.7% of items, so the rise in picking D alone cannot tell the two apart. The test is the items on which that combined policy picks the longest option rather than D: trained models still pick D on 36%–76% of them, and agree with its first choice on only 45%–50% of all items. Training locked the position rule, not the combined policy.

Although the coherence gap is modest (about 1.4×; not significant on its own), the prediction written before training went against the base model's preference for length, and it held in all three seeds and with weaker (40%) cues.

AuthorsOpenReview ↗

General Response (3/3)

D. HRA and a weaker lock: one trade-off line (Appendix: A.28, Figure 3)

HRA moves the cost in the directions the diagnosis predicts, and on NYC Vanilla it does so in the same way as a weaker lock: ten configurations lie within 0.11 points of one line whose slope can be computed in advance.

HRA tests the diagnosis: with the same steps and hyperparameters, replacing only the group mean by a fixed baseline moves the cost in the predicted directions, IAC (which keeps the group mean) moves it the opposite way, and on TKY-B the spillover returns to within a point of base. We then plot ten configurations on NYC Vanilla (base; GRPO at steps 200 and 400; HRA; IAC; DAPO; binary reward; top-3 reward at three checkpoints), gain in ranking the most-visited candidate first against loss on misaligned prompts.

  • A line fitted to base and GRPO-200/400 only lies within 0.11 points of the seven other configurations (HRA: 0.06; its two other seeds are also on the line). Each point of lock-in costs about 0.38 points of misaligned accuracy.
  • The slope can be computed in advance, from the target distribution and where the base model's picks go when it leaves the top candidate: 0.36–0.39.
  • Recovery concentrates at rank 2 because the model's next choice is mostly the rank-2 candidate. Per unit of lock weakening, HRA recovers 0.74 at rank 2, against 0.73 and 0.80 for the binary and top-3 rewards (intervals of the differences contain 0).

HRA therefore moves along the line, as §4.3 implies for any recombination of sampled rollouts. The line is an empirical relation for this task and model, not a general bound. HRA is a probe of the explanation, not a method for improving accuracy; recovering the misaligned prompts needs information that separates them or an objective that values them (§5).

AuthorsOpenReview ↗

General Response, addenda E and F: HRA at matched accuracy; training without the group-mean subtraction

We add two results obtained during the discussion period.

E. HRA at matched aggregate accuracy

At matched aggregate accuracy, HRA lies between two GRPO checkpoints on 58 of 60 comparisons fixed in advance, as §4.3 implies for any recombination of sampled rollouts.

Reviewers asked whether HRA's recovery differs from that of GRPO trained for fewer steps (Reviewer eJXE) and why it is only partial (Reviewers Aaet and zDmX). On NYC Vanilla, where visit count is the only cue (seed 42, full test set), GRPO at steps 125 and 150 brackets the aggregate HR@1 of HRA at step 200 (0.4830 and 0.4918, against 0.4885).

Before evaluating these checkpoints, we fixed 60 comparisons: HR@1 at each target rank (1–8), and HR@1 and count-rule following overall, on non-popular prompts and in each quadrant, each pooled and within the W=0 and W=1 layers. HRA lies between the two checkpoints on 58; the other two, HR@1 at target rank 4 in the W=0 and W=1 layers, differ from both checkpoints by a single prompt.

HRA therefore moves along the trade-off line of Part D: it recovers the misaligned prompts in proportion to how much it weakens the lock. The full table is in our follow-up to Reviewer eJXE.

F. Training without the group-mean subtraction

Without the group-mean subtraction, training locks the predicted rule and reproduces the W=0 cost.

Design. To test whether lock-in requires the group-mean subtraction, we trained two variants on TKY-B for 200 steps:

  • V0 divides each reward by the standard deviation over the batch (16 prompts × 5 rollouts) and subtracts no baseline, i.e. REINFORCE without a baseline.
  • V1 also subtracts the batch mean, i.e. REINFORCE with a batch-mean baseline (the "constant baseline" row of Part B).

All other settings follow our GRPO runs, with a 200-step cosine learning-rate schedule. V0 was run with two seeds and V1 with one.

RunCategory-rule following (all prompts)Count-rule following (all prompts)W=1 HR@1W=0 HR@1Prediction test95% CI
Base66.2%66.9%55.0%22.6%––
V0, seed 4276.4%72.3%66.8%21.2%+0.0480.044 to 0.053
V0, seed 123477.3%73.1%67.9%21.1%+0.0500.045 to 0.055
V1, seed 4280.3%77.3%71.4%21.3%+0.0440.039 to 0.049

The prediction test is computed on the W=1 layer from the quantities in Table 3: the rise in category-rule following minus the rise in count-rule following, relative to the base model. It passes if this is positive with an interval that excludes 0.

Results. In all three runs, category-rule following rises more than count-rule following and the prediction test passes; the two V0 seeds give nearly the same value (+0.048 and +0.050). W=1 HR@1 rises, and W=0 HR@1 falls below the base model by 1.36 to 1.50 points (user-level paired bootstrap; every interval excludes 0), reproducing the spillover cost of §3.5.

Interpretation. This is the outcome we predicted before training (our reply to Reviewer eJXE, item 1). A baseline that does not depend on a rollout's own action leaves the expected policy gradient unchanged. The group mean and the batch mean each include the rollout's own reward, with weight 1/5 and 1/80, so before standard-deviation normalisation, subtracting either leaves the direction of the expected gradient unchanged and rescales it by 4/5 or 79/80. The update rules of these runs differ from GRPO's only in the centre (none, or the batch mean) and in dividing by the batch rather than the per-prompt standard deviation, and training confirms that the same rule wins.

Together with Part B, this separates two roles. Which rule is locked is the same with or without centring. What centring contributes is a per-prompt gradient structure in which the two rules' difference in coherence is clearly measurable before training (Part B), and this is what allows the lock to be predicted. This is the role the revised abstract and §3.4 give it: centring exposes the coherence rather than creating it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.