- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates a potential failure mode of Group Relative Policy Optimization (GRPO), where optimizing an accurate and verifiable reward can improve aggregate performance while systematically degrading performance on certain subgroups. Using next-point-of-interest (POI) recommendation as a controlled testbed, the authors demonstrate that GRPO tends to converge toward a dominant feature-based ranking rule, benefiting samples aligned with this rule while penalizing misaligned samples. The paper attributes this asymmetric behavior to gradient coherence rather than gradient magnitude, showing that gradients from aligned samples reinforce each other while those from misaligned samples tend to cancel out. To explain this phenomenon, the authors derive an advantage–position identity and investigate how group-relative advantage normalization influences gradient geometry. They further propose a gradient-coherence-based criterion for predicting feature lock-in and introduce Heuristic-Relative Advantage (HRA) as an intervention to redistribute subgroup performance. Experiments across different feature constructions, cities, and model backbones provide empirical support for the proposed analysis.
Critical strengths
- The paper investigates an important and relatively underexplored failure mode of group-relative reinforcement learning. Unlike conventional reward hacking, which often involves misspecified rewards, this work highlights that subgroup performance degradation can occur even when the reward is accurate. The distinction between aggregate reward improvement and subgroup-level deterioration is meaningful, particularly for recommendation systems and other applications with heterogeneous user preferences.
- The paper provides a systematic analysis connecting gradient coherence, advantage normalization, and feature lock-in. Rather than merely reporting performance degradation, the authors investigate gradient directions, aggregation efficiency, and the relationship between target rank and gradient similarity. The advantage–position identity and the empirical gradient measurements offer a useful perspective for understanding how subgroup contributions interact during GRPO optimization.
Critical weaknesses
-
The generalizability of the proposed findings is limited by the highly controlled experimental setting. The empirical investigation is primarily conducted on next-POI recommendation with eight candidates, a position-based reward, and a small set of explicitly constructed features, such as historical visit count and category. This setting provides a convenient environment for analyzing ranking behavior, but it differs substantially from more general RLVR applications, particularly mathematical reasoning, code generation, and complex decision-making tasks. The proposed gradient-coherence mechanism depends on the relationship between candidate rank, output position, and gradient direction, which may not naturally extend to tasks without such explicit positional structures. Although the authors include experiments across two cities and two model backbones, these experiments still share essentially the same task formulation. Consequently, the evidence is insufficient to establish gradient coherence as a general explanation for feature lock-in in group-relative reinforcement learning.
-
The theoretical contribution does not fully establish the claimed relationship between gradient coherence and feature lock-in. Theorem 1 characterizes the optimal ranking strategy under a strictly decreasing target-rank distribution, but this result follows directly from the rearrangement inequality and is independent of GRPO or gradient-based optimization. Therefore, it establishes the optimality of the ranking rule rather than explaining why GRPO specifically learns that rule. Similarly, Proposition 1 provides an exact algebraic decomposition of the group-relative gradient under simplified assumptions, but it does not establish that gradient coherence is a necessary or sufficient condition for feature lock-in. The argument connecting advantage centering, gradient alignment, and the eventual dominant feature still relies heavily on empirical observations. A more rigorous characterization of the conditions under which gradient coherence determines the optimization outcome would substantially strengthen the theoretical contribution.
-
The causal role of group-relative advantage centering is not sufficiently isolated from other optimization mechanisms. A central claim is that the observed gradient coherence arises from centered advantages rather than the model's learned representations. The authors support this claim by comparing advantage-weighted gradients with unweighted gradients on identical rollouts. However, removing advantage weighting fundamentally changes the optimization signal, and the disappearance of the observed gradient structure does not necessarily establish that group-relative centering is the unique cause of the phenomenon. In particular, action-independent baselines are a standard component of policy-gradient methods, and subgroup gradient interference may also arise under alternative reinforcement learning objectives. Comparisons with REINFORCE, PPO using alternative baselines, or other advantage estimators would help determine whether the reported behavior is specific to GRPO or reflects a more general property of policy-gradient optimization.
-
The experimental evidence for the predictive capability of gradient coherence remains insufficient. One of the paper's central contributions is a criterion that predicts which feature will dominate training based on gradient coherence measured before optimization. However, the strongest prospective validation is primarily conducted in the Tokyo Construction B setting, where the coherence criterion favors category over count. Moreover, the category rule also has a higher achievable performance ceiling than the count rule, making it difficult to completely disentangle the effects of gradient coherence and objective strength. The current evidence does not establish whether the proposed criterion consistently predicts feature selection across a broader range of competing features, correlation strengths, reward distributions, and model initializations. Furthermore, the training experiments are conducted with a single seed, which limits confidence in the robustness of the reported feature-selection dynamics. Additional controlled experiments in which feature informativeness and gradient coherence vary independently would provide stronger evidence for the predictive claims.
-
The proposed HRA intervention provides limited practical improvements and does not fully resolve the identified failure mode. Although Heuristic-Relative Advantage is primarily introduced as a diagnostic intervention, its experimental effectiveness remains limited. In Table 4, HRA only partially recovers performance on the misaligned subgroups, while reducing performance on the aligned subgroups and slightly decreasing overall accuracy compared with GRPO. The misaligned subgroups still perform below the base model. Moreover, HRA relies on a predefined heuristic and its expected reward, which may be difficult to obtain in more complex applications where the dominant shortcut or reference strategy is unknown. The method also introduces additional design choices, including advantage clipping and fallback thresholds, whose effects are not thoroughly isolated. Therefore, while the intervention supports the qualitative observation that subgroup performance can be redistributed, it provides limited evidence for a generally applicable solution to the identified problem.
Questions for authors
See Weaknesses.
6 comments
Response to Reviewer zDmX (2/2)
5. HRA's limited gain; clip and fallback (W5)
HRA is a controlled probe of the diagnosis (§4.1, §5). Partial recovery, a slight overall drop and accuracy still below base are what §4.3 implies: recombining the gradients of sampled rollouts can shift the cost between subgroups but not remove it.
HRA anchors on a known rule's expected reward because it is a probe; for unknown shortcuts, the introduction gives three checks that need no closed-form baseline. Ten configurations lie within 0.11 points of one trade-off line whose slope can be computed in advance, and HRA moves along it (General Response D). Replaying HRA on archived GRPO rollouts, the ±10 clip is never reached, and 98.6% of fallback groups have identical rewards (fallback rates in Table 24); these carry no gradient under either method, and the fallback only prevents a non-zero from being divided by .
We hope these answers resolve your concerns and would be grateful if you would consider updating your score.
Follow-up on W3: REINFORCE without a baseline and with a batch-mean baseline
Both REINFORCE variants from item 3 lock the rule predicted before training on TKY-B (200 steps; two seeds without a baseline, one with the constant, batch-mean baseline). They pass the same prediction test as GRPO (+0.044 to +0.050; all intervals exclude 0) and lower W=0 HR@1 below the base model by 1.36–1.50 points while raising W=1 HR@1. This answers your question whether the behaviour is specific to GRPO: the lock and its cost do not require the group-mean subtraction. What the group mean adds is a per-prompt gradient structure in which the two rules' difference in coherence is clearly measurable before training (General Response B), and this is what makes the lock predictable. This matches the revised claim in item 3: centring exposes the structure rather than creating it. Details are in General Response, addendum F.
Response to Reviewer zDmX (1/2)
Thank you for the systematic reading, and for recognising that this failure mode differs from reward hacking because the reward is accurate. Your questions about what the theory does and does not show, and about isolating centring, led to the clearest changes in the revision. We also worked on readability: the revision adds a running example, numerical examples in §3.3, §3.4 and §4.1, a conclusion-first §3.4 and a glossary (A.30).
1. Generality and positional structure (W1)
A second task and a second reward both reproduce lock-in, and the identity does not depend on positional structure. Our claims are limited to tasks with a candidate-choice structure.
2. Theorem 1 and Proposition 1 (W2)
We agree, and the revised §3.4 states what each part does.
3. Isolating centring; specificity to GRPO (W3)
With reward weighting kept, removing only the group-mean subtraction removes the rank structure (+0.017 → +0.001; General Response B). Lock-in itself is not specific to GRPO; what is specific is where its gradient structure comes from.
We have also revised the claim you cite: centring does not create the structure but exposes position-dependent differences already present in the model's outputs (abstract, §3.4). Any training that approximately maximises expected reward ends at the same rule (A.2); what is specific to GRPO is the source of the gradient structure and its predictability.
4. Evidence for the predictive power of coherence (W4)
The confound is real in the Tokyo experiment. In the new test, informativeness is equal by construction, and the prediction based on coherence measured before training held in all three seeds, against the base model's preference (General Response C).
Coverage.
Not yet covered: varying payoff and coherence over a wider range, including payoff favouring the less coherent rule, and testing the coherence prediction on several base models for the same task.