acceptodds
Under review as a conference paper at ICLR 2027

HFC-GRPO: Strict-Lag Selective Advantage Estimation for Zero-Variance Reward Groups

Abstract

Group Relative Policy Optimization (GRPO) loses its reward-derived advantage signal when all responses to a prompt receive the same binary reward. We introduce HFC-GRPO, a strict-lag selective advantage estimator that recovers signed advantages for these homogeneous groups while preserving GRPO advantages exactly on mixed groups. The estimator separates where to intervene from how much: the observed group type determines intervention and sign, while a historical forecast formed before generation determines the homogeneous-group correction magnitude. This design uses historical prediction to complement, rather than replace, within-group comparison. Our analysis identifies the distinction between all-rollout calibration and calibration on the outcome-selected homogeneous population, and bounds the resulting advantage and gradient-contribution errors under explicit assumptions. Across two Qwen3-8B reasoning protocols, paired comparisons favor HFC-GRPO, including a 0.65-percentage-point Acc@8 gain over RL-ZVP under a common training and evaluation pipeline. Transfer to multi-turn search agents yields a 2.12-percentage-point mean exact-match gain over paired outcome-only GRPO. Targeted replacements examine the roles of historical alignment, selective intervention, and adaptive prediction.

Primary area:
reinforcement learning
Keywords:
group relative policy optimization, reinforcement learning with verifiable rewards, zero-variance reward groups, historical feedback control, large language model reasoning, online prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WfD9OpenReview β†—
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes HFC-GRPO, which gives non-zero advantages to all-correct and all-wrong groups in GRPO using a historical predicts, while leaving mixed groups unchanged. Experiments on math reasoning and search-agent tasks show improvements over GRPO and other baselines.

Critical strengths

The idea of restoring signed advantages only on homogeneous groups, while keeping mixed-group GRPO exactly intact, is simple and well-motivated. The evaluation covers a broad range of settings.

Critical weaknesses

The notation is hard to follow; there are some unnecessary notations (For example, L173, β„Žπ‘‘ ∈ [πœ€β„Ž, 1 βˆ’ πœ€β„Ž], with πœ€β„Ž = 0.02, it does not need πœ€β„Ž.). In addition, the main paper does not define the RL-ZVP and GRPO-DBB protocols or the ID/OOD split, and all main results and ablations use Qwen3-8B. Moreover, the method targets homogeneous groups, but these are only 21–31% of groups in the main runs, where GRPO already trains reasonably well. Settings dominated by homogeneous groups should be tested to verify their methods. The gain also seems small relative to the added complexity (For example, NGRPO is a very simple method compared to their method). Finally, this is not the official format for ICLR 2027.

Questions for authors

Please see Weaknesses.

Reviewer_NUBfOpenReview β†—
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents HFC-GRPO, a strict-lag history-based estimator of advantages designed to deal with zero reward variance groups. By design, the method preserves the standard GRPO advantage estimations on mixed (having positive reward variance) groups. All-wrong groups receive βˆ’ht-h_t and all correct groups get 1βˆ’ht1-h_t, where hth_t is a success prediction computed before the rollout from past outcomes success rates. HFC-GRPO was shown to improve on Qwen-3-8B over the strongest baselines under both the RL-ZVP and GRPO-DBB protocols (+0.52 to +0.75pp Acc@8), on three other model families on DeepMath and for Search-R1 agents (+2.12EM over GRPO) at just 0.62% extra step time over GRPO under RL-ZVP.

Critical strengths
  1. The method only changes the advantages of groups with zero-variance reward. Groups with mixed rewards have exactly the same advantages as GRPO
  2. HFC-GRPO doesn’t require training a separate network, and reported overhead is just 0.62% step time
  3. The paper covers two reasoning protocols for Qwen-3-8B, extends to more model families (Llama-3.1-8B, Mistral-7B-Instruct). It also includes a set of ablations covering the used statistics, aggregation, scope, and history alignment.
Critical weaknesses
  1. Improvements over the source-best envelope are modest, up to +0.75pp Acc@8 and +0.60pp Pass@8, and per benchmark they are up to 0.75pp. On AIME24 +0.69pp Acc@8 corresponds to at most two additional correct responses out of 240 per run.
  2. Comparison with a learned predictor is missing. Using count statistic instead of an auxiliary reward model is presented as a design choice, but without such comparison, it is hard to tell whether the non-parametric estimate is a strength or a limitation . A learned predictor could also potentially generalize to unseen prompts and by construction encode prompt specifics
  3. The proposed approach heavily relies on multiple reuse of the same prompt, while RL-ZVP visits each prompt at most 2 times. For a first visit, the predictor falls back to a global success rate over all prompts. It’s unclear if the results would translate to large datasets where each prompt is visited once or a few times. A sweep over per-prompt visit limit could clarify this
  4. Advantages within homogeneous groups are uniform for all trajectories, so [correct CoT, wrong answer], [partially correct CoT, wrong answer] and [wrong CoT, wrong answer] will all be unlearned equally. Comparison against intermediate step rewards is conducted only in agent setting, and math-reasoning experiments do not include such a baseline.

Because of weaknesses 1 and 2 I am leaning to rejection

Questions for authors
  1. It would be interesting to see if a Value model’s or trained success probability head’s predictions would correlate with hth_t, and if using this trained predictor improves over HFC-GRPO
  2. On mixed groups hth_t is unused, but it would be interesting to see how accurately it predicts the real success rate ZtZ_t. Table D.5 reports Brier score over all rollouts and on homogeneous groups, could you extend it to mixed groups, split by the visit number?
  3. Tables 2-3 combine your HFC results with baseline numbers copied from Le et al. (2026) and Kim et al. (2026). Please confirm that your evaluation matches the source protocols exactly: responses per problem, decoding settings, and answer extraction.
  4. How is the policy loss aggregated in the main runs, per-sequence or token-level? Given that homogeneous groups receive a const uniform advantage, per-sequence normalization could cause a length bias. Please report response length change during training split by all correct / mixed / all wrong, compared with GRPO and RL-ZVP.
Reviewer_F6h5OpenReview β†—
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

When a group of rollouts get the same reward, there is 0 advantage. This paper tries to solve this by using historical rewards as a baseline. The method shows improvements over GRPO and recent baselines on math reasoning benchmarks and on a multi-turn search agent setting.

Critical strengths

The method is simple and intuitive. And easily applied to exiting GRPO pipeline.

Critical weaknesses

[W1] Supporting experimental results are quite weak. In Table 2, the improvement over the baselines seems quite marginal. In addition, most of the improvement comes from just giving a non-zero advantage to same reward groups, not from the historical forecast.

[W2] The paper is very hard to understand. I guess the essence of this paper is quite simple, but it is written much harder that what actually it is. I understand the authors’ efforts in theoretical explanations, but I believe it would be better to consider moving them in the Appendix.

Questions for authors

How did the authors calculate std of RL-ZVP in Table 1? I am wondering if they actualy run 5 times with different seed and calculate the std. After the calculations, the mean of the numbers of RL-ZVP in Table 2 (which are single-seed results) are exactly same as the mean reported in Table 1 (i.e., 49.94 and 65.29).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.