acceptodds
Withdrawn from ICLR 2027

Learning from Correctness Trends: Sample-Wise Historical Advantage Shaping for Mathematical Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning is becoming an important approach for improving the reasoning ability of large language models. Group Relative Policy Optimization (GRPO) avoids the training cost of an additional value network through group-wise relative rewards and has been widely used for reasoning model training. However,the standard GRPO constructs its optimization signal primarily from the current candidate group and does not retain how the same training sample has evolved across previous visits. Consequently, candidate groups with similar current relative rewards receive similar update strengths even when their sample-wise learning trajectories differ. In this paper, we propose TrendGRPO, a sample-wise historical advantage shaping method that enhances within-group comparison with cross-iteration information. TrendGRPO maintains a correctness history for each sample across repeated visits and uses the deviation between the current group accuracy and historical accuracy to asymmetrically reshape positive and negative normalized advantages. We evaluate TrendGRPO on multiple base models and mathematical reasoning benchmarks. Ablations and training diagnostics support historical sample-wise conditioning in advantage shaping.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Math Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JN7eOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TrendGRPO, a sample-wise, history-conditioned advantage shaping method for GRPO in mathematical reasoning. The motivation is that standard GRPO builds advantages only from the current rollout group. Two prompts with identical current reward patterns therefore receive identical updates, even if one is improving across repeated visits and the other is deteriorating. TrendGRPO keeps a persistent correctness history for each training sample and computes the deviation between current group accuracy and historical accuracy. After GRPO normalization, it uses this deviation to rescale positive and negative advantages asymmetrically, together with a warmup coefficient, a difficulty weight and an auxiliary failure-streak term. The method also adds a numerical-proximity reward for incorrect answers. The authors train Qwen3-0.6B, Qwen3-8B and OLMo-3-7B-Instruct on a 2,048-sample subset of DAPO-Math-17K. They report average gains of 0.7–1.9 points over GRPO across seven math benchmarks, supported by history-correspondence ablations, component ablations and training diagnostics.

Critical strengths
  1. The research motivation is clear, and TrendGRPO can be integrated into existing GRPO training pipelines with only minor implementation changes.

  2. The paper clearly distinguishes between maintaining correctness statistics and replaying historical trajectories, providing a useful basis for understanding the positioning of the proposed method.

  3. The controlled experiments cover three models, including 7B and 8B models from two model families, and demonstrate improvements on most of the evaluated benchmarks.

Critical weaknesses
  1. Insufficient comparison with the most closely related methods; missing baselines. Section 2 discusses GRPO-LEAD and HA-DW, but the controlled comparisons are essentially GRPO versus the paper's own internal variants. There is no comparison, under the same reward and training budget, against representative difficulty-aware or history-based advantage reweighting baselines, so it is hard to judge what the proposed design actually gains over existing conditioning strategies.

  2. The training-cost evidence is useful but incomplete. The relative costs of 1.05, 0.971 and 0.966 suggest that the additional end-to-end overhead may be limited. However, the paper does not report the GPU model, the actual wall-clock units, the run-to-run timing variation, or the number of generated tokens.

  3. Validation is concentrated in a training regime tailored to repeated sample visits. All experiments train for six epochs on the same 2,048-sample subset, and the key ablations are conducted only on Qwen3-0.6B.

  4. The existing ablations do not adequately separate temporal information from sample difficulty. Shuffling the historical state across samples breaks not only the sample–history correspondence but also the link between a sample and its historical difficulty. The resulting performance drop therefore supports using statistics tied to the current sample, but it does not by itself establish that the signed current-to-historical deviation is the decisive factor. Adding controls that use only historical difficulty and only current group accuracy would make this much clearer.

  5. Some presentation choices weaken the clarity of the core evidence. Table 1 places externally reported results alongside the controlled experiments, and some of the averages are computed over different subsets of available benchmarks.

Questions for authors
  1. Under matched training conditions, how does TrendGRPO perform relative to a purely difficulty-based reweighting baseline, and to at least one of the related methods discussed in Section 2?

  2. What are the exact decoding, answer-verification, validation-set and checkpoint-selection settings used for Table 1?

  3. Could you provide multi-seed results for the main comparisons?

  4. Under a fixed generation budget, how do the gains of the method change when the number of distinct training problems is increased and the number of training epochs is reduced?

Reviewer_GtsFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TrendGRPO, a modification of GRPO that uses each training example’s correctness history to shape its normalized advantages. The method compares current group accuracy with cumulative historical accuracy, strengthening positive advantages when performance exceeds the historical reference and negative advantages when it falls below it. It also incorporates history warmup, difficulty weighting, a numerical proximity reward, and an auxiliary failure-streak correction.

Experiments train three models on a fixed 2,048-example subset of DAPO-Math-17K for six epochs and evaluate on seven mathematical reasoning benchmarks. Reported average improvements over GRPO are 1.58, 0.71, and 1.88 percentage points. Ablations investigate sample-history correspondence, asymmetric shaping, and individual components. Appendix A.6 characterizes the resulting reweighted optimization objective.

Critical strengths
  1. The method provides a simple, inexpensive way to use persistent sample information. Maintaining correctness counts and a failure counter requires little additional state and avoids an additional learned model or stored trajectories. Applying the transformation after normalization also makes the intervention easy to identify and integrate. The reported training costs are close to those of GRPO, which makes even moderate improvements potentially useful.

  2. The empirical study includes meaningful attempts to test the proposed mechanism. Table 1 reports improvements on six of seven benchmarks for each backbone, spanning two model families. Table 2 goes beyond component removal by testing global history, shuffled history, and symmetric shaping. These controls provide preliminary support for the importance of matching historical information to the correct example. I regard this as the strongest evidence supporting the contribution, although the interpretation needs further qualification.

Critical weaknesses
  1. The experiments do not sufficiently isolate the benefit of temporal trends from other uses of historical information. In the supplied implementation, replacing a sample’s history with global or shuffled history changes both the deviation gig_i and the difficulty weight did_i. Consequently, Table 2 establishes that sample-specific conditioning helps, but does not isolate whether the benefit comes from tracking improvement or from estimating example difficulty. Moreover, gig_i compares one eight-response group with a cumulative reference, so its sign can reflect sampling noise as well as policy change. A historical-difficulty-only control would help resolve this distinction. A matched comparison with the already cited HA-DW method, which also uses historical information for advantage reweighting, is also missing. The Global History ablation is not equivalent to that method. These omissions leave the incremental contribution insufficiently established.

  2. The reported improvements need stronger evidence of robustness and a more complete evaluation specification. The average gains are modest, particularly the 0.71-point improvement for OLMo-3-7B, and no variability across independent training runs is reported. All experiments use one training subset, while the mechanism ablations are restricted to Qwen3-0.6B and three benchmarks. This does not invalidate the results, but makes it difficult to assess how reliably the gains would recur. Evaluation decoding parameters, the number of responses per test problem, answer-verification details, and the checkpoint-selection rule are also not sufficiently documented. The supplementary package contains useful implementation code, but lacks the evaluation scripts and experiment configurations needed to resolve these uncertainties.

Questions for authors
  1. Does the GRPO baseline in Tables 1–3 use exactly the same numerical proximity reward with α=0.2\alpha=0.2? Appendix A.2 suggests that only advantage shaping differs. Please make the reward settings explicit for the baseline and each ablation, since they determine how the improvements should be attributed.

  2. What decoding settings, response counts, and answer verifier were used for each benchmark? Were results obtained from the final checkpoint or a checkpoint selected using validation performance, and which data were used to select checkpoints and λ\lambda?

  3. Does TrendGRPO retain its advantage over a control using sample-specific historical difficulty without the signed current-to-history deviation? A comparison that holds the difficulty weights fixed while changing the trend signal would directly address the confounding in Table 2. A matched HA-DW comparison would further clarify the contribution.

  4. Can the authors report results across independent training seeds for a representative backbone and the principal historical-conditioning ablations? Whether the full method consistently outperforms GRPO and symmetric shaping would materially affect my assessment.

Reviewer_FoKJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TrendGRPO, which uses each training sample's correctness history to adjust GRPO updates. It compares the current group's accuracy with the sample's historical accuracy and scales positive and negative advantages separately. The method also includes a numerical proximity reward and a correction for repeated failures. Experiments on three base models and seven mathematical reasoning benchmarks report higher average accuracy than GRPO.

Critical strengths
  1. This paper presents a clear and lightweight way to incorporate sample-specific learning history into GRPO.

  2. This paper includes useful ablations of its central design. Table 2 compares sample-specific history with global history, shuffled history, and symmetric advantage scaling.

Critical weaknesses
  1. The advantage over existing adaptive GRPO methods is not sufficiently established. The main controlled comparison is against standard GRPO, while Section 2.2 discusses related approaches such as HA-DW and GRPO-LEAD that also adapt advantage weights using historical information or problem difficulty. The history ablations help explain the proposed design, but do not establish its benefit over these alternatives.

  2. The scope of the improvements remain uncertain. Table 1 reports average gains of 1.58, 0.71, and 1.88 percentage points for Qwen3-0.6B, OLMo-3-7B-Instruct, and Qwen3-8B, respectively, without results across multiple training seeds or uncertainty estimates. These gains are encouraging, but their reliability is difficult to judge.

  3. All experiments also use 2,048 training examples over six epochs. Since the historical signal depends on revisiting samples, this setting is particularly relevant to the method. The paper does not establish whether the benefits persist when the same rollout budget covers more distinct problems with fewer repeated visits. These limitations reduce my confidence in the broader practical benefit.

Questions for authors
  1. How does TrendGRPO compare with related methods such as HA-DW or GRPO-LEAD under the same reward, training data, and rollout budget?

  2. Are the gains over GRPO consistent across independent training runs? What is the variability of the reported accuracy differences?

  3. How do the gains vary with the number of visits to each training sample? Do they persist when the same rollout budget is used for more distinct prompts and fewer repeated visits?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.