acceptodds
Under review as a conference paper at ICLR 2027

Self-Hinting Language Models Enhance Reinforcement Learning

Abstract

Group Relative Policy Optimization (GRPO) often stalls on hard prompts under sparse terminal rewards: identical rewards within a rollout group collapse relative advantages and eliminate updates. We propose self-hint aligned GRPO with privileged supervision (SAGE), an on-policy method that uses training-only hints compressed from reference solutions to reshape the rollout distribution. For each prompt , the model samples a compact hint and generates a solution conditioned on , while retaining the same verifier reward . At each training step, the scheduler starts without a hint and increases hint strength only if a rollout group has no correct response. This makes informative groups more likely under finite sampling and can restore learning signals on hard prompts. At test time, we deploy the no-hint policy with . By refreshing self-hints as the learner changes, SAGE provides a better-calibrated curriculum than fixed hints from an initial policy or an external model. Across six benchmarks and three backbones, SAGE consistently outperforms GRPO, gaining up to 4.0 percentage points in average accuracy when trained on hard prompts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, hint, LLM.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Sza2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to sovle a main challenge of RL that rollout quality collapse when given hard prompts. The paper claims that exsiting methods solve this by resampling or skip such prompts, which skew its distribution or introduce context mismatch. To close this gap, this paper propose SAGE, in which the policy model generate tiwce: (1) first generate a hint given a hard prompt and its reference solution, (2) then the policy model rollouts given the generated hint and the hard prompt. The following process of rewarding and policy updating are based on the second generation. During the test time, the hints are removed from the policy models' context but just given prompts. The experiments show that SAGE achieves the best average among all baselines (GRPO, LUFFY, Scaf-GRPO, SFT) with three LLMs (Llama-3.2-3B, Qwen2.5-7B, Qwen3-4B) on six mathematical benchmark.

Critical strengths
  1. Simple and practical method: The rollout quality collapse is an important challenge in RL. This paper aims solve this with only adding self-generated hints, keeping reward strategies unchanged. So it is easy to plug into existing training pipelines.

  2. Relatively extensive experiments: The paper conducts extensive analytical experiments, including main comparisons with existing methods on both easy and hard training sets across three LLMs, ablations of the key design choices, and an analysis of training dynamics. SAGE shows relatively consistent advantages over the baselines cross these experiments.

Critical weaknesses
  1. The motivation is somewhat incremental, and the comparison with existing methods is not well supported. The paper claims that existing methods "skew the training distribution or introduce context mismatch", but SAGE does not appear to solve the context mismatch it criticizes. For example, compared with Scaf-GRPO, which already triggers hints only on collapsed groups and samples hinted rollouts on-policy, SAGE mainly introduces online self-generated hints. The context mismatch seemly remains: training is conditioned on the prompt plus the hint, while at test time the model sees only the prompt. In fact, since this paper discards more no-hint rollouts, the mismatch might be larger. So possibly the main gain lies self-generated hints provide a better-calibrated curriculum during training.
  2. The context mismatch may need direct analysis and evidence. Figure 7 shows that the number of prompts failing without hints decreases during training, but it does not tell where the improvement comes from. Since batches are randomly sampled, the improvement might come from the model's general improvement through RL on non-hinted prompts, rather than from the hinted updates. A GRPO counterpart of Figure 7, or tracking the prompts previously trained with hints, may help.
Questions for authors
  1. Could the authors clarify more about the motivation and the gain over existing methods, like Scaf-GRPO?
  2. Could the authors add a direct analysis of the context mismatch?
Reviewer_WGULOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose SAGE (Self-hint Aligned GRPO with Privileged Supervision), which shifts the sampling distribution via training-time hints so that difficult problems more often produce mixed answer groups, some correct and some wrong, thus restoring GRPO's learning signal.

Critical strengths
  • Using reference solutions as hints lets the model attempt the solution itself rather than simply copy a complete answer. This gives GRPO a practical way to obtain learning signals on problems that would otherwise produce only incorrect responses.

  • The experiments cover multiple models and benchmarks, with consistent gains. Comparing online hints against multiple offline hints helps address the role of hint diversity, while removing hints during updates tests the importance of keeping sampling and training contexts consistent.

Critical weaknesses
  • SAGE takes about 1.5 times as long to train as GRPO, mainly because it samples additional responses on failed problems. Comparisons at equal training steps cannot separate the benefit of hints from that of extra sampling. Missing controls include GRPO with larger sampling groups and additional unassisted attempts on all-incorrect problems under the same compute budget. Hint generation and unsuccessful resampling also need to be included in the cost comparison.

  • The ablations omit offline hints with dynamic scheduling, making it hard to separate the contributions of online updating and scheduling. They also do not directly show whether online hints produce more mixed correct–incorrect groups or bring success rates closer to the theoretically useful range. The stopping rule accepts any group with at least one correct response, including all-correct groups with zero group-relative advantages. The results therefore support the method’s effectiveness more clearly than the proposed calibration mechanism.

  • Since hints are generated from reference solutions, they may supply answer-specific information rather than just useful guidance. The experiments do not systematically compare different amounts of disclosed information or audit whether successful responses still require valid independent reasoning. The existing error analysis also does not show how incorrect hints affect final training outcomes. Better hint-free test scores are encouraging, but they do not by themselves rule out shortcuts during training.

Questions for authors
  1. Could the authors compare SAGE with GRPO using larger sampling groups and extra unassisted sampling on all-incorrect problems under matched token and GPU-time budgets? Please include hint generation and failed resampling in the accounting, and report hint-free accuracy against cumulative compute rather than only training steps.

  2. Could the authors complete the offline/online × fixed/dynamic comparison by adding offline hints with dynamic scheduling under the same setup? Please also compare a few hint-refresh frequencies, such as every step, every several steps, and every epoch, reporting both accuracy and cost to show whether frequent online updating is necessary.

  3. Could the authors report success-rate distributions across hint levels and the proportions of all-incorrect, mixed, and all-correct groups at several training checkpoints for both online and offline hints? It would also help to report how often the stopping rule selects an all-correct group and whether increases in mixed groups correspond to larger gains in hint-free accuracy.

  4. Could the authors compare concept-only hints, intermediate-step hints, near-complete solutions, direct answers, and irrelevant length-matched hints on the same problems? Alongside final hint-free accuracy, an audit of sampled hints and successful responses for answer leakage and reasoning validity would help clarify what information drives the gains.

Reviewer_UvGrOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAGE, a hint-augmented GRPO framework for difficult problems that produce zero-reward groups. During training, the current policy generates progressive hints from reference solutions. Sampling starts without hints, and hint strength increases when all responses are incorrect. Policy updates retain the same hinted context used for generation, while evaluation uses the original problem without hints. Experiments cover three backbones, multiple mathematical benchmarks, and additional generalization tasks.

Critical strengths
  1. The paper addresses a practical limitation of GRPO: zero-reward groups provide no gradient signal for the corresponding problem.

  2. The approach is straightforward and requires no hints at inference time.

  3. The improvements are consistent across the three backbones. The additional baselines, offline multi-hint comparison, and partial multi-seed experiments in the appendix strengthen the empirical evidence.

Critical weaknesses

Major Concerns

The contribution is limited. Several components have been explored in prior works. GHPO [1] already identifies zero-reward groups and introduces reference guidance, progressively increases guidance after failure, and conditions its policy ratio on the augmented context. HINT [2], Scaf-GRPO [3], HiPO [4], and Critique-GRPO [5] likewise also perform an initial unhinted rollout followed by hint-assisted resampling.


[1] GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning, 2025.

[2] Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts, 2025.

[3] Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning, 2025.

[4] HiPO: Self-Hint Policy Optimization for RLVR, ICLR 2026.

[5] Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback, ICML 2026.

Questions for authors

How are the hints provided by the external GPT-5 teacher generated, online or offline? In Figure 3, why do GPT-5 hints underperform online self-hints? Additionally, why does the performance of online self-hints decrease from level 2 to level 3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.