acceptodds
Under review as a conference paper at ICLR 2027

Teach to Learn: Hint Annealing for Self-improving LLM Reasoning

Abstract

Group Relative Policy Optimization (GRPO) improves language-model reasoning by comparing verified rewards among multiple solution rollouts for each query. However, difficult training queries can yield only incorrect rollouts, leaving GRPO with no reward contrast or learning signal. Prior hint-based methods construct auxiliary hints from solution evidence and use them to re-solve failed queries, recovering learning signal. Yet the resulting trajectories are typically treated as ordinary solution trajectories despite being generated under an assisted condition unavailable at evaluation. We discover *hinted reward shift*: recovered reward contrast can concentrate policy updates on hinted trajectories, limiting improvement without hints. This also creates a trade-off: increasing hinted trajectories can accelerate early learning but intensify reward shift later. To address this problem, we propose *HATCH (Hint-Annealed Self-Teaching)*, an online single-policy framework that learns from both generating and using its own hints to improve reasoning without assistance. To mitigate hinted reward shift, we introduce online weighting to anneal the contribution of hinted trajectories. However, learning to generate hints can conflict with improving query solving. We therefore use gradient projection to remove the opposing component of hint-generation updates. Together, these designs support self-improvement by enabling the policy to create learning opportunities for itself and turn them into stronger reasoning without hints. We evaluate our method on mathematical reasoning benchmarks and outperform state-of-the-art methods by 1.02 pp on Llama-3.2-1B-Instruct, 2.84 pp on Qwen3-1.7B, and 4.32 pp on Qwen3-8B.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KAy2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the lack of group-relative learning signals when all sampled responses to a difficult problem are incorrect. The authors identify “hinted reward shift,” where sustained optimization of hint-conditioned trajectories may limit improvement in unassisted reasoning. They propose HATCH, a single-policy framework that jointly trains original solving, reference-conditioned hint generation, and hinted re-solving. HATCH adaptively reduces the contribution of hinted-solve advantages and projects out components of hint-generation gradients that oppose the combined solution-side gradient. Experiments on three backbones and nine mathematical benchmarks show improvements over GRPO, QuESTA, and HiLL.

Critical strengths
  1. The paper provides useful diagnostics of hint-assisted training. The advantage statistics, gradient-angle measurements, and experiments with different hint counts illustrate a potential trade-off between early learning gains and later unassisted performance. These observations motivate studying how auxiliary learning signals should be regulated.

  2. HATCH achieves higher average no-hint accuracy than the evaluated baselines across all three backbones. The Online Hints ablation also provides evidence that explicitly optimizing hint generation contributes to performance. Removing this objective reduces Qwen3-8B accuracy from 53.76% to 50.16%, although the limited number of ablation runs warrants caution.

Critical weaknesses
  1. The comparisons do not sufficiently establish the contribution relative to closely related methods. SAGE [1] and NuRL [3] are cited, but neither is evaluated directly. SAGE is particularly relevant because it generates reference-conditioned hints using the current policy and adapts hint strength. HATCH should distinguish its explicit hint-generation objective and loss weighting from these mechanisms. HiLL [2] is already included, and HATCH explicitly adopts its transfer-aware hint reward. The additional contribution beyond this reward therefore needs clearer isolation. Guide-GRPO [5] and BHA [6] are also relevant to the central motivation because they address learning across guided and unguided conditions through different mechanisms. GHPO [4] and HDPO [7] would provide further context. Direct comparisons with the closest alternatives, supported by faithful implementations and comparable budgets, would strengthen the claimed advance.

  2. The ablations do not cleanly isolate the sources of improvement. Appendix C.1 states that Offline Hints uses one hint per query, whereas HATCH uses four. This comparison changes both the hint source and the amount of assisted training. Furthermore, each ablation uses a single training run, while HATCH reports a three-run average. The Online Hints comparison is more informative, but repeated runs are needed to assess its reliability. The fixed-weight result of 44.96% demonstrates that keeping β=1 performs poorly in this configuration. It does not establish that adaptive annealing is superior to a well-tuned constant weight. Comparisons with several constant weights and matched hint counts would better separate the effects of hint generation, weighting, and projection.

  3. The evidence for the proposed mechanism is suggestive rather than conclusive. Larger mean absolute advantages do not directly establish larger gradient contributions, which also depend on trajectory counts, token weighting, and policy derivatives. Likewise, nearly orthogonal gradients do not by themselves demonstrate harmful interference. Increasing the number of hints also changes the amount and composition of training. The observed degradation is therefore consistent with the proposed explanation, but does not uniquely identify it. Scalar weighting can reduce the influence of hinted updates without converting them into updates of the no-hint objective. Similarly, projection uses a reference gradient that still includes hinted solving. The paper should clarify these limits and provide stronger evidence linking the measured dynamics to improved unassisted performance.

  4. The scope of the self-improvement claim needs clarification. HATCH generates hints conditioned on ground-truth answers and reference solutions, and relies on a verifier for outcome feedback. Its learning signals therefore depend on externally supplied supervision, even though one policy performs both hint generation and solving. The experiments demonstrate improved unassisted performance after reference-supervised training. The paper should clarify that “self” refers to the shared policy and its jointly trained roles, rather than independence from external solution information. Without this distinction, the framing risks overstating the autonomy of the learning process.

[1] Self-Hinting Language Models Enhance Reinforcement Learning

[2] Learning to Hint for Reinforcement Learning

[3] Nudging the Boundaries of LLM Reasoning

[4] GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning

[5] Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models

[6] Mitigating Distribution Sharpening in Math RLVR via Distribution-Aligned Hint Synthesis and Backward Hint Annealing

[7] HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation

Questions for authors
  1. Appendix A.2 describes the handling of thinking content during hint extraction. Please specify the thinking settings for original solving, hint generation, hinted re-solving, and evaluation. What fraction of responses reaches the 8,192-token limit in each setting?

  2. Appendix A.3 describes the leakage detector, and Appendix D.2 reports its flagged-hint rate. Have you assessed false negatives through manual inspection, particularly for paraphrased answers or revealing intermediate computations? Are flagged hints and their associated re-solves excluded from policy updates?

  3. Table 3 reports model-specific values of κ and γ, and Appendix C.2 describes validation-based selection. What search ranges and tuning budgets were used? How sensitive are results to κ and the EMA update coefficient, and how well do these settings transfer across backbones?

  4. What are the realized β values at the selected checkpoints for each model? Could you disable hinted-solve updates at different training stages to distinguish their early contribution from their continuing contribution later in training?

  5. Figure 5 provides a GPU-hour comparison with HiLL, and Appendix E reports component timings. Could you also report end-to-end cost and accuracy under matched compute budgets against GRPO? Please clarify the hinter initialization and training configuration used in the HiLL implementation.

  6. Do the gains extend beyond pass@1 to larger values of k? A modest pass@k evaluation would help assess whether HATCH improves solution coverage as well as single-sample accuracy.

  7. Can you conduct experiments using more frontier LLMs, such as Qwen3.5, MiniCPM5?

10 comments

Thank you for the detailed review and for recognizing the value of our diagnostics and improvements in unassisted reasoning. We clarify our contributions and provide additional baseline comparisons, controlled ablations, and broader evaluations to address your concerns.

1. Clarification of contribution and distinctions from related methods (Weakness 1)

We focus on hinted reward shift, which prior hint-based methods do not explicitly address. SAGE and NuRL explore self-generated hints but do not examine this shift. HiLL, Guide-GRPO, BHA, and GHPO seek to make hint construction more effective, without explicitly addressing how sustained hinted feedback can shift optimization toward assisted solving. HDPO instead combines GRPO with privileged self-distillation, addressing a different research problem. We have added the relevant citations and discussion to the paper (lines 126–138).

We additionally evaluate the two closest methods SAGE and NuRL on Qwen3-8B using the same setting. The table reports macro-average accuracy over the nine benchmarks, averaged over three independent runs.

BackboneGRPO (%)HiLL (%)QuESTA (%)SAGE (%)NuRL (%)HATCH (%)
Qwen3-8B47.5849.4449.0749.2248.0553.76

HATCH exceeds SAGE and NuRL by 4.54 and 5.71 percentage points, respectively. These results complement the methodological distinction with direct comparisons to online and offline self-hinting; the controlled ablations below further examine which aspects of HATCH contribute to its gains.

2. Further multi-seed ablation and computational fairness (Weakness 2 and Question 5)

We provide further multi-seed ablations that support our design. The two tables below report Qwen3-8B results averaged over three independent training seeds, using the same setting as the main experiments.

Additional rollouts and matched hint counts. We increase GRPO's original rollouts from 8 to 32, extend HiLL to retain four hinted-solve groups, and compare against four fixed external hints per query. Each retained hint has eight re-solves.

Method / variantOriginal rollouts per queryRetained hint groups per failed queryAccuracy (%)
GRPO8047.58
GRPO, increased rollouts32048.86
HiLL, four-hint variant8447.23
External Hints, Qwen3-30B-A3B8448.38
HATCH8453.76

Neither increasing GRPO rollouts nor matching HATCH’s four-hint setting with HiLL or external hints closes the performance gap. We have also added GRPO to Figure 5 (middle), alongside HiLL and HATCH, to compare accuracy against the same end-to-end cumulative GPU-hour axis. At a matched budget, HATCH achieves a best-so-far no-hint accuracy of 51.2%, exceeding HiLL’s 50.3% and GRPO’s 47.5%. Together with the larger-rollout and matched-hint-count controls above, this provides complementary evidence that the observed gains are not explained solely by additional trajectories.

Constant weights and predefined schedules. We also provide ablations on different weights under the same self-teaching framework from S5.3 (line415-464) and Appendix C.1 (line846-860). The results support our design of adaptive annealing. A carefully chosen time schedule may work well for a particular training setting, but it prescribes the same weight at a given step regardless of the policy's learning state. HATCH instead uses the balance between solve-none and solve-partial groups to adjust assistance as training evolves.

Weighting strategyWeight / scheduleAccuracy (%)
Fixed weightingβ=1\beta=145.66
Fixed weightingβ=0.5\beta=0.547.09
Linear decayFrom 1 to 0 over steps 1--30047.97
Staged exponential decayStarts at 1 and halves every 25 steps50.99
HATCHData-driven online weighting53.76

For HiLL settings, hint generation and reasoning use Qwen3-8B initialization. The common learning rate, optimizer, decoding parameters, and length limits are specified in Appendix A.2. The main comparison uses best-hint selection; the additional four-hint variant retains four hinted-solve groups, with eight re-solves per hint.

3. Mechanism validation and gradient interference (Weakness 3)

Advantage statistics and gradient contributions. Mean absolute advantage does not directly measure total gradient contribution. We therefore complement it with token-averaged gradient comparisons on the same questions at the same model state, sampling eight responses per condition and retaining questions with mixed outcomes under both conditions.

ModelMean cosineCosine rangeAngle from mean cosine (degrees)
Qwen3-8B0.0594780.01705--0.105986.59

Directional differences persist after controlling question identity, motivating us to treat hinted solving as a distinct auxiliary objective and regulate its contribution through online weighting. This weighting scales the auxiliary gradients rather than converting them into gradients of the unassisted objective.

Directional differences and harmful interference. The benefit of gradient correction is supported by controlled ablations from S5.3 (line417-428) and Appendix C.1 (line847-859), not inferred from near-orthogonality. The diagnostic compares original and hinted solving, whereas projection removes only hint-generation components opposing the combined solution-side gradient. We share PCGrad’s conflict-removal rule, but investigate role-specific correction of auxiliary-learning conflicts in self-teaching. We have added the citation and discussion. The below ablations show that hint-generation learning improves over Online Hints, our projection yields further gains, and reverse projection and an original-only reference perform worse.

Qwen3-8B variantAccuracy (%)
Online Hints, no hint-generation learning50.06
Hint-generation learning without projection51.39
Projection with original-only reference51.03
Reverse projection49.15
HATCH, combined solution-side reference53.76

4. Clarification of self improvement (Weakness 4)

In our work, we study self-improvement by training a single policy to learn from both solving and teaching based on external information. Existing works in other areas such as SISL [1] and PLD [2] also explore self-improvement while relying on external solution or verifier information. In HATCH, solving outcomes improve both solution behavior and hint generation, closing a feedback loop between learning and teaching.

[1] Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learning, ICLR 2026 [2] Self-Improving Vision-Language-Action Models with Data Generation via Residual RL, ICLR 2026

5. Thinking settings and generation limits (Question 1) For Qwen3-1.7B and Qwen3-8B, thinking is enabled in original solving, hint generation, hinted re-solving, and evaluation. Llama-3.2-1B-Instruct does not use this thinking mode. All compared methods use the same settings within each backbone, with a maximum prompt length of 4,096 tokens and a generation limit of 8,192 tokens. For hint injection, the thinking block is stripped from the generated text; this extraction is distinct from disabling thinking during generation. These have been stated in Appendix A.2 (line720-737). In the reported training statistics, HATCH's average at-limit response rate is approximately 40%, compared with approximately 50% for the baselines. These aggregate rates show that its accuracy gains are obtained under the same response-length constraint, with fewer responses reaching the cap.

6. Leakage detection and treatment (Question 2)

The detector checks for the reference answer and answer-revealing patterns, as described in Appendix A.3. Its flagged-hint rate fluctuates around 5%. We also manually inspect 100 sampled hints and observe almost no missed answer leakage. This provides a complementary check for cases that literal matching may miss, although it does not establish a zero false-negative rate. Flagged hints receive a hint-generation reward of −0.2-0.2 and re-solves retain correctness-based rewards.

7. Parameter selection and sensitivity (Question 3)

The parameters control different aspects of the schedule: κ\kappa sets its overall scale, while γ\gamma controls how strongly the weight responds to changes in rollout composition. We select configurations using the held-out 200-problem validation set. The main configurations are (κ,γ)=(1.2,4)(\kappa,\gamma)=(1.2,4) for Llama-3.2-1B-Instruct, (65,10)(65,10) for Qwen3-1.7B, and (5,5)(5,5) for Qwen3-8B. To examine local sensitivity, we vary γ\gamma while fixing κ=5\kappa=5, and vary κ\kappa while fixing γ=5\gamma=5, on Qwen3-8B. Accuracy is the three-run mean of the macro average over the same nine no-hint benchmarks.

Sensitivity comparisonκ\kappaγ\gammaAccuracy (%)
Fixed κ\kappa5449.93
Fixed κ\kappa, main configuration5553.76
Fixed κ\kappa5650.56
Fixed γ\gamma, lower κ\kappa4550.49
Fixed γ\gamma, main configuration5553.76
Fixed γ\gamma, higher κ\kappa6544.65

The EMA update coefficient is 0.1 in the main configuration. In our additional checks, increasing this coefficient above 0.1 produces small accuracy variations over the tested settings. The method is evaluated across backbones with model-specific κ\kappa and γ\gamma; this supports applicability across models, rather than transfer of one universal parameter pair without retuning.

8. Realized weights and stage-specific removal of hinted-solving updates (Question 4)

The mean realized β\beta at the selected checkpoints across three runs is 0.705 for Llama-3.2-1B-Instruct, 0.206 for Qwen3-1.7B, and 0.259 for Qwen3-8B. We set β=0\beta=0 after selected cutoff steps while retaining hint generation and re-solving for hint rewards. Results average three runs, with the no-cutoff baseline taken from the main HATCH experiment. These results support a continuing contribution from hinted-solving feedback, rather than attributing all benefits to early assisted exploration.

Qwen3-8B continuationHinted-solve cutoff stepNo-hint accuracy (%)
Continue data-driven online weightingNo cutoff53.76
Earlier cutoff5049.13
Later cutoff15052.16

9. Pass@k results (Question 6)

We extend evaluation to pass@16 and pass@32. Results are averaged equally across the same nine benchmarks. HATCH exceeds the strongest compared baseline, HiLL, by 2.54 pp at pass@16 and 2.19 pp at pass@32.

Methodpass@1 (%)pass@16 (%)pass@32 (%)
GRPO47.5863.7366.82
HiLL49.4469.2772.84
QuESTA49.0768.5170.93
HATCH53.7671.8175.03

10. Evaluation on additional backbones (Question 7)

HATCH outperforms GRPO and HiLL on both Qwen3.5-2B and MiniCPM5-1B, extending the comparison to other recent backbones. The table reports average accuracy over the same nine mathematical benchmarks. On Qwen3.5-2B, HATCH improves over GRPO and HiLL by 5.52 and 3.55 pp, respectively; the corresponding gains on MiniCPM5-1B are 1.72 and 1.51 pp.

MethodQwen3.5-2B (%)MiniCPM5-1B (%)
GRPO36.0323.89
HiLL38.0024.11
HATCH41.5425.62

All the additional results have been added to the paper.

Reviewer_Thu1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HATCH to address hinted reward shift in hint-based GRPO, where learning from hinted trajectories may not improve unassisted reasoning. The method jointly trains hint generation and problem solving within a single policy, using online hint annealing and gradient projection to regulate auxiliary learning signals.

Experiments on three language models and nine mathematical reasoning benchmarks show improvements of 1.02, 2.84, and 4.32 percentage points over the strongest baselines. Ablation studies and training analyses further support the effectiveness of joint hint learning, adaptive weighting, and gradient projection in improving no-hint reasoning performance.

Critical strengths
  1. Joint hint generation and solving. HATCH trains a single policy to generate hints and solve problems, allowing hint-generation rewards to directly improve the solver.
  2. Useful empirical insights. The paper identifies hinted reward shift through analyses of reward advantages, gradient directions, and training dynamics.
  3. Consistent improvements. HATCH outperforms GRPO, QuESTA, and HiLL across three model scales and nine mathematical reasoning benchmarks.
Critical weaknesses
  1. Limited generalization. Experiments focus exclusively on mathematical reasoning using one training dataset, with model sizes up to 8B.
  2. Insufficient ablation reliability. Key ablations use only a single training run, and the online weighting hyperparameters differ substantially across models.
  3. Incomplete mechanism validation. The gradient analyses do not fully establish harmful interference, and the choice of projection reference lacks sufficient justification.
  4. Incomplete isolation of performance gains. HATCH retains multiple hinted-solve groups and introduces additional optimization signals. Existing comparisons do not fully disentangle the proposed mechanisms from the benefits of increased rollout and training budgets.
Questions for authors
  1. Computational fairness. Could the authors provide comparisons under matched rollout or training-token budgets, in addition to the existing GPU-hour comparison with HiLL, to clarify how much improvement comes from the proposed mechanisms rather than additional trajectories?
  2. Gradient projection. Eq. 11 resembles one-sided PCGrad. What motivates projecting hint-generation gradients against the combined solution-side gradient rather than the original-solve gradient? Could the authors provide a corresponding ablation and clarify the relationship to PCGrad?
  3. Online weighting design. Eq. 7 excludes solve-all groups, so the hint weight is not guaranteed to decrease as overall no-hint performance improves. For example, the weight may increase when solve-partial groups become solve-all faster than solve-none groups become solve-partial. Could the authors clarify the rationale for this formulation and whether such non-monotonic behavior affects training stability?

If the authors can adequately address my concerns, I would be very willing to reconsider my assessment and raise my score.

6 comments

Thank you for recognizing the value of joint hint learning and our empirical findings. We address your concerns through additional budget comparisons, projection-reference ablations, and an explanation of our state-dependent weighting rule.

1. Evaluation beyond mathematical reasoning (Weakness 1)

We extend evaluation to MMLU and ARC-Challenge, covering broad subject knowledge and science questions. HATCH achieves the highest MMLU and ARC-Challenge accuracy among the compared methods. These evaluations broaden the evidence beyond mathematical tasks.

Qwen3-8B methodMMLU (%)ARC-Challenge (%)
GRPO67.7277.73
HiLL69.1872.44
QuESTA68.0672.87
HATCH70.1382.08

2. Multi-seed ablation and online weighting (Weakness 2)

We expand the key Qwen3-8B ablations to three independent training seeds. The multi-seed ablations support the contributions of learning from hint construction, correcting gradient conflicts, and adapting hinted-solving weights.

Qwen3-8B variantAccuracy (%)
HATCH53.76
External Hints, M=4M=448.38
Online Hints, without hint-generation learning50.06
Fixed weighting, β=1\beta=145.66
Fixed weighting, β=0.5\beta=0.547.09
Linear decay47.97
Staged exponential decay50.99
No Projection51.39
Original-only projection reference51.03
Reverse Projection49.15

3. Choice of projection reference (Weakness 3 and Question 2)

Our ablations from S5.3 (line417-428) and Appendix C.1 (line847-859) below support both gradient correction and the combined solution-side reference. Like PCGrad, we operate at the gradient level, but investigate how to correct auxiliary-learning conflicts in hint-based self-teaching. We have added the corresponding citation and discussion in S4.3 (line 291-301). We add an original-only reference that overlooks recovered hinted-solving feedback when assessing hint-generation conflicts. Incorporating this feedback into the reference improves accuracy from 51.03% to 53.76%, supporting the use of both solution-learning signals.

Projection variantReference directionAccuracy (%)
No ProjectionNone51.39
Original-only ReferenceOriginal solving51.03
Reverse ProjectionOriginal + weighted hinted solving49.15
HATCHOriginal + weighted hinted solving53.76

Both the original-only reference and reverse projection, which retains the opposing component, perform worse than HATCH, supporting our choices of reference direction and retained gradient component.

4. Comparison under fair rollout budget (Weakness 4 and Question 1)

We provide comparisons under fair budget, which supports our design. We increase GRPO's original rollout group from 8 to 32 and compare HATCH with two four-hint controls. The HiLL extension retains four hinted-solve groups rather than only the best group; External Hints uses four fixed hints generated. Results are averaged over three training seeds on Qwen3-8B.

Method / variantOriginal rollouts per queryRetained hint groups per failed queryAccuracy (%)
GRPO8047.58
GRPO, increased rollouts32048.86
HiLL, four-hint variant8447.23
External Four Hints8448.38
HATCH8453.76

Increasing unassisted exploration improves GRPO, but leaves a 4.90-point gap to HATCH. Retaining four hint groups in HiLL or supplying four external hints also does not reproduce HATCH's performance. These comparisons complement the Online Hints and component ablations: more original samples or more assisted groups are not sufficient by themselves to explain the observed gains.

GPU hours comparison. We additionally include GRPO alongside HiLL and HATCH in Figure 5 (middle) (line378), comparing best-so-far no-hint benchmark accuracy against cumulative GPU-hours. GRPO improves quickly at lower budgets but HATCH attains approximately 51.2%, compared with 50.3% for HiLL and 47.5% for GRPO. These tests support HATCH's advantage in the later budget range without claiming faster progress at every budget.

5. Online weighting design and non-monotonic behavior (Question 3)

Our weighting tracks learning-signal availability rather than accuracy or training time, since neither necessarily implies more reward contrast. Therefore, it need not decrease monotonically. Solve-none groups require signal recovery, whereas solve-partial groups already provide original reward contrast. Solve-all groups provide no group-relative contrast and are not the failed queries targeted for assistance, so they contribute to neither side of this comparison.

Comparison with fix weight schedule. We compare our rule with constant weights and predefined monotonic schedules while keeping the remaining framework unchanged as below from S5.3 (line415-464) and Appendix C.1 (line846-860). HATCH outperforms all tested alternatives, supporting state-dependent adjustment of assistance without requiring monotonic decay.

Weighting strategyWeight / scheduleAccuracy (%)
Fixed weightingβ=1\beta=145.66
Fixed weightingβ=0.5\beta=0.547.09
Linear decayFrom 1 to 0 over steps 1--30047.97
Staged exponential decayStarts at 1 and halves every 25 steps50.99
HATCHData-driven online weighting53.76

All the additional results have been added to the paper.

Dear Reviewer Thu1,

Thank you again for your constructive feedback. We would appreciate your thoughts on our additional budget comparisons, projection-reference ablations, and clarification of the state-dependent weighting rule.

As the discussion period approaches its end, please let us know whether our responses address your concerns or if any points require further clarification. We would be happy to provide additional explanations.

Kind regards, Authors

Reviewer_N6ECOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors propose HATCH to improve LLM reasoning on difficult questions where GRPO gets no learning signal because all sampled answers are wrong.

The model generates hints using reference solutions and then tries solving the questions again with these hints. It also learns to generate better hints based on how useful they are.

Since too much training with hints may not improve solving without hints, HATCH uses hint annealing to gradually reduce the contribution of hinted answers.

It also uses gradient projection to prevent hint-generation updates from conflicting with learning to solve the questions.

Experiments on 9 mathematical reasoning benchmarks, using 3 model sizes from the Llama and Qwen families, show improvements of 1.02, 2.84, and 4.32 percentage points over the strongest baselines.

Critical strengths

The problem is well motivated. Training with hints can create a mismatch between training and testing, where the model has to solve questions without hints. The gradient analysis and training curves help explain why reducing the hint contribution is useful.

The results are good across all 3 model sizes and 9 math benchmarks. Comparisons with GRPO, QuESTA, and HiLL, along with component ablations, support the effectiveness of the method.

Critical weaknesses

Limited novelty: HATCH combines ideas related to HiLL (hinted-to-unhinted transfer), QuESTA (reducing guidance during training), and gradient projection with a new online weighting scheme. The projection step is similar to a one-sided version of PCGrad (Yu et al., 2020), which the authors have not cited or discussed. The novelty over these methods should be explained more clearly.

Limited evaluation: Experiments only cover mathematical reasoning with reference solutions. It is unclear how well the method would work in other domains.

Questions for authors

Can the authors verify hinted reward shift while controlling for which questions contribute to the gradients? Original and hinted gradients with nonzero advantages may come from different subsets of questions, which could affect the observed gradient differences.

Can the authors provide multi-seed ablations and a clearer sensitivity study for the hint-annealing hyperparameters?

4 comments

Thank you for recognizing our motivation, diagnostics, and consistent empirical gains. We clarify the methodological contributions and provide controlled same-question gradient comparisons, multi-seed ablations, and hyperparameter sensitivity results.

1. Novelty relative to prior methods (Weakness 1)

HATCH identifies hinted reward shift, which prior hint-based methods do not explicitly address, and introduces online weighting to mitigate it. HiLL and QuESTA focus on constructing effective hints, but do not address how sustained hinted updates can redirect subsequent optimization. This overlooked consequence is the focus of our work.

Prior methods also do not explore the gradient interactions between teaching and solving within a single policy. We investigate these interactions and introduce role-specific gradient projection using PCGrad’s conflict-removal rule to enable the policy to benefit from both learning processes. We have added the PCGrad citation and clarified this relationship (lines 291–301). The below ablations from S5.3 (line417-428) and Appendix C.1 (line847-859) show that this design outperforms no projection, reverse projection, and an original-only reference.

Projection variantReference directionNo-hint accuracy (%)
No ProjectionNone51.39
Original-only ReferenceOriginal solving51.03
Reverse ProjectionOriginal + weighted hinted solving49.15
HATCHOriginal + weighted hinted solving53.76

2. Evaluation beyond mathematical reasoning (Weakness 2)

We extend evaluation to MMLU and ARC-Challenge, covering broad subject knowledge and science questions. HATCH achieves the highest MMLU and ARC-Challenge accuracy among the compared methods. These tests broaden the evidence beyond mathematical tasks without implying uniform gains across domains.

Qwen3-8B methodMMLU (%)ARC-Challenge (%)
GRPO67.7277.73
HiLL69.1872.44
QuESTA68.0672.87
HATCH70.1382.08

3. Original vs. hinted gradients on the same questions (Question 1)

Original and hinted gradients remain nearly orthogonal on the same questions. We add a paired diagnostic on Qwen3-8B that compares original and hinted solving on the same questions. For each question, we sample eight responses without a hint and eight with a generated hint.

ModelMean cosineCosine rangeAngle from mean cosine (degrees)
Qwen3-8B0.0594780.01705-0.105986.59

The two aggregate gradients remain nearly orthogonal after matching the contributing questions. Thus, the observed directional difference cannot be explained solely by original and hinted updates coming from different question subsets. It persists when comparing the two solving conditions on the same questions.

4. Multi-seed ablations and annealing sensitivity (Question 2)

We expand the key ablations to three independent training seeds. The following table reports three-run mean accuracy on Qwen3-8B averaged across the nine benchmarks.

Qwen3-8B variantMean accuracy (%)
HATCH53.76
External Hints, M=4M=448.38
Online Hints50.06
Fixed weighting, β=1\beta=145.66
Fixed weighting, β=0.5\beta=0.547.09
Linear decay47.97
Staged exponential decay50.99
No Projection51.39
Original-only projection reference51.03
Reverse Projection49.15

The repeated comparisons support the separate contributions of the framework.

  • Hint-generation learning. With projection disabled, learning hint generation improves accuracy from 50.06% to 51.39%, supporting the value of learning from teaching.
  • Gradient projection. HATCH outperforms no projection, reverse projection, and an original-only reference, supporting our choices of which gradient component to retain and which reference direction to use.
  • Online weighting. HATCH outperforms the tested constant weights and predefined decay schedules, supporting adaptation of hinted-solving contributions to the policy's learning state.

For sensitivity, we vary one parameter at a time: γ\gamma with κ=5\kappa=5 fixed, and κ\kappa with γ=5\gamma=5 fixed. The following results are also three-run means. The EMA coefficient remains 0.1.

Qwen3-8B sensitivity comparisonκ\kappaγ\gammaAccuracy (%)
Fixed κ\kappa5449.93
Fixed κ\kappa, main configuration5553.76
Fixed κ\kappa5650.56
Fixed γ\gamma, lower κ\kappa4550.49
Fixed γ\gamma, main configuration5553.76
Fixed γ\gamma, higher κ\kappa6544.65
  • κ\kappa controls the overall scale. At a fixed rollout composition, increasing κ\kappa increases the auxiliary coefficient until clipping applies and can keep it at its upper bound for longer. Decreasing κ\kappa reduces assisted-learning strength throughout training.
  • γ\gamma controls dependence on rollout state. For 0<pt<10<p_t<1, increasing γ\gamma lowers ptγp_t^\gamma and increases the relative sensitivity of the weight to changes in ptp_t. When ptp_t falls as original reward contrast becomes available, a larger exponent produces a stronger relative reduction; a smaller exponent retains more assistance at the same state.

The sensitivity results suggest that overly weak assistance can discard useful feedback, while overly strong assistance can sustain hinted reward shift. Effective annealing should therefore match the contribution of hinted learning to the policy’s evolving learning state. All the additional results and multi-seed update results have been added to the paper.

Dear Reviewer N6EC,

Thank you again for your constructive feedback. We would appreciate your thoughts on our response, particularly the same-question gradient diagnostic, multi-seed ablations, and annealing sensitivity analysis provided to address your questions.

As the discussion period approaches its end, please let us know if any concerns remain unresolved. We would be happy to provide further clarification and would appreciate your consideration of the updated evidence.

Kind regards, Authors

Dear Reviewer N6EC,

With only a few hours left in the discussion period, we would like to kindly follow up on whether our responses and additional experiments have addressed your questions.

Please let us know if anything remains unclear. We would be happy to provide further clarification before the discussion ends.

Kind regards, Authors

Comments on the paper

1 comment

Dear all, The discussion phase will end soon. The authors have provided their responses to your comments. If the provided responses do not address (even in part) your concerns, this is your chance to raise them. Please engage with the authors timely. We really appreciate your efforts! Thank you to those reviewer(s) who have responded! Thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.