acceptodds
Under review as a conference paper at ICLR 2027

If You Can't Beat It, Copy It: Hint Reliance and Learning Signals in RLVR

Abstract

Reinforcement learning with verifiable rewards (RLVR) checks only the final answer. When a prompt contains a candidate solution, a policy can earn the same reward by solving the problem or by copying the candidate. We study which route RLVR strengthens, using Countdown problems with hints of controlled reliability. Checkpoints trained with always-correct hints reach near-perfect correct-hint accuracy yet range from 0.00 to 0.60 in wrong-hint accuracy. In Countdown with Qwen2.5-7B and Llama-3.1-8B-Instruct and in an arithmetic task with Qwen2.5-Math-7B, copying grows when the hint is more reliable than the accuracy the model reaches without hints at the same budget. A model that cannot beat the hint learns to copy it, and the updates spent copying are lost to learning. Copying persists because a saturated copier gives every response to a prompt the same reward, which leaves group-relative RL no contrast within a prompt. Adding no-hint questions restores this contrast. The first response token can choose between the routes a model has. On held-out questions, replacing one hidden state lowers wrong-hint copying from 0.97 to 0.35 and from 0.80 to 0.28, and forcing only the first token lowers it to 0.32 and 0.31. The same control applied during training creates reward contrast only while it is on. RLVR learns from the alternatives that the policy itself samples.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, reinforcement learning with verifiable rewards, reward aliasing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_seFkOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates how RLVR reinforces hint copying versus problem solving when candidate solutions are provide in the prompt. Through controlled experiments on Countdown and arithmetic tasks, the authors show that copying tends to increase when the hint is more reliable than the accuracy the model can achieve through training without hints. They further explain that saturated copying can persist even when hints become unreliable, as uniform rewards within rollout groups eliminate the contrast needed for group relative optimization. The authors examime how mixing questions without hints can restore learning signals and reduce copying behavior. Through hidden state replacement and first token interventions, they demonstrate that copying and solving behaviors can be redirected without updating model parameters, particularly when the model already has sufficient solving capability.

Critical strengths

Section 6 provides an interesting causal analysis of copying and solving behaviors through hidden state replacement and first token interventions. In particular, the experiments separating the effects of the first token distribution from those of the KV cache provide evidence beyond observational correlations.

Critical weaknesses
  1. Many of the findings through section 5 appear relatively intuitive given the known properties of RLVR.

    • For example, copying is reinforced when the hint provides higher expected reward then solving, and saturated copying elimintates the reward contrast needed for group relative optimization. It is unclear how much new understanding the authors provide beyond existing knowledge.
    • The finding that mixing questions with and without hints is more effective than training entirely without hints also seems largely expected, since the latter does not directly train the modle to resist misleading hints. Understanding how to prevent the emergence of saturated copying in the first would offer more valuable insight into RLVR training dynamics.
  2. The interpretation of the first token as a mechanism for selecting between copying and solving behaviors may be overly broad.

    • The observed effects appear closely tied to the specific response patterns learned by the evaluated checkpoints, particularly the distincion between responses starting with "To" and "The". Although the interventions show a causal effect in these settings, it remains unclear whether the same mechanism generalizes to different prompt formats, reponse patterns, or models that do not exhibit such distinctive opening tokens. Additional evidence would strength the finding that the first token plays a general role in selecting between reasoning behaviors rather than reflecting a particular learned response pattern.
Questions for authors

Please see weaknesses above.

4 comments

Dear Reviewer seFk, we express our deep appreciation for your time and insightful comments. In our updated manuscript, we highlight the changes in blue\color{blue}{\text{blue}}.

In what follows, we address your comments one by one.


W1-1. Many findings through Section 5 appear intuitive given known properties of RLVR. How much new understanding does the paper provide?

We agree that the two premises are intuitive: copying pays when the hint is more reliable than solving, and saturated copying removes the within-group contrast. The introduction of the updated manuscript now presents them as the starting point. The findings are the outcomes these premises do not determine:

  • Same hints, opposite lessons. We labeled 60 randomly drawn wrong-hint responses at update 24 in each of 16 base-model runs of three models. In the six runs whose copying ends at 0.90 or above, at most 5 of 60 responses check the hint and reject it; in nine of the other ten, 27 to 58 of 60 do. At c = 0.65, two seeds of the same setting end at 0 and 52 of 60 (Section 4.1 and Table 25 of the updated manuscript). One LLM labeled the responses, a second LLM from another developer relabeled 119 of them without seeing the first labels (Cohen's κ = 0.955), and two team members relabeled 100 by hand.
  • Same reward, different behavior. Seven checkpoints trained with always-correct hints all score 0.996–1.000 with correct hints, yet their wrong-hint accuracy ranges from 0.00 to 0.60 (Section 4.1 of the manuscript).
  • Persistence beyond what the reward favors. With 79% of hints wrong, copying no longer pays, and blind copiers (0.93) fall to 0.01, while saturated copiers (≥ 0.998) stay at 1.00. DAPO-style filtering, 16 samples per prompt, temperature 1.2 and a product reward over hint pairs all leave them there (Sections 5.2 and 5.5 of the updated manuscript).
  • Order with fixed data (W1-2 below).

W1-2. Mixing questions with and without hints seems expected to beat training without hints. How can the emergence of saturated copying be prevented in the first place?

(a) Mixing versus removing hints. Misleading hints alone do not teach resistance: at q = 0 every training question has a hint, 79% of them wrong, and copying stays at 1.00. Removing hints (q = 1) leaves test-time copying at 0.80 and 0.97. The mixtures lower copying, and the smallest (q = 0.25) gives the lowest copying in both replicates (Section 5.3 of the manuscript).

(b) Preventing saturated copying from emerging. To address your question, we train five pairs of runs from the base model on the same 1,536 Countdown questions, each used once, at c ≈ 0.81 with a quarter of the questions without hints. One run of each pair sees the no-hint questions in updates 1–6; the other sees them in random order. The data and the hints are identical within a pair. We present the results in Table R1.

Table R1. Countdown accuracies after 24 updates, mean of five pairs.

MetricRandom orderNo-hint first
Wrong-hint copying0.7620.398
Wrong-hint accuracy0.0850.218
Correct-hint accuracy0.9880.981
No-hint accuracy0.2570.260

In random order copying keeps rising; with the no-hint questions first it levels off after update 16. Copying is lower in all five pairs (paired difference −0.364 ± 0.221 across pairs). No-hint accuracy at update 8, shortly after hints arrive, is 0.05–0.08 in all ten runs, so the gap does not come from a stronger solver at that point. Setting the reward of wrong-hint groups to zero in the no-hint-first schedule raises copying by 0.31, 0.32 and 0.53, so the reward on wrong-hint groups is what holds copying down. We have incorporated this into Section 5.4 (Figure 4) of the updated manuscript.


W2. The first-token interpretation may be overly broad and tied to the "To"/"The" patterns of these checkpoints. Does it generalize to other prompt formats, response patterns or models?

We agree that "To" and "The" are patterns these checkpoints learned, and we have narrowed the claim accordingly. The abstract, introduction, Section 6 and conclusion of the updated manuscript now state the result for checkpoints that copy immediately, opening with the answer. Within that scope, the intervention forces each checkpoint's own no-hint opening token rather than a fixed word, and it lowers copying under three paraphrased hint templates. In two checkpoints that copy after writing a derivation, prompt-end state replacement changes copying by −0.002 and +0.003 (Section 6.2 of the updated manuscript).

Follow-up to W1-2: does presenting the no-hint questions first only delay copying?

To test this, we continued all ten update-24 checkpoints for 24 more updates on 1,536 new questions, with a freshly initialized optimizer and the same data and order for both runs of a pair. At update 48, every no-hint-first run copies wrong hints at 0.03–0.11. The two random-order runs that copied at 0.986 and 0.920 at update 24 stay saturated, at 1.000 and 0.997 with wrong-hint accuracy 0.000 and 0.001; over the last 12 updates, 0 of 103 and 4 of 100 of their wrong-hint groups are mixed. The other three random-order runs fall to 0.02–0.12, close to their pairs. With the no-hint questions first, none of the five runs reaches saturated copying; in random order two of five do, and they stay there under continued training, as the saturated copiers in Section 5.1 do. We have incorporated this into Section 5.4 and Appendix H.1 (Figure 9) of the updated manuscript.

Reviewer_seFkOpenReview ↗

Official Comment by Reviewer seFk

Thank you for the authors' efforts to address my concerns and for the additional experiments. I find the observations that copying behavior can persist after saturation even when it is no longer favored by the reward, and that training order can affect whether such saturation occurs, to be interesting empirical findings. The additional experiments and clarifications have also strengthened my confidence in the soundness of the results. Accordingly, I've raised my scores for informativeness, soundness, and overall rating.

My remaining uncertainty is mainly about the broader significance of the work. While I appreciate the value of carefully controlled experiments, I am still unsure how relevant the identified hint-reliance dynamics are to more realistic RLVR settings and to what extent these findings might inform future research in this area. I would have liked to see a clearer discussion of the broader implications of these findings and the new understanding they provide beyond the specific experimental settings.

Dear Reviewer seFk,

We sincerely thank you for your time and for recognizing the additional experiments and the two findings on persistence after saturation and on training order.

Q. How relevant are the hint-reliance dynamics to more realistic RLVR settings, and what do the findings imply for future research beyond the specific experimental settings?

The experiments in the paper do not fully answer this question. They establish the dynamics in Countdown, where we can tell how each response reaches its answer. Beyond Countdown, the paper shows that training Qwen2.5-Math-7B on MATH with always-correct hints raises wrong-hint adoption by 0.057 and 0.053, and that reliance learned on Countdown carries over to HumanEval and MBPP by 0.12–0.17 (Section 7 and Appendix H.4 of the updated manuscript). In those experiments we built the hints ourselves.

Published guided-RLVR methods put correct guidance in the prompt, mostly for questions the policy cannot yet solve, which is the regime where Section 4.2 finds that reliance grows. GHPO appends the beginning of the ground-truth solution when no rollout is correct (Liu et al., 2025), QuestA adds the first 50% and later 25% of a solution to hard questions (Li et al., 2026), and StepHint gives the first steps of a stronger model's reasoning (Zhang et al., 2026). This line of work already treats reliance as a concern: QuestA shortens its hints to "gradually reduce reliance on hints", and Xie et al. (2026) drop hints per question and report that hinting every question hurts no-hint performance.

To test a published method, we evaluate the GHPO checkpoints released by one of its authors together with their GRPO counterparts, which are trained from the same base model on the same data without hints. We use three pairs: Qwen2.5-Math-7B trained for 5 epochs and for 1 epoch, and Qwen2.5-7B trained for 3 epochs. On the 455 MATH-500 questions of Appendix H.4, we either print a wrong answer after the question or append the first half of a wrong solution in GHPO's hint format. Table R2 reports how much more often the final answer equals the wrong answer than without a hint.

Table R2. Increase in adoption of a supplied wrong answer on MATH-500, GHPO checkpoint minus its GRPO counterpart.

Checkpoint pairNo-hint accuracy (GHPO / GRPO)Printed wrong answer, first setPrinted wrong answer, second setHalf of a wrong solution
Qwen2.5-Math-7B, 5 epochs0.785 / 0.768+0.060+0.055−0.024
Qwen2.5-7B, 3 epochs0.724 / 0.728+0.173+0.139+0.038
Qwen2.5-Math-7B, 1 epoch0.685 / 0.697−0.007+0.062−0.032

Bold: the 95% interval over questions excludes 0. The second set of wrong answers and the wrong solutions cover 158 of the 455 questions.

In every pair, no-hint accuracy differs by at most 0.017. In the 5- and 3-epoch pairs, the GHPO checkpoint still adopts a printed wrong answer more often on both sets of wrong answers, and no-hint benchmarks do not reveal this difference. The 1-epoch pair is mixed, and no pair differs when given the first half of a wrong solution.

The paper already discusses what follows for RLVR in practice. Section 7 (Practical implications) recommends evaluating with wrong and absent hints, since reward and correct-hint accuracy cannot tell a solver from a copier; tracking the fraction of mixed-reward groups; and presenting no-hint questions first, which lowers wrong-hint copying on Countdown from 0.762 to 0.398 on identical data. Section 2 and Appendix G relate these results to hint-based methods, including the per-question hint dropout of Xie et al., whose effect on reward contrast Section 5.3 explains. More broadly, early experience is known to shape later learning in RL (Nikishin et al., 2022). The Scope paragraph of Section 7 states when this happens in our setting: two routes earn the same reward and the policy samples one far more often, which makes implicit cues in real training data the natural next test. Once the policy stops sampling the alternative, changing the reward or the hint reliability does not bring it back, while mixing in no-hint questions does (Sections 5.1, 5.3 and 5.5).

We have added GHPO and QuestA to Section 2 and Appendix G, and this test with its full protocol to Section 7 and Appendix H.5 (Table 27) of the updated manuscript.

References

  • Jiazheng Li et al. QuestA: Expanding reasoning capacity in LLMs via question augmentation. In ICLR, 2026.
  • Ziru Liu et al. GHPO: Adaptive guidance for stable and efficient LLM reinforcement learning. arXiv:2507.10628, 2025.
  • Evgenii Nikishin et al. The primacy bias in deep reinforcement learning. In ICML, 2022.
  • Pei-Xi Xie et al. Mitigating distribution sharpening in math RLVR via distribution-aligned hint synthesis and backward hint annealing. arXiv:2604.07747, 2026.
  • Kaiyi Zhang et al. StepHint: Multi-level stepwise hints enhance reinforcement learning to reason. In ACL, 2026.
Reviewer_piD9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates what happens when the model has access to the hint during RLVR training. Using the Countdown setting, the authors find that the model tends to rely on the “hint” when the model is weaker.

Critical strengths
  • The paper provides various analyses. I find it particularly interesting that there is a clear negative correlation between how capable the model is and how much the model exploits the hint (Figure 2).
Critical weaknesses
  • Experiment: The experimental setup (the Countdown dataset and the “hint”) seems to be somewhat artificial. I don't think this is a major limitation on its own, but the paper doesn't discuss its implications for realistic setups, making it hard to understand the experiments' motivation and interpret their significance in a practical context. Also, the angles of many analyses seem not particularly novel, yet the paper does not discuss their relation to prior works. For example, “reasoning can be steered by pre-fixing the first token(s)” (Section 6) is discussed in various papers, such as [1-3].

  • Writing: I find the draft to be hard to read. For example, there is no explanation/citation for “Countdown,” which makes the experimental setup unclear. Similarly, I could not find explanations/definitions for many terms that seem crucial to understanding the content, such as “hint”, “saturated/blind copier”.

[1] https://arxiv.org/abs/2503.02875 [2] https://arxiv.org/abs/2512.15274 [3] https://arxiv.org/abs/2605.17026

Questions for authors
  • As mentioned as a weakness, I would like to know the practical implications of each of the findings (Sections 4- 6).
4 comments

Dear Reviewer piD9, we express our deep appreciation for your time and insightful comments. In our updated manuscript, we highlight the changes in blue\color{blue}{\text{blue}}.

In what follows, we address your comments one by one.


W1-1/Q. The setup (Countdown and the "hint") seems artificial. What are its implications for realistic setups, and what are the practical implications of each finding in Sections 4–6?

RLVR pipelines already put hints, expert traces and partial solutions in the prompt (Section 2 of the manuscript). To address your question, the new Practical implications paragraph (Section 7) of the updated manuscript gives one implication per section:

  • Section 4. Seven checkpoints trained with always-correct hints all score 0.996–1.000 with correct hints, while their wrong-hint accuracy ranges from 0.00 to 0.60. Reward and correct-hint accuracy cannot tell them apart, so evaluation should include wrong and absent hints.
  • Section 5. The fraction of mixed groups in the training log shows when training has stopped comparing alternatives. Order matters too: on the same 1,536 Countdown questions, presenting the no-hint questions first lowers wrong-hint copying from 0.762 to 0.398 and raises wrong-hint accuracy from 0.085 to 0.218 (new Section 5.4 and Figure 4 of the updated manuscript).
  • Section 6. For a model that already copies immediately, forcing its own no-hint opening token tests whether a solving route still exists. It lowers copying from 0.97 to 0.32 in a checkpoint that can solve, and by 0.09–0.19 in copiers that rarely solve.

The same reliance appears outside Countdown. Qwen2.5-Math-7B trained on MATH with always-correct hints adopts a wrong hint more often than the same model trained without hints, by 0.057 on 455 MATH-500 questions and 0.053 on a held-out set. Checkpoints trained with hints only on Countdown reproduce a wrong candidate function on HumanEval and MBPP more often than their no-hint counterparts, by 0.12–0.17 in Qwen and Llama. We have incorporated these into Section 7 and Appendix H.4 of the updated manuscript, which give the protocol and per-seed results.


W1-2. The angles of many analyses seem not novel, and the relation to prior work is not discussed; e.g., steering reasoning by pre-fixing the first token(s) (Section 6) is discussed in [1-3].

(a) First tokens. We agree and now cite all three in Section 2 of the updated manuscript: prefix fine-tuning (Ji et al., 2025), prefix-focused RL (Sun et al., 2025), and direct intervention on opening tokens (Nguyen et al., 2026). Section 6 builds on them. It asks which of two routes that earn the same reward a trained model takes, and separates the first token from the prompt cache: in S1, changing only the first-token distribution lowers copying from 0.969 to 0.395, while changing only the cache leaves it at 0.965.

(b) Novelty more broadly. The introduction of the updated manuscript now states what the expected-reward account does not predict. The same unreliable hints can teach one run to check the hint and reject it and another to copy it: at c = 0.65, two seeds end at 0 and 52 of 60 sampled responses that check and reject (Section 4.1 of the updated manuscript). Equal reward hides different behavior. Saturated copiers stay at 1.00 even when 79% of hints are wrong, and DAPO-style filtering, larger groups and higher temperature leave them there. Neither wrong hints alone nor removing hints undoes copying, while a mixture does. In Countdown, the order of the same data changes whether copying saturates.


W2-1. There is no explanation or citation for "Countdown".

The introduction and Section 3 of the updated manuscript now cite Countdown (Gandhi et al., 2024) and describe it: the model combines given numbers into an arithmetic expression that equals a target, and an exact verifier checks the answer.


W2-2. Terms crucial to the content, such as "hint" and "saturated/blind copier", are not defined.

Section 3 of the updated manuscript now defines the terms the results use, in one place and before the first table:

  • Hint: a complete candidate expression printed in the prompt, correct on a fraction c of training questions. Example: to make 26 from 8, 4, 3 and 2, a correct hint is 8×4−3×2 and a wrong hint is (8+4)×2+3, which makes 27.
  • Solver, selective start, blind copier, saturated copier: starting checkpoints named by wrong-hint copying (no hints in training; ≤ 0.19; about 0.93; ≥ 0.998).
  • Route and mixed group: how a response reaches its answer, and a group whose responses do not all earn the same reward.

Figure 1's caption and the arithmetic task now use the same terms.

Dear Reviewer piD9,

As the discussion period is about to close, we would like to thank you again for the time and effort you have put into reviewing our paper. We hope our response has addressed your concerns. If you have any further questions, we would be glad to discuss them in the remaining time. Thank you again for your valuable comments.

Thank you for your rebuttal.

Now I see that the overall rigor of the writing is improved compared to the initial draft, at least with respect to what I mentioned in the review, and, given that, I increase my score by one.

However, I still believe the overall quality (scope, the significance of new insights, insight for a realistic setup) remains below the bar. Therefore, I keep my score below the acceptance threshold. I believe it requires a significant improvement beyond review-based additions.

Thank you for reading our response and the revision, and for raising your score. We appreciate your feedback on the writing, which has made the paper clearer.

We take your remaining concern about scope and realistic setups seriously. Countdown lets us tell how each response reaches its answer, which is what the mechanistic analysis needs; beyond it, the updated manuscript reports wrong-hint adoption after hint training on MATH and transfer of reliance to code (Section 7, Appendix H.4). To test a setting we did not construct, Appendix H.5 (Table 27) evaluates released checkpoints of a published guided-RLVR method, GHPO, against their GRPO counterparts on MATH-500: in two of three pairs the GHPO checkpoint adopts a printed wrong answer more often while no-hint accuracy differs by at most 0.017, a difference that no-hint benchmarks do not show. We agree that testing implicit cues in real training data is the natural next step, as stated in the Scope paragraph of Section 7.

Thank you again for your time and constructive comments.

Reviewer_CoJmOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
3(+1)
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper investigates when reinforcement learning with verifiable rewards reinforces copying a supplied candidate answer rather than independently solving the problem. Through controlled hint-reliability experiments, it connects copying to budget-matched no-hint accuracy, explains persistent copying through vanishing within-group reward contrast, and uses activation replacement and first-token interventions to investigate behavioral selection.

Critical strengths
  • The paper demonstrates that nearly identical training rewards can conceal substantial differences in independent problem-solving ability and robustness to incorrect hints.

  • Reliability sweeps, different initializations, multiple seeds, and matched evaluations provide useful evidence about how copying emerges and persists.

  • Held-out activation replacement, random-state controls, and first-token/cache separation go beyond correlation to identify a causal influence on copying in the studied checkpoints.

Critical weaknesses
  • Equation 4 concerns the current solving success rate, whereas substituting the independently trained endpoint accuracy (A_B) yields an empirical heuristic rather than an established training-dynamics result.

  • The central mechanistic evidence comes mainly from Countdown and selected Qwen checkpoints; the arithmetic, partial-hint, and MATH experiments do not establish the complete mechanism in realistic reasoning pipelines.

  • Copying lineages generate substantially fewer tokens, so update-matched comparisons demonstrate learning deficits at a fixed update budget but do not establish the corresponding disadvantage under matched compute.

  • Two-update interventions and a specific importance-weighted trainer provide limited support for broader claims about whether externally supplied successful trajectories can produce lasting learning.

Questions for authors
  • Can you explain why endpoint no-hint accuracy (A_B), rather than time-varying or question-dependent solving accuracy, predicts copying, and test this relationship across substantially different training budgets?

  • Do the copying dynamics and first-token effects persist under paraphrased prompts, partial-hint training, other model families, and more realistic mathematical or coding tasks?

  • Under matched generated-token or FLOP budgets, how much of the no-hint performance gap between copier and solver lineages remains?

  • Can longer intervention schedules or alternative updates that preserve credit for low-probability successful trajectories produce durable improvements after control removal?

2 comments

Dear Reviewer CoJm, we express our deep appreciation for your time and insightful comments. In our updated manuscript, we highlight the changes in blue\color{blue}{\text{blue}}.

In what follows, we address your comments one by one.


W1/Q1. Equation 4 concerns the current solving rate. Why does endpoint no-hint accuracy A_B, rather than time-varying accuracy, predict copying, and how does this depend on the training budget?

Equation 4 holds for the current success rate y of the non-copying route. We measure y directly as the accuracy of responses to wrong hints that do not copy them, and A_B is an empirical reference for where the model's solving can go within budget B. Reading y every eight updates on 41 hinted runs, the two references answer different questions:

  • For the first change in copying, the sign of c − y gives the direction in all 30 runs where copying moves, against 23 for c − A_B.
  • Over later intervals, c − A_B does better, 38 of 46 against 31 of 43.

Current accuracy predicts how copying starts, and reachable accuracy predicts where it goes. We have incorporated this into Section 4.2 and Appendix H.3 of the updated manuscript, which gives the counting rule. We have not retrained A_B references at substantially different budgets; the 48- and 72-update lineages in Table 1 and Figure 3b of the manuscript are training histories rather than such a sweep.


W2/Q2. Do the copying dynamics and first-token effects persist under paraphrased prompts, partial-hint training, other model families, and realistic mathematical or coding tasks?

  • Paraphrases. Forcing the first token lowers copying under three paraphrased hint templates in S1 and S2 (Section 6.2 of the updated manuscript).
  • Other families. The reliability pattern holds in Llama-3.1-8B and Qwen2.5-14B (Section 4.2 of the manuscript). The first-token effect is stated for checkpoints that copy immediately; in two checkpoints that copy after a derivation, prompt-end state replacement changes copying by −0.002 and +0.003.
  • Mathematics. Qwen2.5-Math-7B trained on MATH with always-correct hints adopts a wrong hint more often than the same model trained without hints, by 0.057 on 455 MATH-500 questions and 0.053 on a held-out set, and by 0.043 at c = 0.8. Appendix H.4 of the updated manuscript gives the protocol and each seed (0.059 and 0.054 on MATH-500).
  • Code. Checkpoints trained with hints only on Countdown reproduce a wrong candidate function on HumanEval and MBPP more often than their no-hint counterparts, by 0.12–0.17 in Qwen and Llama.
  • Partial hints. At inference, checkpoints build 0.43–0.88 of their answers on a wrong first operation (Appendix B.3 of the manuscript). Training with partial hints is not in the paper.

W3/Q3. Copying lineages generate fewer tokens. Under matched generated-token budgets, how much of the no-hint gap remains?

You are right, and the claim is now per update. We continued three copier lineages without hints until they had generated as many response tokens as two lineages trained without hints. Table 24 of the updated manuscript gives all six pairs: the differences in no-hint accuracy lie between −0.011 and +0.048. Matching total tokens with prompts included, two of the six intervals lie below zero. At matched updates the copier lineages are 0.14 and 0.25 lower after 48 updates. The abstract, Section 4.3 and the conclusion of the updated manuscript state the cost per update.


W4/Q4. Can longer intervention schedules or updates that preserve credit for low-probability successful trajectories produce durable improvements?

We have not tested longer schedules or other updates, and the Scope paragraph (Section 7) of the updated manuscript now states that both training controls last two updates and use one weighting rule. Our measurements show why the two-update controls do not last. The pulse raises the fraction of mixed groups to 0.65–0.71 while it is on, and the fraction falls to 0.16 one update after removal. Under truncated importance weights, the first token of a controlled response has a median probability ratio of 0.04–0.09 and receives little weight.


A new result on prevention. From the base model and on the same 1,536 Countdown questions, presenting the no-hint questions first lowers wrong-hint copying from 0.762 to 0.398 and raises wrong-hint accuracy from 0.085 to 0.218 over five paired runs (Section 5.4 and Figure 4 of the updated manuscript).

Dear Reviewer CoJm,

As the discussion period is about to close, we would like to thank you again for the time and effort you have put into reviewing our paper. We hope our response has addressed your concerns. If you have any further questions, we would be glad to discuss them in the remaining time. Thank you again for your valuable comments.

Comments on the paper

1 comment

Dear Reviewers and Area Chair, we express our deep appreciation for your time and insightful comments. In our updated manuscript, we highlight the changes in blue\color{blue}{\text{blue}}. The title, Figures 1–3, the main table and the section structure are unchanged.

Reviewers CoJm and seFk noted that the first-token and cache interventions in Section 6 go beyond correlation, and Reviewer CoJm noted that nearly identical training rewards can conceal large differences in solving and in robustness to wrong hints. The revision keeps these results and makes the rest of the paper easier to check against them. We summarize the main changes below.


  1. Preventing saturated copying (Section 5.4, Figure 4). Reviewer seFk asked how saturated copying can be kept from forming. From the base model, five pairs of runs train on the same 1,536 Countdown questions, with the no-hint questions either first or in random order. After 24 updates, wrong-hint copying is 0.398 against 0.762 and wrong-hint accuracy 0.218 against 0.085. Copying is lower in all five pairs; correct-hint and no-hint accuracy are similar on average. When all ten runs continue for 24 more updates on new questions, every no-hint-first run copies at most 0.11, while the two random-order runs that copied above 0.90 stay saturated (Appendix H.1, Figure 9).
  2. What the expected-reward account does not predict (abstract, introduction, Sections 4.1 and 5.3, Table 25). The same unreliable hints can teach one run to check the hint and reject it and another to copy it. In 16 base-model runs, at most 5 of 60 sampled wrong-hint responses check and reject in the six saturated runs, against 27 to 58 of 60 in nine of the other ten (Reviewers seFk and piD9).
  3. Definitions, citations and readability (Sections 1–3). Countdown is cited and described; hint, route, mixed group and the four starting checkpoints are defined in Section 3 before the first table, with a worked example; borrowed methods and benchmarks are cited where they first appear (Reviewer piD9).
  4. Practical implications (Section 7). One implication per section (Reviewer piD9).
  5. Prior work on first tokens (Section 2). The three papers suggested by Reviewer piD9 are cited.
  6. Compute (abstract, Section 4.3, Table 24). The learning cost is stated per update; at matched generated tokens the six differences lie between −0.011 and +0.048 (Reviewer CoJm).
  7. Reference accuracy (Section 4.2, Appendix H.3). A_B is described as an empirical reference; current accuracy predicts the onset of copying and A_B its later direction (Reviewer CoJm).
  8. First-token scope (abstract, Section 6). The result is stated for checkpoints that copy immediately, with the paraphrase check (Reviewers seFk and CoJm).
  9. Other tasks (Section 7, Appendix H.4). MATH training with hints raises wrong-hint adoption by 0.057 and 0.053; reliance learned on Countdown carries to code problems. The Scope paragraph lists what remains untested.
  10. Section 5.2. One sentence on the KL penalty is corrected to match the token-averaged gradient in our implementation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.