acceptodds
Under review as a conference paper at ICLR 2027

Steer, Don’t Tell: Breaking Reasoning Plateaus with Reachability-Aligned Self-Distillation

Abstract

Post training for large reasoning models remains insufficient on challenging problems, whose correct trajectories are hard to sample. Though prior works address this difficulty by integrating supervised fine tuning on expert trajectories with reinforcement learning, such external experts are not always available. Existing self improvement methods instead use reference solutions to create a privileged self-teacher. Yet privileged teacher correctness does not imply student reachability. Solution conditioning shifts the teacher distribution toward actions unlikely under the unprivileged student, producing seemingly ”correct” trajectories that are unreachable or even mal-formed learning targets. Accordingly, we identify reachability alignment as the key to self constructed supervision and propose Self-Steered Reasoner (SSR). SSR realizes this principle through student support projection, which transforms the privileged teacher distribution into a directional signal only over reasoning actions locally reachable under the unguided student. Verification then retains correct trajectories sampled from this reachable space. To further boost the sampling efficiency, an iterative refinement framework is adopted, using diagnostic guidance based on current trajectories. On problems with pass@64 = 0 even after extensive RL, SSR recovers correct and learnable trajectories for 74.6% without parameter updates . Fine tuning on these trajectories followed by resumed reinforcement learning improves aggregate accuracy by 4.9 percentage points on AIME25/26 and HMMT26. These results show that privileged guidance becomes effective self-improvement data when aligned with the reasoning actions the student can reach.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, self-improve, self-distillation, hindsight-bias, reasoning plateau

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gDmuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Self-Steered Reasoner (SSR), a framework for constructing supervision on problems that a plateaued reasoning model fails to solve within a fixed sampling budget. Its central operation restricts a privileged teacher’s next-token distribution to tokens exceeding a probability threshold under the unprivileged student. The same frozen model supplies both distributions under different conditioning. The full procedure combines this projection with filtered diagnostic guidance, KL-based intervention placement, prefix retention, verification, and iterative refinement. Verified trajectories are used for SFT followed by resumed RL. SSR recovers 147 of 197 empirically unsolved problems for Qwen3-8B and reports peak aggregate accuracy gains of 4.9 and 5.2 percentage points over continued RLVR for the 8B and 4B settings, respectively.

Critical strengths
  1. The distinction between correctness under privileged conditioning and suitability as student supervision is useful. Table 2 provides a controlled generation-side comparison that isolates projection under matched diagnostic guidance and retained prefixes, showing substantially fewer explicit references to privileged information while transparently reporting the associated reduction in success rate.
  2. The complete pipeline produces downstream improvements in both evaluated model settings. The equal-continuation-budget comparison in Figure 3 also provides evidence that iterative refinement improves problem recovery over independent single-round corrections. These results support the practical usefulness of the complete system.
Critical weaknesses
  1. The central claim that student-support projection produces more learnable supervision is not directly established. Table 2 isolates projection only through generation outcomes and privileged-dependence metrics; it does not compare downstream learning from the matched raw-guided and projected trajectories. Table 1 instead compares complete methods that differ in multiple aspects of data construction. Likewise, Table 4 establishes differences in recovery, not training utility. Its regenerate-from-start variant recovers fewer problems, but this does not establish inferior supervision—particularly when the paper itself argues that more correct trajectories need not yield better training data.

  2. The robustness of the headline training gains is insufficiently documented. Section 3.1 selects each method’s checkpoint using the highest aggregate accuracy on the same benchmarks used to report the main results. Although the selection rule is shared, the paper does not provide learning curves or clearly establish comparable training horizons and checkpoint-selection opportunities. Consequently, it is difficult to assess whether the reported improvements are sustained or sensitive to peak selection. The operational basis for declaring a plateau and the claim that OPSD quickly deteriorates with continued training are also not supported by displayed training curves. Figure 3 concerns search recovery and does not resolve these issues.

Questions for authors
  1. Central empirical clarification: Do you have downstream SFT and resumed-RL results for the matched raw-guided and projected settings in Table 2, controlling for training problems and training budget? Does the regenerate-from-start variant in Table 4 remain inferior when evaluated by downstream learning rather than recovery? These comparisons would directly affect my assessment of the claimed role of projection and the necessity of the surrounding search procedure.
  2. Metric clarification: How should the overall lexical-anchoring reduction be interpreted given the much smaller difference on jointly solved problems? What evidence validates the anchoring metrics as measures of harmful privileged dependence rather than correctness or legitimate mathematical similarity?
  3. Evaluation clarification: For each method, how many training steps and evaluated checkpoints contributed to Table 1? Please provide the available learning curves, identify the selected checkpoints, and clarify whether an independent validation set was used. Fixed-budget or final-checkpoint results would help assess whether the reported ranking is robust.
5 comments

Thank you for the targeted requests. Both downstream comparisons are now complete, and both favor SSR: learning from raw-guided versus projected targets, and learning from targets regenerated without the retained prefix. We give the full reported curves and the matched-success audit below.

Q1. Do projection and prefix retention improve downstream learning?

Reviewer question

Central empirical clarification: Do you have downstream SFT and resumed-RL results for the matched raw-guided and projected settings in Table 2, controlling for training problems and training budget? Does the regenerate-from-start variant in Table 4 remain inferior when evaluated by downstream learning rather than recovery? These comparisons would directly affect my assessment of the claimed role of projection and the necessity of the surrounding search procedure.

Projection: direct training evidence

Projection improves the learning outcome, not just the generation-side measurements. With the SFT budget and number of training trajectories matched, we compare three SSR seeds and two raw-guided runs after SFT and resumed RL.

Table 1. Raw-guided versus projected supervision: every reported checkpoint.

Resumed-RL stepRaw run 1Raw run 2SSR seed 42SSR seed 43SSR seed 44
1060.3562.2364.6861.0062.07
2062.6160.3862.7464.6264.62
3064.7758.9165.3766.9365.06
4063.3363.5566.7966.5268.83
5061.4559.7664.3264.1765.48
6063.2459.2064.6266.7866.73
7063.0960.6765.0665.7363.55

Table 2. Across-run downstream accuracy.

EvaluationSSR, 3 seedsRaw, 2 runsSSR advantage
Common step 7064.78 ± 1.1261.88 ± 1.71+2.90 pp
Mean of each run’s peak on steps 10–7067.52 ± 1.1464.16 ± 0.86+3.36 pp

At every reported checkpoint from step 20 through step 70, every SSR run exceeds both raw runs. The common step-70 advantage is 2.90 percentage points. Even the lowest SSR peak, 66.79, exceeds the highest raw peak, 64.77. The ordering therefore holds across runs and across common checkpoints, not only after selecting a favorable peak.

The original fixed-input experiment varies only the continuation distribution while keeping failures, steers, prefixes, and intervention points identical. Raw decoding achieves 92.89% first-round pass@16 versus 31.98% for projected decoding, but it does not deliver better downstream learning in the new comparison. Answer verification and training utility are distinct, and the new training results directly support the proposed distinction.

Prefix retention: downstream training, not only recovery

We also trained on verified trajectories regenerated from the first token with the steer but without the retained prefix.

Table 3. Downstream training with and without the retained prefix.

Resumed-RL stepSSR originalNo retained prefixSSR advantage (pp)
1064.6859.70+4.98
2062.7459.24+3.50
3065.3759.97+5.40
4066.7963.69+3.10
5064.3260.27+4.05
6064.6263.03+1.59
7065.0662.79+2.27

Both runs peak at step 40: 66.79 for SSR versus 63.69 without the prefix, a 3.10-point advantage. At the common step 70, SSR is 65.06 versus 62.79, retaining a 2.27-point advantage. SSR is higher at every reported checkpoint.

The earlier equal-continuation-budget search comparison yielded 166 verified trajectories over 60 problems with the full design versus 17 over seven problems when regenerating from the start. The new experiment additionally demonstrates downstream value: the no-prefix variant remains inferior when evaluated by learning, not merely by recovery.

Q2. How should anchoring and private dependence be interpreted?

Reviewer question

Metric clarification: How should the overall lexical-anchoring reduction be interpreted given the much smaller difference on jointly solved problems? What evidence validates the anchoring metrics as measures of harmful privileged dependence rather than correctness or legitimate mathematical similarity?

What the anchoring scores tell us

Lexical anchoring measures weighted word overlap. There are many possible different ways to describe a mathematical solution. We want the generated trajectory to have its own way of describing the solution. But some problems simply have limited ways to describe their solution. So lower lexical anchoring means better trajectory, but higher lexcical anchoring does not mean that the trajectory must be worse. It is more of a diagnostic metric, while prob anchoring and avg. leakage are more important.

Probabilistic anchoring asks a more specific question: does reading the generated trajectory make the particular reference proof easier for the model to reproduce? We measure this by having a frozen model score the same proof, first given only the question and then given the question plus the trajectory. The normalized improvement gives the probabilistic anchoring score.

This matters because strongly reference-anchored supervision may encourage the student to replay a particular proof rather than learn reasoning that supports different valid solutions. We want the student to learn how to solve the problem, not just how to recite its reference proof. Probabilistic anchoring diagnoses this reference-specific tendency. The training comparison tests whether the resulting supervision actually improves learning.

Table 4. Anchoring on the same 60 jointly solved problems.

MeasurementRaw guidedProjected
Lexical anchoring85.04%84.47%
Probabilistic anchoring, mean8.41%5.58%
Probabilistic anchoring, median5.20%3.36%

The lexical gap is small, but probabilistic anchoring decreases in both mean and median on the same problems. We also went beyond these scores and checked what the solutions actually say.

Uniform full-text attribution audit

We examined 166 matched pairs in which both continuations passed answer verification. Each pair shared the same question, retained prefix, steer, and intervention point.

Using GPT-5.6-Sol (xhigh) with the decoding method hidden, we read each full continuation and asked: does the solution credit a reasoning step or answer to a hint, reference, or feedback that the public question never supplied? Each positive judgment required a supporting quote. Simply using the same mathematical fact as the steer did not count.

Table 5. References to private guidance in 166 matched successful pairs.

MeasurementRawProjected
Continuations flagged by the fixed-phrase detector164/1660/166
Continuations flagged by full-text review166/1666/166
Full-text attribution rate, equal problem weights100%5.76%

The problem-weighted gap is 94.24 percentage points here.

Successful-output phrase counts and length normalization

We also examined 850 raw and 180 projected successful continuations from problems solved by both decoders in the original generation experiment. We removed the retained prefix and applied the same phrase detector to the newly written text. We first averaged within each problem and then weighted problems equally.

Table 6. References to private guidance in newly generated text.

MeasurementRaw, 850 continuationsProjected, 180 continuations
Mean phrase matches per continuation19.070.00
Continuations with at least one match99.27%0.00%
Phrase matches per 1,000 tokens1.2130.000

The gap remains after removing the prefix and accounting for output length. It is present in the newly generated reasoning of successful outputs.

Finally, the SFT-to-RL comparison in Q1 tests training value directly. At the common step 70, projected supervision achieves 64.78 mean macro Avg@8, compared with 61.88 for raw, a 2.90-point advantage.

Our conclusion does not rest on treating lower overlap as better learning. The anchoring scores describe how closely trajectories track a particular reference. The full-text audit shows substantially fewer appeals to private guidance. The training experiment establishes that the projected supervision produces better downstream learning.

Q3. Checkpoint selection & training curves

Reviewer question

Evaluation clarification: For each method, how many training steps and evaluated checkpoints contributed to Table 1? Please provide the available learning curves, identify the selected checkpoints, and clarify whether an independent validation set was used. Fixed-budget or final-checkpoint results would help assess whether the reported ranking is robust.

The original-run ranking survives a common checkpoint grid

SSR’s advantage is sustained throughout the reported training grid. Here are all seven reported evaluation points, steps 10–70, for the original resumed-RL runs.

Table 7. Original-run learning curves on the same resumed-RL checkpoint grid.

Resumed-RL stepSSRHinted DecodeAnswer-conditionedDAILDASD
1064.6860.0458.4853.9052.07
2062.7459.6559.8053.8031.76
3065.3760.2060.3856.3430.33
4066.7960.8061.0756.8939.12
5064.3262.7561.0757.0840.16
6064.6261.1161.5255.2543.43
7065.0659.8761.9953.7144.85

Table 8. Peak locations and the common last checkpoint of the original runs.

MethodPeak on displayed gridPeak stepStep 70
SSR66.794065.06
Hinted Decode62.755059.87
Answer-conditioned61.997061.99
DAIL57.085053.71
DASD52.071044.85

SSR exceeds every listed offline baseline at every displayed step, not only when separate peaks are selected. It also exceeds continued RLVR’s reported peak of 61.9 at every displayed step, with a minimum of 62.74.

Reviewer_hWcMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Self-Steered Reasoner (SSR), a post-plateau self-distillation framework for mathematical reasoning. SSR starts from failed student trajectories. A frozen, privileged version of the same model reads the failed trajectory together with a reference solution and produces a diagnostic steer. The method identifies an intervention point using disagreement between guided and unguided token distributions. It then regenerates the suffix while restricting each next-token choice to tokens that exceed a probability threshold under the unguided student.

Correct continuations are verified, added to a training set, and used for SFT followed by resumed RLVR. The paper evaluates SSR on hard problems unsolved by the plateau checkpoint within 64 samples, using Qwen3-8B and Qwen3-4B-Instruct. SSR is reported to recover correct trajectories for 74.6% of the Qwen3-8B hard set and improve aggregate benchmark accuracy over continued RLVR.

Critical strengths

First, SSR requires a diagnostic model that has access to the complete reference solution and proof. Filtering explicit answers from the generated steer does not guarantee that decisive reasoning structure, constructions, or answer-equivalent information is not conveyed indirectly. This is substantially stronger supervision than the answer-only verifier setting normally associated with RLVR.

Second, the main results use the best aggregate evaluation checkpoint for each method. Since AIME2025, AIME2026, and HMMT are repeatedly evaluated during training and used to select the reported peak, the final comparison is vulnerable to selection bias. The threshold is also tuned on the same Qwen3-8B hard set used for the method analysis. No multi-seed results or independent development protocol are provided.

Finally, the privileged-dependence evidence is not fully convincing. In the matched subset solved by both raw and projected decoders, lexical anchoring changes only from 85.04% to 84.47%. The much larger overall reduction may therefore partly reflect different success sets rather than reduced dependence within comparable successful trajectories. The regex leakage metric is also too limited to capture semantic leakage.

Critical weaknesses
  1. The student-support projection is a clear and well-motivated mechanism. It directly addresses the concern that a solution-conditioned teacher can place probability mass on actions the unprivileged student would not plausibly take.

  2. The paper includes meaningful internal controls. The projection study holds failures, steers, prefixes, and intervention points fixed. The iterative-refinement study also compares against a wide-search allocation under the same continuation budget.

Questions for authors

Q1. The reachability claim is stronger than what the method demonstrates. A token being locally probable under the student does not imply that the full trajectory is likely under the unguided model, nor that it will be reliably learned by SFT or RL. The method also depends on full reference proofs, making it inapplicable to many answer-verifiable settings without such privileged supervision.

Q2. The key empirical claims are vulnerable to evaluation and measurement bias. Peak-checkpoint selection on the evaluation benchmarks, threshold tuning on the analyzed hard set, and single-run reporting weaken the performance claim. The leakage metrics do not provide a sufficiently strong, uniform test of semantic dependence on reference solutions.

3 comments

Thank you for recognizing the projection mechanism and the fixed-input controls. We now provide corresponding results and experiments for them.

Q1. Reference supervision, local reachability, and downstream learning

Reviewer question

The reachability claim is stronger than what the method demonstrates. A token being locally probable under the student does not imply that the full trajectory is likely under the unguided model, nor that it will be reliably learned by SFT or RL. The method also depends on full reference proofs, making it inapplicable to many answer-verifiable settings without such privileged supervision.

Having access to reference proofs is a common situation.

Every answer comes with a reason. From the perspective of curating a new training dataset, it is natural to figure out the correct solution before writing specific verification/grading rules. Therefore, it is a common situation to have a reference solution without having a model rollout with complete CoT pointing to the correct answer. This method studies how additional privileged supervision can be transferred into model performance. All compared baselines tries to do this, but our methods are more competent, especially for a model that can think.

The local constraint is explicit; learning is tested through training

Section 2.2 defines the allowed next-token set at prefix s as {v : p_student(v | s) > δ}. The guided distribution is renormalized on this set. This is the formal local reachability constraint. Its training value is an empirical question, and the new experiments now answer it directly.

Projected supervision outperforms raw-guided supervision after SFT and resumed RL. With the same SFT budget and training-trajectory count, we obtain the following complete reported curves for three SSR seeds and two raw runs.

Table 1. Raw-guided versus projected supervision: every reported checkpoint.

Resumed-RL stepRaw run 1Raw run 2SSR seed 42SSR seed 43SSR seed 44
1060.3562.2364.6861.0062.07
2062.6160.3862.7464.6264.62
3064.7758.9165.3766.9365.06
4063.3363.5566.7966.5268.83
5061.4559.7664.3264.1765.48
6063.2459.2064.6266.7866.73
7063.0960.6765.0665.7363.55

Table 2. Across-run downstream accuracy.

EvaluationSSR, 3 seedsRaw, 2 runsSSR advantage
Common step 7064.78 ± 1.1261.88 ± 1.71+2.90 pp
Mean of each run’s peak on steps 10–7067.52 ± 1.1464.16 ± 0.86+3.36 pp

At every reported checkpoint from step 20 through step 70, every SSR run exceeds both raw runs. Even the lowest SSR peak, 66.79, exceeds the highest raw peak, 64.77. The local constraint and its downstream learning benefit are supported by empirical evidence here.

Q2. Evaluation robustness and a uniform test of private dependence

A. The difference remains on matched successful outputs

The reviewer noted that lexical anchoring changes little on the jointly solved problems. Lexical overlap and references to private guidance measure different things. The original comparison on the same 60 problems also shows lower probabilistic anchoring:

Table 3. Anchoring on the 60 jointly solved problems.

MeasurementRaw guidedProjected
Lexical anchoring85.04%84.47%
Probabilistic anchoring, mean8.41%5.58%
Probabilistic anchoring, median5.20%3.36%

Rather than relying on these scores alone, we checked the generated solutions directly. We examined 166 matched pairs in which both outputs passed answer verification and shared the same question, prefix, steer, and intervention point.

We used GPT-5.6-Sol (xhigh) to read each full continuation with the decoding method hidden. The check asks a simple question: does the solution credit a step or answer to a hint or reference that the public question never supplied? Each positive judgment required a supporting quote. Using the same mathematical fact does not, by itself, count as a reference to private guidance.

Table 4. Private attribution in 166 matched successful pairs.

MeasurementRawProjected
Continuations flagged by the fixed-phrase detector164/1660/166
Continuations flagged by full-text review166/1666/166
Full-text attribution rate, equal problem weights100%5.76%

The problem-weighted gap is 94.24 percentage points. Full-text review catches six projected cases missed by the phrase detector, while the raw–projected separation remains large.

Both methods succeeded on these matched inputs. The attribution difference therefore remains when we hold the generation inputs fixed and use the same full-text rule, rather than comparing different sets of solved problems.

B. The difference also remains after accounting for output length

We checked a larger collection of 850 raw and 180 projected successful continuations from problems solved by both decoders. We removed the retained prefix and applied the same phrase detector to the newly generated text. Measurements were computed within each problem and then averaged with equal problem weights.

Table 5. References to private guidance in newly generated text.

MeasurementRaw, 850 continuationsProjected, 180 continuations
Mean phrase matches per continuation19.070.00
Continuations with at least one match99.27%0.00%
Phrase matches per 1,000 tokens1.2130.000

The gap remains after length normalization and after removing the retained prefix. It is present in the newly written reasoning of successful outputs.

Together, the phrase counts and full-text review show substantially fewer references to private guidance under projection. The training comparison in Q1 answers the separate practical question: the projected supervision also produces better downstream learning.

C. The ranking persists across checkpoints and training runs

The expanded results address the single-run and selected-peak concerns with complete reported curves. The original offline runs are:

Table 6. Original-run learning curves on the same resumed-RL checkpoint grid.

Resumed-RL stepSSRHinted DecodeAnswer-conditionedDAILDASD
1064.6860.0458.4853.9052.07
2062.7459.6559.8053.8031.76
3065.3760.2060.3856.3430.33
4066.7960.8061.0756.8939.12
5064.3262.7561.0757.0840.16
6064.6261.1161.5255.2543.43
7065.0659.8761.9953.7144.85

The original SSR run exceeds every listed baseline at every displayed step. It also exceeds continued RLVR’s reported peak of 61.9

Here are all 3-seed runs of each of the four baselines. Seed 42 is the original run.

Table 7. Hinted Decode: all three training runs.

Resumed-RL stepSeed 42Seed 43Seed 44
1060.0459.4356.97
2059.6559.4962.02
3060.2061.2959.57
4060.8062.8862.83
5062.7562.5659.95
6061.1162.8361.05
7059.8761.4557.85

Table 8. Answer-conditioned: all three training runs.

Resumed-RL stepSeed 42Seed 43Seed 44
1058.4860.0460.27
2059.8059.7258.55
3060.3860.9562.13
4061.0759.2250.96
5061.0749.3252.13
6061.5254.3851.00
7061.9958.5955.15

Table 9. DAIL: all three training runs.

Resumed-RL stepSeed 42Seed 43Seed 44
1053.9055.5956.48
2053.8056.6951.21
3056.3459.4959.44
4056.8962.7357.06
5057.0858.9159.79
6055.2562.3058.55
7053.7154.0061.24

Table 10. DASD: all three training runs.

Resumed-RL stepSeed 42Seed 43Seed 44
1052.0758.8358.18
2031.7656.3152.78
3030.3341.1450.97
4039.1258.7144.80
5040.1650.9748.70
6043.4343.5746.30
7044.8551.3947.60

At all seven reported steps, every SSR seed exceeds every one of the twelve baseline runs.

Table 11. Conservative cross-run comparison at each checkpoint.

Resumed-RL stepLowest of 3 SSR scoresHighest of 12 baseline scoresSSR margin (pp)
1061.0060.27+0.73
2062.7462.02+0.72
3065.0662.13+2.93
4066.5262.88+3.64
5064.1762.75+1.42
6064.6262.83+1.79
7063.5561.99+1.56

Table 12. Common step 70: mean ± sample standard deviation across three seeds.

MethodMacro Avg@8 (%)
SSR64.78 ± 1.12
Hinted Decode59.72 ± 1.80
Answer-conditioned58.58 ± 3.42
DAIL56.32 ± 4.27
DASD47.95 ± 3.28

OPSD’s trajectory is reported on its own update schedule:

Table 13. OPSD learning curves on the OPSD update schedule.

OPSD updateOriginal runSeed 43Seed 44
1063.5960.2964.46
2064.3464.9164.52
3061.6263.8363.28
4061.2261.1461.78
6041.4455.7856.76
800.000.000.00

All three OPSD runs peak at update 20 and decline therefore. At update 80, they all fall into catastrophic repetition and fail to provide an answer. OPSD uses its own update schedule; its axis is labeled separately from resumed RL.

D. Strong downstream results persist when the threshold is halved

We additionally completed the full downstream training pipeline at δ=0.01, alongside the original δ=0.02 run.

Table 14. Downstream threshold sensitivity: all reported checkpoints.

Resumed-RL stepSSR, δ = 0.02SSR, δ = 0.01
1064.6863.12
2062.7465.13
3065.3762.95
4066.7966.09
5064.3258.64
6064.6262.26
7065.0664.67

Halving δ gives 66.09 versus 66.79 at the peak and 64.67 versus 65.06 at step 70.

Reviewer_N3y8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that a privileged self-teacher can produce a correct reasoning trace that the unprivileged student cannot reach, so imitating it can transfer answer leakage and malformed actions. SSR keeps the model frozen and projects the steer-conditioned next-token distribution onto tokens whose unguided probability exceeds 0.02. It diagnoses a failed rollout, resumes from a high-disagreement prefix, verifies the continuation, and repeats for up to four rounds. The recovered traces are used for supervised fine-tuning and then reinforcement learning. On plateaued Qwen3 models, this search recovers verified traces for most pass@64=0 problems and improves later RL accuracy.

Critical strengths
  1. The projection comparison holds the search inputs fixed. On the same failures, prefixes, and steers, removing the support constraint raises shortcut leakage from 0.02 to 25.44 matches per rollout and lexical anchoring from 42% to 76%.

  2. The search ablations separate the main design choices. Four refinement rounds recover 147 of 197 problems, against 102 when the same 64 continuations are all spent on one round. Replacing KL intervention, the diagnostic steer, or the retained prefix each reduces first-round coverage.

Critical weaknesses
  1. The training claim is not isolated from projection. Raw guided decoding verifies 92.9% of problems in one round, while projected decoding verifies 32.0%. The paper does not fine-tune and resume RL on those raw traces under the same prefixes and budget. Downstream gains are measured against other pipelines, which also differ in coverage and objective. Answer-conditioned training on all 197 problems reaches only 62.0 aggregate accuracy, against 61.9 for continued RL, so the results do not directly show that high-leakage traces damage the student.

  2. The steer still carries much of the reference solution. It is written by the same model while looking at the private proof, and the prompt requests four to eight repair steps, replacement relations, and the first derivations. Screening removes the final answer string and some unseen numbers. On the recovered traces, lexical anchoring remains 71.13%, compared with 77.83% for the next-lowest method. A token threshold of 0.02 also makes local support much larger than the actions the student would actually sample.

  3. The reported improvement is a selected peak on a small evaluation. Table 1 uses the checkpoint with the highest aggregate accuracy. The 8B gain is 4.9 points on AIME 2025, AIME 2026, and HMMT February 2026 with eight samples, and no seed or interval is reported. OPSD is described as declining after its peak, but the curves needed to compare final rather than best checkpoints are not shown.

Questions for authors
  1. Using the same failures, steers, intervention points, and number of traces, what are SFT-plus-RL accuracy and pass@8 when the continuations are sampled from the raw guided teacher rather than the projected distribution?

  2. What fraction of a recovered trace's content is copied or paraphrased from the steer, as opposed to the retained student prefix? Please also report lexical and probabilistic anchoring of the steer itself against the private proof.

  3. How sensitive is both recovery and downstream accuracy to the floor 0.02, and does the ranking against continued RL and OPSD remain if every method is evaluated at a fixed training step rather than at its best aggregate checkpoint?

  4. Do any Int-Hard reference solutions overlap AIME 2025, AIME 2026, or HMMT February 2026? Please state the decontamination rule.

5 comments

Thank you for the specific questions. We have completed downstream comparisons and additional ablations for those questions.

Q1. Does projection improve downstream learning?

Reviewer question

Using the same failures, steers, intervention points, and number of traces, what are SFT-plus-RL accuracy and pass@8 when the continuations are sampled from the raw guided teacher rather than the projected distribution?

The point is that projected supervision produces stronger downstream learning than raw-guided supervision. With matched sft budgets, we evaluated SSR and raw-guided runs after SFT and resumed RL. The results here (table 1&2) shows that SSR benefis downstream performance more.

Table 1. Raw-guided versus projected supervision: every reported checkpoint.

RL stepRaw (seed 42)Raw (seed 43)SSR (seed 42)SSR (seed 43)SSR (seed 44)
1060.3562.2364.6861.0062.07
2062.6160.3862.7464.6264.62
3064.7758.9165.3766.9365.06
4063.3363.5566.7966.5268.83
5061.4559.7664.3264.1765.48
6063.2459.2064.6266.7866.73
7063.0960.6765.0665.7363.55

Table 2. Across-run downstream accuracy.

EvaluationSSR, 3 seedsRaw, 2 seedsSSR advantage
Common step 7064.78 ± 1.1261.88 ± 1.71+2.90 pp
Mean of each run’s peak on steps 10–7067.52 ± 1.1464.16 ± 0.86+3.36 pp

At every reported checkpoint from step 20 through step 70, every SSR run exceeds both raw runs. Even the lowest SSR peak, 66.79, exceeds the highest raw peak. We attribute this to better learning signals projected teachers can provide. Even though, in the Table 2 from the original paper, raw decoding reaches 92.89% pass@16 vs 31.98% for projection, but their explicit shortcut-pattern matches increase from 0.02 to 25.44 per rollout also. The new training experiment indicate that raw-guided decoding's higher answer-verification rate does not identify better supervision. Therefore, from the perspective of downstream performance, projected decoding is also beneficial.

Q2. What comes from the retained prefix and steer, and what does the steer contain?

We directly measured how much of the final solution was retained from the student and how much of the newly generated reasoning matched the steer. We analyzed successful raw and projected continuations generated from identical questions, prefixes, steers, and intervention points.

Table 3. Content correspondence in successful continuations.

MeasurementRawProjected
Retained prefix as a share of the complete solution19.36%13.30%
New suffix matching the steer, excluding overlaps with the question or prefix(like what we did in lexical anchoring)4.89%2.07%
New reasoning steps corresponding to the steer(annotated by external expert models)77.70%47.72%

The analysis uses one matched successful pair from each of the 60 auditable problems in the original 60-problem set. Text matching requires at least five consecutive tokens. For semantic correspondence, we used GPT-5.6 sol-xhigh to annotate. Projected continuations are longer, while the absolute amount of steer-matched text is similar. The lower percentage therefore indicates that steer-matched content occupies a smaller share of the generated reasoning.

We also measured how much information the steer itself carries from the private proof.

Table 4. Anchoring of the screened steer against the full reference proof.

MeasurementResult
Weighted lexical recall of the reference proof45.34%
Mean probabilistic anchoring10.84%
Improvement in reference-proof log-likelihood0.278 bits/token

These measurements use the same 60 screened steers provided to both decoders. Lexical recall measures overlap with the reference proof, while probabilistic anchoring measures how much the steer improves the prediction of that proof.

The steer is supposed to provide useful mathematical guidance. What matters is whether the generated solution relies on that guidance as an invisible source of authority. To examine this, we used GPT-5.6-Sol with xhigh reasoning effort to review 166 pairs of verified-correct continuations, with decoder identities hidden. The review checked whether the generated solution explicitly credited a mathematical step or conclusion to guidance unavailable in the public problem.

Table 5. Private attribution in 166 matched successful continuation pairs.

MeasurementRawProjected
Continuations flagged by the literal-phrase detector164/1660/166
Continuations flagged by GPT-5.6-Sol (xhigh) full-text review166/1666/166
Private-attribution rate, weighted equally across problems100%5.76%

Both methods received the same steer, yet their final solutions used it very differently. Raw continuations frequently referred to private guidance when justifying their reasoning, whereas projected continuations rarely did so. This difference remains clear even when we compare only verified-correct outputs generated from identical inputs.

Q3. Does the result survive common checkpoints, repeated runs, and a threshold change?

Reviewer question

How sensitive is both recovery and downstream accuracy to the floor 0.02, and does the ranking against continued RL and OPSD remain if every method is evaluated at a fixed training step rather than at its best aggregate checkpoint?

Common-checkpoint rankings

Peak selection is not needed to obtain the ranking. The comparison that list the performance of all ckpts is shown below. You can also refer to the complete 3-seed experiment we present later.

Table 6. Original-run learning curves on the same resumed-RL checkpoint grid.

Resumed-RL stepSSRHinted DecodeAnswer-conditionedDAILDASD
1064.6860.0458.4853.9052.07
2062.7459.6559.8053.8031.76
3065.3760.2060.3856.3430.33
4066.7960.8061.0756.8939.12
5064.3262.7561.0757.0840.16
6064.6261.1161.5255.2543.43
7065.0659.8761.9953.7144.85

The original SSR run exceeds all four listed baselines at every displayed checkpoint. It also exceeds continued RLVR’s reported peak of 61.9 at every displayed checkpoint; its lowest value is 62.74. Thus the advantage over continued RLVR is not confined to SSR’s selected peak.

We additionally provide every reported point for all three runs of each offline baseline. Seed 42 is the original run shown above.

Table 7. Hinted Decode: all three training runs.

Resumed-RL stepSeed 42Seed 43Seed 44
1060.0459.4356.97
2059.6559.4962.02
3060.2061.2959.57
4060.8062.8862.83
5062.7562.5659.95
6061.1162.8361.05
7059.8761.4557.85

Table 8. Answer-conditioned: all three training runs.

Resumed-RL stepSeed 42Seed 43Seed 44
1058.4860.0460.27
2059.8059.7258.55
3060.3860.9562.13
4061.0759.2250.96
5061.0749.3252.13
6061.5254.3851.00
7061.9958.5955.15

Table 9. DAIL: all three training runs.

Resumed-RL stepSeed 42Seed 43Seed 44
1053.9055.5956.48
2053.8056.6951.21
3056.3459.4959.44
4056.8962.7357.06
5057.0858.9159.79
6055.2562.3058.55
7053.7154.0061.24

Table 10. DASD: all three training runs.

Resumed-RL stepSeed 42Seed 43Seed 44
1052.0758.8358.18
2031.7656.3152.78
3030.3341.1450.97
4039.1258.7144.80
5040.1650.9748.70
6043.4343.5746.30
7044.8551.3947.60

At all seven reported steps, every SSR seed exceeds every one of the twelve baseline runs. The table below compares the lowest SSR score with the highest score among all baseline runs at each step. Every margin is positive. This is stronger than comparing either a single run or separately selected peaks.

Table 11. Conservative cross-run comparison at each checkpoint.

Resumed-RL stepLowest of 3 SSR scoresHighest of 12 baseline scoresSSR margin (pp)
1061.0060.27+0.73
2062.7462.02+0.72
3065.0662.13+2.93
4066.5262.88+3.64
5064.1762.75+1.42
6064.6262.83+1.79
7063.5561.99+1.56

Table 12. Common step 70: mean ± sample standard deviation across three seeds.

MethodMacro Avg@8 (%)
SSR64.78 ± 1.12
Hinted Decode59.72 ± 1.80
Answer-conditioned58.58 ± 3.42
DAIL56.32 ± 4.27
DASD47.95 ± 3.28

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.