acceptodds
Under review as a conference paper at ICLR 2027

Learning from Unverified Peers via Shared Residual Distillation

Abstract

A language model can generate several solutions to the same problem, some leading to correct answers and others to incorrect ones. Many post-training methods turn these trajectories into learning signals using supervision from reference answers, verifiable feedback, or external teachers. Without such supervision, learning from the model’s own generations becomes less straightforward, as their correctness cannot be reliably assessed. An incorrect solution may nevertheless contain useful intermediate reasoning, so its value as supervision need not be determined solely by its final answer. Can a model improve by learning from its unverified peer solutions without first determining which are correct? We propose Shared Residual Distillation (SRD), an on-policy self-distillation method that learns from the model’s own unverified peer trajectories. SRD compares how peers change the prediction distribution relative to a control, retaining adjustments that agree in direction and are conservative in magnitude. This yields supervision without treating any complete solution as correct, and the trained model requires no peers at inference time. Starting from the corresponding base models, SRD raises mean AIME24–26 avg@12 from 11.02 to 16.20 on Qwen3-1.7B, from 19.07 to 45.46 on Qwen3-4B, and from 21.39 to 54.44 on Qwen3-8B. Controlled ablations support the value of the shared-residual construction, and we observe no degradation relative to the base models on two general-capability benchmarks. Our code is available at https://anonymous.4open.science/r/SRD-E37E/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post training, Self improvment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oNNhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a self-distillation method called Shared Residual Distillation (SRD). The method uses the model’s own multiple answer trajectories (peer trajectories for the same task). While each peer answer is unverified, the authors argue that the trajectories may contain useful local information for self-training. In particular, if multiple trajectories agree on the direction of change for the next token probabilty, in relation to a solution to a different problem, then this direction change is retained in the self-training update. They conducted experiments on SRD and compared its performance with GRPO and OPSD.

Critical strengths
  • The paper proposed an interesting self-training method. The main idea of the method is intuitive.

  • The experiments show substantial gains using this method, especially when there is clipping.

Critical weaknesses
  • I would have liked to see an experimental comparison between SRD and some other label-free distillation method (maybe U-OPSD or CoDA). This will help to understand whether it’s really the local information that makes SRD work and assess SRD’s novelty relative to other label-free methods.

  • The paper seems to demonstrate that the procedure as a whole works well experimentally. However, whether its strong agreement is due to better identification of reliable direction, or stronger shrinkage or some other choices in the experiment is unclear.

Questions for authors
  • Can you separate the effect that SRD selects more reliable directions vs. strong shrinkage avoids big intermediary mistakes? Maybe having a control in the experiment where the control matches the magnitudes of changes while randomizing their locations can help separate the effects.

  • Have you compared SRD with other lable-free distillation methods?

3 comments

Thank you for reviewing our paper and for your valuable feedback!

To W1 & Q2: Other label-free distillation methods (U-OPSD or CoDA)

Our submission already cites and discusses U-OPSD and CoDA. We share the broad question of how to construct training supervision without external labels. These two methods construct supervision using answer-level consensus, whereas SRD uniformly samples unverified peers and aggregates their local residuals without selecting a consensus solution or routing supervision by answer agreement. This distinction concerns how supervision is constructed, rather than simply whether the method uses external labels.

Both U-OPSD [1] and CoDA [2] appeared as arXiv preprints in August 2026, less than two months before the submission deadline. The ICLR 2027 reviewer guidelines [3] do not mandate comparisons with contemporaneous or arXiv-only work. We also had not found a released implementation in U-OPSD's official repository [4] as of November 10. To address your suggestion, we implemented U-OPSD from the paper and completed an additional Qwen3-4B comparison. Where details are unspecified, we use the shared SRD/OPSD setup and will document these choices explicitly. All scores are AIME avg@12 under the same evaluation protocol.

MethodAIME24AIME25AIME26AverageMean tokens
OPSD22.50%26.39%25.00%24.63%9,562
U-OPSD30.00%25.83%22.78%26.20%9,953
SRD47.78%41.11%47.50%45.46%7,537

SRD exceeds this label-free distillation baseline by 19.26 percentage points in mean AIME avg@12, with shorter average outputs.

[1] U-OPSD: https://arxiv.org/abs/2608.06296

[2] CoDA: https://arxiv.org/abs/2608.08764

[3] ICLR 2027 reviewer guidelines: https://iclr.cc/Conferences/2027/ReviewerGuidelines

[4] U-OPSD official repository: https://github.com/williamium3000/u-opsd

To W2 & Q1: Separating informative residual directions from strong shrinkage

To separate these effects, we added three controls under the same training settings:

  1. Permuted residual: At each target-token position, independently permute the SRD residual across the entire vocabulary, including all zero coordinates. This preserves the exact multiset of signed residual values while breaking their association with particular candidate tokens.
  2. Scalar mean: Replace SRD’s coordinatewise construction with the mean residual multiplied by a fixed scalar, λ=0.62\lambda=0.62. Before training, we calibrated λ\lambda on 32 preselected held-out questions and 64 fixed target trajectories, using the ratio of question-averaged, q0q_0-weighted absolute magnitudes of the SRD and mean residuals.
  3. Cross-pair: Replace both same-problem peers with the original peer pair of another problem, using a one-to-one reassignment. The donor must differ from the current problem and its cross-problem control donor. This tests whether problem–peer correspondence matters.

The following results use Qwen3-4B and report AIME avg@12.

ControlAIME24AIME25AIME26AverageDifference from SRD (pp)
Permuted residual19.72%17.78%14.17%17.22%−28.24
Scalar mean44.44%32.22%36.67%37.78%−7.69
Cross-pair19.17%21.67%17.50%19.45%−26.02
SRD47.78%41.11%47.50%45.46%—

SRD retains a clear advantage over all three controls. In particular, permuting the residual values lowers the average from 45.46% to 17.22%, despite preserving their numerical distribution. Thus, the benefit cannot be explained by the residual magnitudes or sparsity alone: their assignment to particular vocabulary coordinates matters. Uniformly shrinking the mean residual performs substantially better than permutation, but still trails SRD by 7.69 percentage points. Together with the signed-mask mean ablation in the submission, this supports retaining the full combination of shared direction and conservative magnitude selection.

The cross-pair control reaches 19.45%, close to the base model's 19.07%. Keeping the residual construction while removing the problem–peer correspondence does not reproduce SRD's improvement. This provides additional evidence that the supervision uses problem-related peer content, rather than merely the presence of extra solution text.

We will add the results and implementation details to the revision.

We would be grateful if you could let us know whether our response addresses your concerns. We are happy to discuss any remaining issues.

Thank you for the additional results. These are helpful. In particular, it's interesting to see that Scalar mean itself achieves such good performance already.

The strong scalar-mean result, together with SRD’s further gains, highlights the potential of learning from unverified peers.

Thank you again for your thoughtful feedback and constructive discussion!

Reviewer_76UuOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Shared Residual Distillation (SRD), which leverages reasoning traces (i.e., solution processes) without requiring an external teacher, verifiable rewards, or reference answers. SRD compares the probabilities under different contexts (i.e., without a solution process, with solution process 1, with solution process 2, and with a solution process from another problem) and uses the resulting signal for training. The proposed method demonstrates superior performance compared to the baselines.

Critical strengths

Constructing training signals without relying on answer correctness. This approach does not require determining whether an answer is correct or incorrect, enabling the construction of training signals from intermediate reasoning steps.

Consistent performance gains across benchmarks. On mathematical reasoning benchmarks, the proposed approach outperforms the baselines (GRPO, TTRL, and OPSD) while preserving general knowledge, as demonstrated by the out-of-distribution (OOD) experiments.

Critical weaknesses

Lack of comparison with recent baselines in self-distillation. Recent methods, such as SRPO [1] and HDPO [2], demonstrate stronger performance than OPSD, which serves as a relatively simple self-distillation baseline. To better validate the contributions of the proposed method, comparisons with these recent baselines are necessary.

Limited analysis of shared useful information and errors. If two solutions share the same errors, the training signal could be amplified in the wrong direction. However, in Table 9, the "Both incorrect" setting shows substantially better performance than the "Both correct" setting. Although the two settings use different training samples, the underlying reasons for this counterintuitive result should be further investigated. Furthermore, additional experiments using the same training dataset would help clarify whether this difference arises from the correctness of the solutions or from differences in the training data.

Strong dependence on clipping and training steps. Table 3 shows that SRD with Qwen-4B achieves an average performance of 45.46% with clipping. However, this performance drops to 22.22% when clipping is removed, as shown in Table 4. Furthermore, Figure 3(b) shows that the performance of SRD with Qwen-4B drops from approximately 45% to 30% after 50 training steps. These observations suggest that SRD may suffer from training instability. The authors should investigate and discuss the underlying causes of this instability.

[1] https://arxiv.org/abs/2604.02288

[2] https://arxiv.org/abs/2603.23871

Questions for authors

Since the proposed approach does not require reference solutions, it may be particularly advantageous in settings where verifiable rewards are unavailable, such as open-ended generation tasks. Have the authors considered evaluating SRD in such settings?

8 comments

Thank you for your detailed review!

To W1: Lack of comparison with recent baselines in self-distillation

SRPO and HDPO are interesting developments that combine reinforcement learning with verifiable rewards and self-distillation, and we appreciate the suggestion to discuss them.

However, these two works address a related but different supervision setting: SRPO routes samples using correctness feedback, while HDPO uses ground-truth information to generate and filter privileged solutions. Our study focuses on constructing supervision when these resources are unavailable. OPSD serves as a direct reference for comparing supervision constructions within the inherited distillation framework, while GRPO and TTRL provide outcome-reward comparisons.

We did not include SRPO and HDPO in the main table primarily because their required supervision differs from the resources available to SRD. They would provide additional supervised comparisons. We have also not identified released official implementations that would support a straightforward matched reproduction.

We will cite and discuss both works in the revision, clarifying their relationship to our setting. We have additionally completed a paper-based Qwen3-4B reimplementation of U-OPSD, which addresses the label-free supervision setting directly. Its step-50 checkpoint reaches 26.20% mean AIME24–26 avg@12, compared with 24.63% for OPSD and 45.46% for SRD.

To W2: Limited analysis of shared useful information and errors

SRD does not require reference answers or determine peer correctness in advance. Shared mistakes can therefore be reinforced, a risk also present in answer-consensus approaches. SRD does not assume that agreement establishes correctness. Its motivation is that final-answer correctness is too coarse to determine the value of every local prediction induced by a trajectory. A solution can contain useful intermediate reasoning even if its final answer is wrong, as illustrated by the case study in Appendix B.

As stated at the beginning of Appendix D.4, Table 9 tests whether peers with incorrect final answers can still provide useful training supervision. At each training step, we generate fresh on-policy rollouts and dynamically accept groups satisfying the two-correct or two-incorrect condition, continuing to sample questions otherwise. These conditions consequently select different training-question distributions. The two-incorrect result demonstrates that SRD remains effective with incorrect-answer peers.

To better understand the relative performance of the two conditions, we added two analyses.

1. Same-question comparison. We generated 16 solutions per question and retained questions with at least two correct and two incorrect solutions. For each retained question, we selected either two correct or two incorrect trajectories as the peer contexts for SRD training. Both conditions therefore use the same selected questions. The AIME avg@12 results are:

Peer conditionAIME24AIME25AIME26Average
Two correct43.61%41.39%42.22%42.41%
Two incorrect36.11%34.44%32.50%34.35%

On the same questions, correct peers yield the higher score, with an 8.06-point advantage. Incorrect peers still produce a substantial improvement over the base model's 19.07%. These results support both the value of more reliable peer information and SRD's ability to benefit from incorrect-answer trajectories.

2. Training-question difficulty. We revisited the 1,600 accepted groups from each Table 9 condition and generated 16 fresh solutions per question using the same base model. Pass@16 is 98% for the two-correct condition's questions, versus 61% for the two-incorrect condition's questions, revealing a substantial difference in their initial solvability.

We then pooled the 3,200 problem instances and used the number of correct responses among the 16 rollouts to construct three difficulty subsets: A, high success rate; B, intermediate success rate; and C, low success rate. Difficulty is assigned using the initial base model; subsequent SRD training samples peers without correctness-based filtering:

SubsetAIME24AIME25AIME26Average
A: High success rate38.61%34.72%35.27%36.20%
B: Intermediate success rate42.50%37.22%44.72%41.48%
C: Low success rate41.67%38.33%44.16%41.39%

The intermediate- and low-success subsets yield higher downstream scores than the high-success subset, by 5.28 and 5.19 points, respectively. This supports greater training benefit from questions the base model has not yet reliably mastered. Together, the two analyses support training-question selection as a contributor to the Table 9 ranking: correctness filtering changes both peer reliability and the questions available for learning.

To W3: Strong dependence on clipping and training steps

Clipping. We inherit OPSD's clipping settings and apply the same objective to both methods for a controlled comparison. Table 4 additionally reports the complete unclipped comparison, testing whether SRD's advantage persists when this component is removed.

SRD exceeds the corresponding OPSD baseline under both objectives at every model size. Clipping is important to the strongest results, but its presence alone does not explain the advantage of SRD's supervision construction. Removing clipping changes the optimization objective; the resulting performance difference is not, by itself, evidence of numerical instability.

The new sign-restricted controls further test the supervision construction under unchanged training settings: positive-only residuals reach 33.70%, negative-only 15.46%, and full SRD 45.46%. Thus, the complete signed residual provides greater benefit than either restricted variant under the same clipped objective.

Training duration and checkpoint selection. We held out 300 problems from the training dataset for validation, excluding them from parameter updates, and saved and evaluated checkpoints every 25 steps. For both OPSD and SRD, validation performance plateaued or declined after step 50 across all three model sizes. This informed the common step-50 checkpoint used for both methods across all main-table benchmarks.

The reported curves show that, between steps 50 and 100, 4B SRD decreases from 45.46 to 31.85, while OPSD decreases from 24.63 to 21.48. SRD remains above both OPSD and the base at step 100. At 8B, SRD changes only from 54.44 to 53.61, retaining most of its improvement. Late-stage instability is observed in both OPSD and SRD; its underlying cause remains unclear.

To Q1: Open-ended generation tasks

Open-ended generation is a natural setting for this direction, which our limitation section identifies as future work. SRD's peer sampling and residual aggregation do not require an extractable final answer or a correctness verifier, making the approach applicable beyond answer-level voting.

We have now explored this setting using RaR-Medicine [1] for training and HealthBench Consensus [2] for evaluation, with deepseek-v4-flash as the judge. We are pleased to share the following results:

ModelOverallCommunicationContext awarenessCompleteness
Qwen3-4B82.1587.6771.3082.75
SRD84.5691.1273.2385.08

SRD raises the overall score by 2.41 points, with improvements in communication (+3.45), context awareness (+1.93), and completeness (+2.33). These results provide initial evidence that SRD's benefits extend to open-ended responses beyond mathematical reasoning.

[1] RaR-Medicine: https://huggingface.co/datasets/ScaleAI/RaR-Medicine

[2] HealthBench Consensus: https://arxiv.org/abs/2505.08775

We would be grateful if you could let us know whether our response addresses your concerns. We are happy to discuss any remaining issues.

Thank you for the detailed responses and the additional experiments. These results help clarify several points. However, I still have two remaining concerns.

1. Comparison with recent baselines.

I understand that SRPO and HDPO require correctness feedback or ground-truth information, whereas SRD does not. However, the training dataset used in this paper contains reference solutions, and the paper already compares against reference-conditioned OPSD and answer-supervised GRPO. Therefore, the difference in supervision requirements does not seem sufficient to exclude SRPO and HDPO.

I appreciate the implementation challenges. However, even if HDPO is not included because it remains a preprint, SRPO is a published COLM 2026 paper that was publicly available in April 2026. A comparison with SRPO would help establish the performance trade-off associated with removing reference supervision.

More importantly, could the authors provide concrete examples of practical fine-tuning datasets or scenarios where reference answers are unavailable? Since the experiments use a dataset with reference solutions but deliberately omit them during SRD training, clarifying the intended real-world use cases would help establish the practical motivation for this setting.

2. Understanding why incorrect peers provide useful supervision.

Thank you for conducting the additional experiments.

However, this establishes that incorrect peers can be useful, rather than explaining why. The possibility that an incorrect solution contains useful intermediate reasoning is plausible, but it does not explain whether or how SRD preferentially transfers that information rather than shared errors. The method itself does not distinguish correct shared reasoning from shared mistakes.

Since extracting useful supervision from unverified peers is the paper’s central insight, I believe this point warrants a more detailed analysis.

Reviewer_dVGiOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SRD is a label-free on-policy self-distillation method for math reasoning. For each problem, the student samples 4 rollouts: 2 serve as peer contexts and 2 as targets. A frozen copy of the initial model scores each target prefix three ways: with the student prompt (q0q_0), with each peer in context (qjq_j), and with another problem's rollout in context (qcq_c). Each peer's residual log⁡(qj/qc)\log(q_j/q_c) is centered by its q0q_0-weighted mean. A minmod rule combines the residuals into rr: it keeps the smallest magnitude if all peers agree in sign, and zero otherwise. The target q∩∝q0erq_\cap \propto q_0 e^{r} is distilled with OPSD's clipped forward-KL surrogate. On Qwen3 1.7B/4B/8B in non-thinking mode, mean AIME24–26 avg@12 rises 11.02→16.20, 19.07→45.46 and 21.39→54.44. SRD beats GRPO, TTRL and OPSD with no loss on IFEval or MMLU-Pro.

Critical strengths

S1. A good question, a simple method, large gains. Learning from unverified solutions without choosing a pseudo-label is a cleaner setup than answer-consensus methods (TTRL, U-OPSD, CoDA). SRD adds a few lines to OPSD and costs nothing extra at inference. Its gains include AIME 2026 and HMMT Feb 2026, which post-date Qwen3, so base-model memorization is unlikely to explain them.

S2. Transparent, broad experiments. The paper ablates the aggregation rule, residual reference, clipping, KK, MM and peer correctness, and gives exact prompts and decoding settings. That makes every concern below testable.

Critical weaknesses

W1. The gains depend on a clipped surrogate that does not distill toward q∩q_\cap. Without the cap (Table 4), SRD gains only +1.0/+3.2/+3.3 mean AIME over base (1.7B/4B/8B). With the cap, it gains +5.2/+26.4/+33.1, so 80–90% of the effect requires the cap.

Setup: student p=softmax(z)p=\mathrm{softmax}(z), detached target qˉ=sg[q∩]\bar q=\mathrm{sg}[q_\cap], capped set A={v:qˉ(v)log⁡qˉ(v)p(v)>τ}A=\lbrace v:\bar q(v)\log\frac{\bar q(v)}{p(v)}>\tau\rbrace, and BB the remaining tokens. Eq. (9) then gives

∂ℓτ∂zk=pkqˉ(B)−qˉk1[k∈B].\frac{\partial \ell_\tau}{\partial z_k}=p_k\bar q(B)-\bar q_k\mathbf{1}[k\in B].

Consequences:

  • The gradient never raises a capped token's logit.
  • For a fixed partition, the minimizer qˉ(⋅∣B)\bar q(\cdot\mid B) assigns capped tokens zero mass.
  • Positions where the target is confident (small qˉ(B)\bar q(B)) contribute almost nothing.

At step 0, p=q0p=q_0, and τ=10−6\tau=10^{-6}. The capped set is then essentially {v:r(v)>log⁡Z}\lbrace v:r(v)>\log Z\rbrace, where ZZ normalizes q0erq_0e^{r}; these are exactly the shared boosts. In Appendix B, "7" is capped (q∩/q0≈5.5q_\cap/q_0\approx5.5) and can rise only because "8" is pushed down. The OPSD README confirms these settings, so the SRD-vs-OPSD comparison is fair. But the paper's mechanism, peers jointly boosting useful tokens, is not what gets optimized.

W2. Nothing shows the gains come from peer content or from agreement.

  • Elicitation confound, no null control. Qwen3's thinking mode is far stronger: U-OPSD reports 4B AIME24 of 74.2 in thinking mode vs 25.8 in non-thinking mode, and all methods gain only about 2 points in thinking mode. SRD trains on just 1,600 problems (≈3M loss tokens, first 1,024 tokens only), yet responses grow 61%. Its teacher prompt asks the model to "explore" and "backtrack". Every peer-conditioned variant scores ≥33.9 (base 19.07), and two wrong peers (43.43) beat two correct ones (39.07). Eliciting thinking-mode behavior explains this as well as learning from peers does.
  • Sign agreement adds nothing measurable. Signed-mask mean (37.50) scores no better than random masking at matched sparsity (38.61). Only min-magnitude shrinkage helps (45.46). Agreement is also inflated, because every δj\delta_j subtracts the same log⁡qc\log q_c: only 24.98% of coordinates are zeroed at M=2M=2, vs ~50% for independent signs. The sign test also depends on an arbitrary centering choice.
  • One seed, read at the peak. Step 50 is 4B SRD's peak (45.46 → 31.85 by step 100), and the paper offers no label-free stopping rule.

Other issues

  • Baselines:
    • U-OPSD uses the same data and recipe; it is cited but not compared.
    • GRPO and TTRL use LoRA at LR 10−610^{-6}, about 10× below LoRA's typical optimum. GRPO gains only +2.8 at 4B.
    • Entropy-minimization and self-certainty baselines are missing.
    • At 8B, the OPSD reproduction trails its README (42.2 vs 49.7 AIME24), and Table 10's original K=1 OPSD beats the variant reported (36.1 vs 33.3).
  • Evaluation:
    • OOD covers only two single-sample greedy benchmarks.
    • MATH-500 uses one sample (SE ≈ ±1.6).
    • HMMT25 mixes the Feb and Nov contests.
    • The contamination check is undescribed.
    • ~44% of 1,024-token rollouts are truncated, so many peers are partial solutions.
    • Table 9 confounds peer correctness with problem difficulty.
    • Compute is unreported.
  • Overclaims: "Conservative" holds only in log-ratio space; in Appendix B, "7" goes 18% → 99.8%. "Gains extend to HMMT" fails at 1.7B HMMT25 (5.97 → 6.11).
  • Related work: Missing: context distillation (Askell et al., 2021; Snell et al., 2022; Ye et al., 2026), SDFT/SDPO (2026), label-free RL, spurious rewards (Shao et al., 2025), opinion pooling, and minmod's origin (Roe, 1986). Add U-OPSD, CoDA and TTPO to Table 1.
  • Presentation:
    • Eq. (2) disagrees with Table 6 on the clip range and KL term.
    • Fig. 2 shows a plain KL.
    • Configs are given only for 4B.
    • "EXPERIMENT" → "EXPERIMENTS"; "while separately conditioned" → "while being separately conditioned".
    • App. D.4 omits the two-correct result (39.07).
Questions for authors

Clarifications

  1. Is each per-coordinate summand clamped at τ\tau as in Eq. (9)? What fraction of target mass is capped per position?
  2. Does the student actually move toward q∩q_\cap, e.g. does KL(q∩∥pθ)\mathrm{KL}(q_\cap \Vert p_\theta) fall during training?
  3. How was step 50 chosen, and on what data? What are the results at step 100, and how many seeds were run?
  4. Why does gold-reference OPSD (24.63) trail plain two-peer averaging (35.93)?
  5. Is the prompt-only teacher (Table 8) used as a direct target or as a residual?
  6. Please report accuracy and truncation rate at 4k/8k/16k/32k generation budgets for every method. With mean lengths of 7,537 (SRD) and 9,562 (OPSD) tokens, the 32k budget may drive part of the ranking.

Suggestions

  1. A τ\tau sweep up to no cap, and positive-only vs negative-only rr.
  2. Null peers: cross-problem, empty, or shuffled.
  3. Thinking mode, and one non-hybrid model.
  4. A scalar-shrinkage baseline r=λδˉr=\lambda\bar\delta, with δˉ=1M∑jδj\bar\delta=\frac1M\sum_j\delta_j, plus a centering ablation.
  5. At least 3 seeds with CIs, a label-free stopping rule, and U-OPSD under the same protocol.
  6. Does SRD help a current hybrid family such as Qwen3.5 or Gemma 4 in thinking mode? There, traces run ~30k tokens (Qwen3.5-27B on AIME 2026, per MathArena), and the 1,024-token scoring window covers only their opening.
6 comments

Thank you very much for your patience and detailed review!

To W1, Clarifications 1–2, and Suggestion 1: Clipping and the contribution of shared-residual supervision

We agree with your gradient analysis: pointwise clipping does not preserve ordinary forward-KL attraction toward q∩q_{\cap}. We identified this issue during the study, explicitly described the inherited surrogate in the main text, and included the complete unclipped comparison in Table 4. Our claim concerns the effectiveness of the shared-residual supervision construction under the specified objective, rather than a guarantee of coordinatewise target alignment.

The same mathematical effect applies to reference-conditioned OPSD. We hold its objective fixed when comparing supervision constructions. The detached target's summands av=qˉ(v)log⁡(qˉ(v)/pθ(v))a_v=\bar q(v)\log(\bar q(v)/p_\theta(v)) are upper-capped individually before vocabulary summation and token averaging, with τ=10−6\tau=10^{-6} for 1.7B/4B and 10−710^{-7} for 8B. In a fixed-prefix diagnostic, ordinary forward KL increases while the surrogate decreases. The unclipped experiments separately test SRD under ordinary forward KL.

Using the actual native BF16 clipping mask at τ=10−6\tau=10^{-6}, the mean teacher probability mass on capped coordinates over valid prediction positions was 32.33%, 35.54%, 23.82%, 12.50%, and 12.87% at updates 1, 2, 10, 25, and 50, respectively, measured on the actual on-policy training batches.

Clipping clearly contributes substantially, but the performance lost upon removing it cannot be attributed to clipping independently of the supervision construction:

ModelOPSD without clipSRD without clipOPSD with clipSRD with clip
Qwen3-1.7B10.56%12.04%12.04%16.20%
Qwen3-4B19.81%22.22%24.63%45.46%
Qwen3-8B22.41%24.72%33.33%54.44%

All entries are mean AIME24–26 avg@12. SRD exceeds OPSD under both objectives at every model size. On 4B, the advantage is 2.41 percentage points without clipping and 20.83 points with clipping. Without clipping, mean pass@12 also rises from OPSD's 36.67% to SRD's 52.22%. Neither clipped OPSD nor unclipped SRD reproduces the combined avg@12 result.

Following your suggestion, we additionally retained only the positive or only the negative entries of the shared residual, leaving the remaining training settings unchanged.

ControlAIME24AIME25AIME26Average
Positive-only40.00%30.55%30.55%33.70%
Negative-only20.00%13.89%12.50%15.46%
SRD47.78%41.11%47.50%45.46%

The full signed residual achieves the highest mean score, exceeding positive-only by 11.76 percentage points and negative-only by 30.00 points. This supports combining both signed components under the same clipped objective.

We will also correct the plain-KL notation in the overview figure and clarify the case study. “Conservative” describes the centered log-residual magnitude bound, not a bound on normalized probabilities.

To W2.3 and Clarification 3: Checkpoint selection and robustness

Checkpoint selection. We held out 300 problems from the training dataset as a validation set, excluding them from parameter updates. During training, we saved and evaluated checkpoints every 25 steps. For both OPSD and SRD, validation performance plateaued or declined after step 50 across all three model sizes (1.7B, 4B, and 8B). We therefore selected step 50 for both methods in the main table, using the same checkpoint across all evaluation benchmarks. GRPO and TTRL are reported at step 300.

The full curves also report subsequent behavior: 4B SRD declines from 45.46 to 31.85 between steps 50 and 100, while remaining above OPSD at step 100 (21.48) and the base (19.07). At 8B, SRD retains most of its improvement, moving from 54.44 to 53.61. Late-stage instability is observed in both OPSD and SRD; its underlying cause remains unclear.

Multiple training seeds. We also evaluated Qwen3-4B SRD with three training seeds:

Training seedAIME24AIME25AIME26Average
4247.7841.1147.5045.46
4340.8340.5642.5041.30
4446.1137.5043.8942.50
Mean ±\pm s.d.44.91±3.6344.91\pm3.6339.72±1.9539.72\pm1.9544.63±2.5844.63\pm2.5843.09±2.1443.09\pm2.14

SRD achieves 43.09% mean accuracy across seeds, with individual averages ranging from 41.30% to 45.46%. All three runs retain substantial gains over the Qwen3-4B base (19.07%).

To W2.1–2.2 and Suggestions 2 and 4: Peer content, shrinkage, and null controls

We have added four controls under the same training settings:

  • Permuted residual: independently permute the residual across the entire vocabulary at each target position, preserving all signed values and zeros.
  • Scalar mean: scale the mean residual by a fixed λ=0.62\lambda=0.62, calibrated before training on 32 preselected held-out questions and 64 fixed targets using the ratio of question-averaged, q0q_0-weighted absolute residual magnitudes.
  • Without centering: remove only the weighted-mean subtraction before minmod aggregation, leaving the other steps unchanged.
  • Cross-pair: reassign another problem's original peer pair one-to-one, excluding both the current problem and its control donor. Preserve the original cross-problem control assignment, target trajectories, prompts, and loss.
ControlAIME24AIME25AIME26Average
Permuted residual19.72%17.78%14.17%17.22%
Scalar mean44.44%32.22%36.67%37.78%
Without centering48.61%34.72%39.72%41.02%
Cross-pair19.17%21.67%17.50%19.45%
SRD47.78%41.11%47.50%45.46%

Token-specific placement matters: permutation removes the gain despite preserving the residual-value distribution. Calibrated uniform shrinkage still trails SRD, as does the submitted signed-mask mean control (37.50%). Together, these results favor the full direction-and-magnitude construction over the tested alternatives.

Centering ablation: centering improves mean avg@12 from 41.02% to 45.46%. This supports the weighted-mean reference used for the sign comparison before aggregation.

Problem-related peer content matters: cross-pair reaches 19.45%, close to the base model's 19.07%, whereas SRD reaches 45.46%. With the residual pipeline and control assignment preserved, this comparison supports the importance of matching peer content to the current problem.

To Clarification 6: Output budgets

Short-output diagnostic. We truncated the existing 32,768-token evaluation outputs at earlier token cutoffs and rescored the retained prefixes. The table averages AIME24–26; truncation rates include outputs cut at the corresponding limit.

Retained tokensOPSD avg@12SRD avg@12OPSD truncationSRD truncation
4,09621.29%31.48%33.70%54.72%
8,19223.24%40.93%23.24%32.41%
16,38424.63%45.28%21.58%13.79%
32,76824.63%45.46%20.83%7.96%

SRD already leads by 10.19 points at 4k and remains ahead at every cutoff. At 16k, it reaches 45.28%, nearly matching its 32k score of 45.46%. SRD's advantage is already visible in shorter retained prefixes, with little additional accuracy beyond 16k.

To Suggestions 3 and 6: Thinking-mode transfer

Thinking-enabled Qwen3.5-4B. We additionally tested SRD and OPSD with thinking enabled, following OPSD's thinking-mode training setup. The evaluation uses the same mathematical scoring protocol and sampling parameters as the Qwen3-4B comparison, with thinking enabled. In this setting, we use a 4,096-token training window to give peer trajectories more room to develop their reasoning beyond the opening of the thinking segment.

MethodAIME24AIME25AIME26AverageMean tokens
Qwen3.5-4B67.22%59.44%63.61%63.42%24,316
OPSD38.89%37.78%33.61%36.76%28,259
SRD73.06%62.22%63.33%66.20%22,412

SRD improves mean avg@12 from 63.42% to 66.20% on an already thinking-enabled base. This extends the positive result beyond the original non-thinking setting and shows that SRD's benefit can persist without longer generated reasoning.

To Clarification 4: Reference answers versus peers

Correct reference answers do not by themselves guarantee the most effective token-level supervision along student-generated prefixes. In our comparison, two-peer probability averaging reaches 35.93% versus reference-conditioned OPSD's 24.63%; SRD further improves this to 45.46%.

To Clarification 5: Prompt-only teacher as a direct target

We use the prompt-conditioned teacher distribution as a direct target. Instructions encouraging independent reasoning, exploration, and backtracking, without peer trajectories, reduce mean AIME avg@12 from the base model's 19.07% to 6.57%; these instructions alone do not reproduce SRD's gains.

Other issues

U-OPSD comparison.

Our submission already cites and discusses U-OPSD and CoDA. Both construct supervision through answer-level consensus, whereas SRD uniformly samples unverified peers and aggregates their local residuals without selecting a consensus answer or routing supervision by answer agreement.

U-OPSD [1] and CoDA [2] appeared as arXiv preprints in August 2026, less than two months before the submission deadline. The ICLR 2027 reviewer guidelines [3] do not mandate comparisons with contemporaneous or arXiv-only work. We also had not found a released implementation in U-OPSD's official repository [4] as of November 10. To address your suggestion, we implemented U-OPSD from the paper and completed a Qwen3-4B comparison. Where implementation details are unspecified, we use the shared SRD/OPSD setup and will document these choices explicitly. We report its step-50 checkpoint using the same AIME avg@12 evaluation protocol.

MethodAIME24AIME25AIME26AverageMean tokens
OPSD22.50%26.39%25.00%24.63%9,562
U-OPSD (our reimplementation)30.00%25.83%22.78%26.20%9,953
SRD47.78%41.11%47.50%45.46%7,537

SRD exceeds this label-free distillation baseline by 19.26 percentage points, with shorter average outputs.

[1] U-OPSD: https://arxiv.org/abs/2608.06296

[2] CoDA: https://arxiv.org/abs/2608.08764

[3] ICLR 2027 reviewer guidelines: https://iclr.cc/Conferences/2027/ReviewerGuidelines

[4] U-OPSD official repository: https://github.com/williamium3000/u-opsd

Reproduction. For OPSD, we followed the official reproduction script and evaluated three seeds, but did not reproduce its highest reported score. Main-table OPSD matches SRD's scored-target count for a controlled comparison; Table 10 also reports original K=1K=1 OPSD. At 8B, SRD's 54.44 remains 18.33 points above the stronger original setting (36.11).

For GRPO and TTRL, checking the training scripts and logs confirmed a typo in the reported learning rate. The actual learning rate was 5×10−65\times10^{-6}, matching OPSD's GRPO training configuration. We apologize for the error and will correct the table.

Peer correctness and truncation. Table 9 tests whether peers with incorrect final answers can provide useful supervision. In an additional same-question comparison, two correct peers achieve 42.41% and two incorrect peers 34.35%, both above the base model's 19.07%. The original correctness filters also select questions of different difficulty: base-model pass@16 is 98% on the two-correct subset and 61% on the two-incorrect subset. Our main results use a 1,024-token limit and show that SRD can learn from partial trajectories; we extended rollouts for the correctness analysis to obtain final answers.

Contamination check. We compared training and evaluation problem texts character by character and found no exact matches.

Related work and presentation. Thank you for your careful suggestions; we will update these sections in the revised version.

Thank you again! We would be grateful if you could let us know whether our response addresses your concerns. We are happy to discuss any remaining issues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.