acceptodds
Withdrawn from ICLR 2027

Preference-Based Self-Distillation with Online Contextual Comparisons

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Privileged context gives a self-teacher information unavailable to its student. The challenge is to turn this information into effective supervision. We introduce Preference-Based Self-Distillation (PBSD), which turns contextual guidance into online comparisons between complete teacher and student responses. A frozen contextual teacher serves two roles: generating the positive response and providing the likelihood reference for scoring both responses. The negative response is sampled from the current student, so the comparisons evolve with its behavior. PBSD optimizes a pairwise logistic objective with labels assigned by generation role. The student receives no privileged context, and training requires no separate preference judge. At each of three Qwen3 scales, PBSD achieves the highest average accuracy across the mathematical benchmarks and the highest tool-call accuracy among the compared methods. On Qwen3-4B, it reaches 64.5% mathematical average, compared with 60.7% for imitation of the same teacher responses, 60.8% with fixed negatives, and 59.3% with an unconditioned reference. These comparisons support the joint use of online responses and contextual reference scoring in the tested setting. Context ablations and pair-correctness diagnostics further characterize the supervision. Together, these results support using contextual self-teachers to construct and score online response comparisons.

Primary area:
generative models
Keywords:
self-distillation, preference learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PuxuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Preference-Based Self-Distillation (PBSD) for contextual self-distillation. As in OPSD, a teacher is the base model conditioned on privileged context (e.g., a reference solution), and the student sees only the problem. PBSD replaces OPSD’s token-level distribution matching with a DPKD-style length-normalized pairwise logistic loss. The positive is a (stored) teacher response, the negative is a fresh on-policy student response, and the frozen contextual teacher supplies the reference log-probabilities. Labels come from generation role, not correctness, and no judge or verifier is used. On Qwen3-1.7B/4B/8B, PBSD improves over OPSD and other baselines on math Avg@12 (e.g., 64.5 vs. 63.5 at 4B) and tool-call accuracy (+6 to +7 points). At 4B, ablations show lower scores for teacher-response SFT (60.7), static-negative DPO (60.8), and an unconditioned reference (59.3), all below the untrained base (60.9). Further analyses cover privileged-context variants, pair-correctness composition, response length and hedging, OOD retention, and a longer-horizon run. The appendix adds a reward-tilted interpretation and a conditional local estimation bound.

Critical strengths

Simple, practical method with honest scoping. PBSD needs one teacher and one student response per prompt and no verifier or group rollouts, and it needs no context at inference. The authors are unusually candid about limits: labels are roles rather than correctness, ablations are not at fixed texts, and the theory is a surrogate. They also report the MBPP+ regression (-4.73) and the post-step-500 degradation. Broad empirical coverage. Three scales, math and tool use, plus code, IFEval, OOD checks, and a β sweep. The pair-correctness diagnostic (Table 5) is a useful and rarely reported view of what role-labeled supervision contains.

Critical weaknesses
  • The gain over OPSD is about 1.0 Math Avg. point at 4B/8B on benchmarks with 30 questions each. The tool-use test set has 94 examples, so each example is about 1.06 points. All reported standard deviations come from evaluation-sampling repeats of one trained checkpoint. There is no variance across training seeds, so significance is unknown. All ablations are single 4B runs.
  • β = 0.1 is chosen from a sweep reported on the same test benchmarks. Step 500 is chosen where PBSD is near its best (Appendix F.10 shows 4B/1.7B degrade later), while the selection protocol is ambiguous.
  • PBSD vs. OPSD changes both the loss and the response construction (an extra teacher generation). OPSD’s original setup used 100 steps and, per Fig. 2A, OPSD declines after step 200. Reporting the “latest checkpoint at 500” therefore disadvantages OPSD. Compute is not matched against GRPO/DAPO.
  • All three ablated variants fall below the untrained base, including plain SFT on the same teacher responses. This suggests pipeline brittleness (e.g., 1,024-token truncation vs. 38,912-token thinking-mode evaluation, with truncated pairs kept in the loss). Why the same responses help as pairwise positives but hurt under SFT is unexplained. The labeling scheme is not shown to carry useful signal. Student responses are negatives by construction, even when correct. By Table 5,
Questions for authors
  1. How were step 500 and β = 0.1 selected, and was any selection done on the reported test benchmarks? Can you compare against OPSD at its best/recommended checkpoint (e.g., 100 steps) and with tuned hyperparameters?
  2. What happens when training only on teacher-correct/student-incorrect pairs, or when excluding/reversing both-correct pairs? What is the distribution of σ(−m_i) across the Table 5 categories?
  3. Why do teacher-response SFT and static-pair DPO score below the base model? Is it truncation at 1,024 tokens or the thinking-mode mismatch? Does PBSD hold up with longer training completions?
  4. With ~30K questions and 500 steps × 32 = 16K samples, how often is a stored teacher response actually reused?
  5. How does PBSD compare to AntiSD, the closest recent fix for OPSD’s degradation, and under a compute-matched budget against GRPO/OPSD?
Reviewer_AVVQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary: This paper replaces the token-by-token distribution matching in OPSD with DPKD-style sequence-level pairwise preference loss. Positive examples are derived from frozen self-teachers with privileged contexts, while negative examples are derived from online sampling of the current student. The reference model is also the self-teacher. Improvements are achieved over OPSD on three scales of mathematical reasoning and tool invocation tasks in Qwen3.

Critical strengths

Strengths

The motivation is clear. The phenomenon of token-by-token matching suppressing uncertain expressions is worthy of further study; the method is simple to implement and does not require external teachers or reward models.

The experiments are broadly comprehensive: 3 model sizes and 6 baselines, covering mathematics, tool calls, code, and instruction compliance. Additional features include context ablation, training curves, length and self-test word frequency, and pairing correctness diagnostics.

The statements regarding the boundaries of conclusions are relatively honest, explaining in several places which conclusions cannot be derived.

Critical weaknesses

Weaknesses

Limited novelty.

The objective function is DPKD (acknowledged by the authors), and the teacher construct is OPSD; the main contribution is the combination of the two.

The approach of "using reference or labeled answers as positive examples and model-generated examples as negative examples" is highly similar to SPIN (Self-Play Fine-Tuning, Chen et al., ICML 2024).

"Generating inference given an answer" is also closely related to STaR's rationalization and context distillation (Snell et al., 2022). These works are not discussed or compared.

The main result comparison protocol is unfavorable to OPSD.

The original OPSD paper only trained for 100 steps; this paper uniformly uses the 500th step.

Figure 2(A) shows that OPSD drops significantly after 200 steps. Comparing their respective best checkpoints, the peak scores on AIME25 differ by only about 0.3 points.

The report needs to show the best or early stopping results, or at least compare them under the original OPSD settings (100 steps).

Statistical reliability is insufficient.

All training was done in a single run; the reported standard deviation only reflects the fluctuation of the evaluation sampling, not the fluctuation of the training itself.

Approximately 30 questions each for AIME and HMMT, 1 point is roughly equal to 0.3 questions; the tool test set has 94 questions, 1 question is roughly equal to 1.06 points.

A mathematical improvement of around 1 point cannot be determined to be significant.

Ablation is uncontrolled.

Each variant "independently regenerates the teacher's answer," but could actually share the same cache.

All three ablation variants are worse than the original model, and changing only "whether the reference model includes 'c'" results in a difference of 5.2 points. This sensitivity itself needs explanation, and is even less convincing under single-seed conditions.

The properties of the objective function are not analyzed.

The reference model and student input conditions differ: the teacher's input is π_teach(y|x,c), and the student's input is π_θ(y|x). Therefore, even during initialization (θ = θ0), the margin is not zero.

The teacher's answer has a higher probability of being under the teacher's input, and the student's answer has a higher probability of being under the student's input, so the initial m is likely to be a large negative number, and the sigmoid weights are saturated.

It is recommended to report the distribution of m and σ(−m) during training.

The training and evaluation settings are inconsistent, and the learning content is unclear.

Training generates tokens truncated to 1,024, but the evaluation uses the thinking mode with a maximum of 38,912 tokens, and the length diagnostic uses an upper limit of 2,048. Whether Thinking was enabled during training is not specified.

In 54% of the pairs, students answered correctly but were treated as negative examples; the authors do not explain what the model actually learned. It may primarily be imitating the style of OpenThoughts' reference inference.

These reference inferences themselves are generated by stronger inference models, so the claim of "no external teacher needed" is questionable.

The theoretical part is disconnected from the methodology, and the writing is redundant.

The conclusions in Appendix D/E (optimal solution under KL regularization, Jensen's inequality, local M-bound estimation under fixed distribution) are general conclusions and have no direct connection to the actual length normalization and adaptive sampling objectives. The authors themselves repeatedly state that they cannot explain performance and suggest compression.

The main text contains a large number of repetitive qualifiers, making it difficult to read.

Questions for authors

See Weaknesses

Reviewer_uSwQOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PBSD, a method that pairs responses from a fixed-context teacher model with those from an online student model, computes the difference in their likelihoods under the teacher model, and updates only the student model using a pairwise logistic loss. After training and evaluation, PBSD shows some improvement over OPSD on multiple benchmarks.

Critical strengths
  1. The paper's approach is simple and easy to understand, requires no reward model or external evaluator, and has practical value.
  2. In experiments, the current method shows some improvement over OPSD on multiple benchmarks.
Critical weaknesses
  1. The paper offers limited novelty. Specifically, its loss function is essentially that of DPKD, with the external teacher replaced by the current model conditioned on a fixed context. The method itself makes only a limited contribution, and the paper’s value is limited.

  2. The paper provides little analysis of the current model’s responses when given the answer. If the model does not know how to solve the problem and is only told the answer, the correctness of its output needs further examination. Why can the output of a model given only the answer be treated as a teacher? Could this increase hallucinations in the model’s output?

  3. The paper is not clearly written. I do not think it is appropriate for Figure 1, which serves as the paper’s main introductory figure, to be generated entirely by AI (including its text). Nor do the authors explain the purpose of each column in Table 1 (Online student responses, external teacher ...) or why their particular combination is the most effective.

Questions for authors

See Weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.