acceptodds
Under review as a conference paper at ICLR 2027

Smaller Models, Better Rejects: Preference Distillation Scaling

12 views

Abstract

Preference distillation commonly treats a teacher response as preferred and the student's own response as rejected. This practice rests on two assumptions: the student's own failures are the most informative negatives, and rejects must come from a model as large as the student, which makes reject generation increasingly costly as students scale. We find that neither holds: across students from 7B to 72B, smaller frozen models generate rejects with less inference compute yet train stronger students than the student's own rejects, both before and after sequence-level knowledge distillation, on code generation and mathematical reasoning. To analyze this finding, we ask which reject distribution most improves a given student and derive a finite-horizon utility bound in a linearized feature model of DPO. The bound characterizes a favorable region of reject distributions and motivates three interventions. First, because net transfer in the bound is linear in source mixtures, we randomly mix rejects from a smaller model and from the student-scale model; performance rises as the smaller model's share grows. Second, because the bound predicts that task content independent of the prompt survives when rejects are detached from their prompts, we reassign rejects to other prompts and further shuffle their code tokens; both still outperform length-matched gibberish, so part of the gain comes from task structure rather than from errors specific to each prompt. Third, the analysis shows that selecting candidates with lower likelihood under the SeqKD reference helps when higher-likelihood candidates carry less useful contrast, as may happen for near misses that share features with correct responses. We reselect each source's candidate rejects by this likelihood; lower-likelihood selections outperform higher-likelihood ones for every source. Together, these results suggest a simple design principle: effective rejects preserve task structure while limiting coupling to the reference policy, and smaller frozen models provide both at low cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Black-Box Distillation, Preference Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xzEWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies reject construction in preference distillation and shows that smaller models can generate more effective rejects than student-scale models. It develops a linearized DPO analysis and investigates task-related structure and reference likelihood as potential explanations for this advantage.

Critical strengths
  • Relevant problem. Reject quality is an important but underexplored factor in preference distillation, with direct implications for training cost and data construction.
  • Systematic empirical study. The paper evaluates multiple student scales and two verifiable task domains, with interventions on reject sources, prompt correspondence, and reference likelihood.
Critical weaknesses
  • Limited novelty. The main finding is interesting, but the proposed interventions largely resemble existing strategies for negative sampling and preference data selection. Framing reject construction as inverse data design and introducing transfer mass and adverse share do not sufficiently establish a fundamentally new methodology or insight beyond prior work.
  • Weak connection between theory and the main finding. The linearized DPO analysis characterizes favorable reject distributions but does not explain why smaller-model rejects should be more effective than self-generated rejects. The key theoretical quantities are not directly measurable in practice, and the analysis does not provide quantitative predictions of the observed gains.
  • Insufficient causal evidence for the proposed mechanism. Lower reference likelihood correlates with better downstream performance, and likelihood-based reselection improves results within fixed candidate pools. However, likelihood may be confounded with output style, distributional distance, and error characteristics. The experiments do not sufficiently isolate reduced reference coupling as the causal driver of the overall advantage.
  • Limited generality. The experiments primarily use Qwen2.5 students on code generation and mathematical reasoning. It remains unclear whether the findings generalize across model families, training recipes, and less-verifiable tasks. The observed effects also cannot be attributed to model scale alone, given differences in model family and post-training distribution.
Questions for authors
  1. What fundamental methodological or conceptual advance does the proposed inverse data design framework provide beyond existing work on negative sampling and preference data selection?
  2. Can the theoretical framework predict, before training, which reject source will improve a given student? How do the authors establish that its key quantities explain the observed gains rather than merely describe favorable outcomes?
  3. How do the authors disentangle the effect of reference likelihood from confounding factors such as output style, distributional distance, and error characteristics?
  4. Do the findings generalize to other model families and task domains, and can the authors establish that the advantage is attributable to model scale rather than differences in training and output distributions?
Reviewer_hjAUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper challenges traditional methods in distillation that use the target model's own responses as rejects within preference datasets, and establish with reasonable empirical evidence that using rejects from smaller / weaker models improve the performance of preference optimization.

Critical strengths

The paper presents strong evidence for its main claim, and study in detail many important research questions within its established framework, with a theoretical analysis on a lineared feature model that motivates some of these additional studies.

Critical weaknesses

The paper has confusing and inconsistent terminology, among other writing issues; the theory only prompts several research problems and does not seem to directly support the main claims; the experiment settings are sometimes vague and obscuring.

Questions for authors

Questions

I group my questions into two parts. Part A covers issues the authors should address for me to maintain my score; most concern clarity and presentation. Part B covers what I see as the central limitations of the work; convincing responses to these would lead me to raise my score.

A. Would not lower the score

A1. The illustrative example in §3.1 is disconnected from the surrounding argument.

The max⁡(xs)\max(xs) / min⁡(xs)\min(xs) / max⁡(0,max⁡(xs))\max(0,\max(xs)) example (ll. 185–188) is placed between the linearization argument and the definition of δ\delta. The main text returns to it only in the final sentence of P3. Can you move the example to where it is used, as motivation for P3, with a clearer discussion?

A2. What does the decomposition into transfer mass and adverse share add beyond net transfer?

§3.2 introduces aQa_Q and cQc_Q as two coordinates of a source, and the discussion after Eq. (9) presents a trade-off between them. However, Proposition 1 depends on them only through their product κQ\kappa_Q, and so as does the favorable region. P1 and P3 are likewise stated in terms of κQ\kappa_Q (Eqs. 63, 65).

Please explain what aQa_Q and cQc_Q capture that κQ\kappa_Q does not. If they carry separate meaning, I suggest organizing §3 around them: the transition from §3.2 to §3.3 could state which values of aQa_Q and cQc_Q are desirable and derive P1–P3 as interventions that move them. Otherwise, I suggest simplifying the exposition to κQ\kappa_Q.

A3. Limitations of the linearized feature model.

The model takes ϕ\phi to be the score gradient at θ0\theta_0 and holds these features fixed, a tangent (NTK-style) linearization of the network. The appendix acknowledges this assumption (final paragraphs of §F.2.2 and §F.3). The main text, however, contains no discussion of limitations. For one, P2 relies on the decomposition ϕ=ϕdom(y)+ϕpair(x,y)\phi = \phi_{\mathrm{dom}}(y) + \phi_{\mathrm{pair}}(x,y) and on its "order-invariant part." Neither is a property of score-gradient features. Could you add a limitations discussion to the main text covering these points, and state which conclusions are expected to hold outside the linearized regime?

A4. Which sources does each figure include?

The pool of sources changes across figures without being stated. Fig. 1a ("every smaller Base source available") and the "best observed smaller Base source" in Fig. 2a/b include only Qwen2.5 sources, while Fig. 1b uses Llama-1B. Please state the source pool in each figure caption, and explain why the "best" panels are restricted to Qwen2.5. Also explain why a cross-family source was chosen for the headline mixture in Fig. 1b; the in-family Qwen2.5-3B mixture in Fig. 2c could serve this role.

A5. Terminology for models and reject sources.

The naming is inconsistent and in places contradicts standard usage:

  • "Base" refers to instruction-tuned checkpoints. Qwen2.5-0.5B-Instruct is a "Subscale Base" source (Tab. 9), and Llama-3.2-1B-Instruct is a "Llama Base" source (§C.1).
  • "Base" also denotes the student's original model in the Fig. 2 legend and Tab. 4, while Tab. 3 calls the same row "Vanilla."
  • Tab. 12 labels a smaller 7B source "vanilla 7B," although "vanilla" otherwise refers to the model at the student's scale.
  • One concept appears under at least five names: smaller Base, subscale, strict-subscale, strictly smaller Base, and Subscale Base.

Please adopt one naming scheme for student checkpoints and reject sources, avoid calling instruction-tuned models "Base", and add a short glossary table.

A6. The "Reassigned" and "Aligned" controls.

As far as I can tell, "Aligned" is simply the standard smaller-source construction: its values in Tab. 7 match the endpoints in Tab. 9. "Reassigned" is presented as Aligned with the prompt correspondence removed. Can you rename the arms (e.g., "prompt-matched" vs. "prompt-shuffled") and list every difference between them?

A7. Repulsion, attraction, and coupling.

Repulsion and attraction are defined in §3.3 only through the sign of γ\gamma in Eq. (12), within a single clause. Their operational meaning, favoring lower vs. higher reference likelihood, appears only in §4.4. "Reference coupling" is central to the abstract's design principle, yet it is never formally defined in §3. Please define repulsion, attraction, and coupling formally where they are introduced, and use one sign convention across Fig. 4a, Tab. 11, and §D.4.

B. Would raise the score

B1. Model family of the reject sources.

Whether the smaller sources come from the student's model family is central to interpreting the main result. The paper does not state it where the sources are defined (§4.1).

  • Main scaling matrix uses only Qwen2.5-Instruct sources. This can be inferred from Tab. 9, but Tab. 3 lists only sizes.
  • Llama sources appear only in App. C.1, for the 14B/32B/72B students, with a single pointer in §4.2.
  • The experiment in Fig. 1b uses the cross-family Llama-3.2-1B-Instruct before the reader has been told that Llama sources are used at all.

The cross-family evidence is also thinner than the main-text pointer suggests:

  • There is no 7B student.
  • Fig. 5 shows only MATH-500 for the math line.
  • On AIME, Llama-1B falls below both Self constructions at 32B (29.2 vs. 30.8/30.4) and at 72B (34.2 vs. 35.4/37.5) (Tab. 5).

Please state the family of every reject source in §4.1, and summarize the cross-family results in the main text. Then either qualify any claim that the result is independent of family, or support it, for example with cross-family sources for the 7B student and AIME results alongside Fig. 5.

B2. Positioning of the theory.

The paper presents its theory as supporting the main claims, but §3.3 positions it as a source of hypotheses to test. Each of P1–P3 ends with "motivates testing," and the introduction states that the analysis "motivates three interventions." Under this framing, the theory predicts secondary phenomena rather than supporting the central claim. The introduction asks why the cheaper frozen source also provides stronger negative supervision. The theory never evaluates κQ\kappa_Q for Self versus a smaller source, and model scale does not appear in the analysis.

I would appreciate one of the following:

  • (a) State precisely which claims the theory supports and which it does not, and revise the abstract, contributions, and conclusion accordingly. A careful discussion would suffice here.
  • (b) Strengthen the link between the theory and the main result, for example by deriving what the model implies for Self versus smaller-source rejects and testing that implication.

B3. The favorable region (Eq. 9) is never developed.

The favorable region is presented as a contribution in the abstract, the introduction, and the conclusion. It is also the formal object behind the claim that a reject distribution QQ outperforms Q0Q_0. Yet the paper never develops it, as after its definition, Eq. (9) is revisited only once, at the end of §F.3, to note that the region is nonempty. The authors also directly claim aQa_Q and cQc_Q are "not directly observable" (§3.3), and EHE_H is never estimated. No reject source is ever shown to lie inside or outside the region.

B4. Reference-likelihood reselection (P3) is the most practically useful result, but it is under-developed.

Within a fixed candidate bank, selecting rejects with lower likelihood under the SeqKD reference improves performance for every source (Tab. 11). This suggests a cheap, general recipe for constructing rejects, yet the paper treats it as one of three tests. I would like it discussed more explicitly and with more emphasis.

Reviewer_otoAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the role of rejected-response sources in preference distillation. The authors find that smaller frozen models can generate more effective rejects than the student's own model, while substantially reducing generation cost. This observation is demonstrated across Qwen2.5 students ranging from 7B to 72B on code generation and mathematical reasoning tasks.

To understand this phenomenon, the paper formulates reject-source selection as an inverse data-design problem and develops a linearized DPO analysis with a finite-horizon utility bound. Three experimental interventions investigate the effects of reject-source mixtures, prompt reassignment and lexical structure, and reference-likelihood-based selection. The authors conclude that effective rejects preserve task-relevant structure while maintaining limited coupling to the reference policy.

Critical strengths
  1. A meaningful and well-supported empirical finding. The paper identifies an interesting phenomenon: smaller frozen models can provide more effective rejected responses than student-generated rejects at substantially lower computational cost. The experiments cover multiple student scales and two task families, with controlled comparisons that keep the chosen responses and training configurations fixed.

  2. Useful interventions into reject-distribution properties. The prompt reassignment and lexical permutation experiments are particularly interesting. They suggest that part of the benefit of rejected responses arises from distribution-level properties that do not require exact prompt correspondence or coherent program semantics. These observations provide potentially useful directions for understanding preference optimization.

Critical weaknesses
  1. Insufficient integration with established learning-dynamics research.

The paper omits Ren and Sutherland (ICLR 2025, Learning Dynamics of LLM Finetuning), which provides a directly relevant framework for analyzing how positive and negative gradients in DPO influence other responses through feature interactions.

My main concern goes beyond the novelty of the proposed theoretical formulation. The paper identifies an interesting empirical phenomenon, but does not adequately connect it to the existing theoretical understanding of preference optimization. In particular, the learning-dynamics framework provides a natural starting point for investigating why rejects from different sources produce different downstream effects.

Instead, the paper develops a relatively limited local analysis based on initial gradient alignment and a finite-horizon utility bound, without sufficiently explaining how this analysis relates to or builds upon the established framework. Consequently, the empirical findings are not effectively integrated into a cumulative understanding of DPO learning dynamics. This substantially limits the scientific insight provided by the paper.

  1. Insufficient explanation of the main empirical mechanism and its relationship to existing findings.

The paper identifies task-related structure and limited reference-policy coupling as favorable properties of rejected responses. However, it does not adequately establish how these properties translate into beneficial learning dynamics in the student's representation space.

In particular, the finding that lower-reference-likelihood rejects improve downstream performance raises an important question in relation to Ren and Sutherland's analysis of the squeezing effect associated with negative gradients on low-probability responses. These findings are not necessarily contradictory, as the interventions and evaluation objectives differ. Nevertheless, their relationship requires explanation.

It remains unclear whether the observed benefits arise from more favorable gradient interactions, whether potentially harmful squeezing effects are outweighed by other benefits, or whether a different mechanism is responsible. Reference likelihood is an observable selection signal, but its relationship to the proposed utility-aligned feature transfer is not directly established.

As a result, the paper provides substantial empirical evidence for the phenomenon but a less convincing explanation of why it occurs. This limits its contribution to a unified scientific understanding of preference optimization.

Questions for authors
  1. Connection to existing learning-dynamics theory. How does the proposed analysis relate to the framework developed by Ren and Sutherland (ICLR 2025)? Can the authors explain how their results extend or refine the existing understanding of cross-response gradient interactions, rather than treating reject-source selection as an isolated theoretical problem?

  2. Mechanism of smaller-model rejects. Is there evidence that smaller-model rejects induce more favorable gradient interactions in the student's representation space than self-generated rejects? In particular, could the observed advantage be explained by differences in how negative gradients affect correct responses or other task-relevant outputs?

  3. Relationship to the squeezing effect. How do the authors reconcile the benefits of reference-likelihood repulsion with previous findings on squeezing effects caused by negative gradients on low-probability responses? Do the two phenomena operate under different conditions, or do they represent competing effects within the same learning dynamics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.