acceptodds
Under review as a conference paper at ICLR 2027

Compile2Decide: Distilling Frontier Reasoning Models into Fast Calibrated Decision Models

14 views

Abstract

Large reasoning models have broad semantic competence but are computationally inefficient for repeated bounded decisions. We ask how much of their judgment can be compiled into a much smaller System One decision model that scores a dynamic candidate set directly, without generating explanations or reasoning traces, and can answer “none of the listed options” when the correct answer is absent. A reasoning-capable LLM teacher (, queried once per input with thinking disabled) scores the full label space plus none; its distribution is projected onto each run-time subset by sending the mass of unlisted labels to none and supplies soft supervision to a ModernBERT-base cross-encoder student, while ground-truth labels anchor the probabilities. The components are standard; the contribution is a decision contract and a pre-registered matched-control measurement. On CLINC150 (confirmatory; BoolQ is descriptive) the student is compared with cross-entropy on ground-truth labels, label smoothing and a permuted-teacher control, and evaluated on calibration, accuracy, latency, throughput, GPU time and energy. Over three seeds, distillation lowered dynamic negative log-likelihood by 0.082, 0.092 and 0.027 nats against the three controls (99% family intervals exclude zero); temperature scaling removed about three quarters of the gap to the ground-truth controls, leaving about 0.02 nats, and accuracy was 0.85 points below label smoothing, so the pre-registered claim that the teacher adds information beyond matched controls is not met. The teacher had no accuracy headroom: the served student was more accurate (0.913 against 0.857), about 4.5 times faster per single request (median 38 against 171 ms) and used a quarter to a third of the teacher's GPU-seconds per decision, breaking even after about 46k–72k requests, an advantage that a label-only fine-tuned student shares; a constrained decoder fine-tuned on labels did not dominate it. What compiled was mainly calibrated probability structure rather than accuracy, which suggests that teacher headroom over labels bounds how much expensive generative intelligence can be converted into reusable, low-latency probabilistic decision functions. The study uses one teacher and one student backbone, and its test sets had been used in an earlier study.

Primary area:
foundation or frontier models, including LLMs
Keywords:
knowledge distillation, decision models, reasoning models, calibrated uncertainty, model cascades, System One, efficient inference, selective prediction, teacher–student learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uXDWOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The objective of this paper is to distill the decision-making capabilities of a large reasoning-capable LLM into a smaller encoder model for repeated classification tasks. The teacher model produces probability distributions over possible labels. A smaller student (ModernBERT-base) learns from these soft labels alongside ground-truth labels. The student also supports dynamic candidate sets and a "None" option. The teacher operates with reasoning disabled. The paper proves that distilling an LLM's probability distributions into a small encoder improves probability quality but not classification accuracy compared with standard supervised training.

Critical strengths
  • The paper proves that distilling an LLM's probability distributions into a small encoder improves probability quality but not classification accuracy compared with standard supervised training.
  • Ordinary classification assumes that a model always chooses from a fixed list of labels, while this paper studies a more flexible problem: the list of available choices can change at runtime.
Critical weaknesses

I think one of the main weaknesses is that in the simulations, the teacher has no meaningful performance advantage over the student. Another weakness is that the teacher's reasoning capabilities are never used since its thinking is disabled.

Questions for authors
  • What is actually being transferred if there is no teacher headroom?
  • Why is reasoning disabled if the goal is to do reasoning distillation?
  • What do you consider the main novelty of the work beyond the controlled evaluation of existing techniques?
1 comment

Thank you for the review. The general response (GR) lists the shared changes. One note on the summary: the paper reports measurements on one dataset, not proofs, and the revised text is worded to match.

W1 (no teacher advantage). Agreed. The submitted teacher was less accurate on the test set than both ground-truth students (0.857 against 0.910 and 0.924). On the tuning cohort, its net against a clean GT-CE student was -0.115. To test whether headroom matters, we added the same model with thinking enabled (T1) and a smaller teacher (T2), and we now measure test-set headroom with 95% intervals for every teacher. None had headroom: against the better base student, T1 was -4.7 points [-5.5, -3.8] and T2 -36.0 [-37.2, -34.6]. Every teacher was also below both ModernBERT-large students. Headroom did not order the conditions either: T1 had the smallest deficit but did not improve on KD(T0) (Section 6).

W2 (thinking disabled). Accepted; see Q2.

Q1 (what is transferred without headroom). Calibrated probability structure, not accuracy. The evidence comes from the thinking-off teacher on CLINC150 (Section 5.1, Section 6):

  1. Not generic smoothing. KD beats label smoothing by 0.092 nats of dynamic NLL raw and 0.025 after temperature scaling, at the 99% family level.
  2. Where the teacher puts wrong-label mass matters. KD beats the permuted teacher by 0.027 nats. The permuted teacher keeps the teacher's gold and none mass and its entropy, so we attribute the gap to the teacher's structure over non-gold labels. Descriptively, the gap survives scaling (-0.028). It lies on canonical views (0.305 against 0.366) and gold-present views (0.142 against 0.212).
  3. Most of the raw gain is a scale effect. Scaling removes about 74% and 73% of KD's raw advantage over GT-CE and GT-LS. GT-CE was overconfident (fitted temperature 2.43, against 1.02 for KD).
  4. No accuracy gain. KD is non-inferior to GT-CE (+0.55 points) but about 0.85 points less accurate than GT-LS, so the pre-registered information claim is not met.

This agrees with reports that weak teachers can help (Yuan et al. 2020). The new conditions (99% intervals) refine this reading:

  • The amount depends on the teacher. After scaling, KD(T2) was worse than KD(T0) (+0.038 nats [0.030, 0.047]) and than GT-LS.
  • Tempering did not restore the gain. KD(T1c) was worse than KD(T0) by 0.013 [0.006, 0.020]. Against its permuted control no difference was detected, so we attribute no structure to the T1 or T1c targets.
  • The backbone result is partial. On ModernBERT-large the permuted-teacher gap survives scaling (-0.011 [-0.018, -0.004]), but the scaled advantage over GT-CE was not detected (-0.005 [-0.011, 0.002]). The main finding is therefore stated for ModernBERT-base. The large KD student was also about 1.2 points more accurate than both large controls, a single-backbone observation under reused learning-rate candidates.

Q2 (why reasoning was disabled). The submission scored the teacher's direct judgment. With thinking disabled, one forward pass gives the full distribution over the 151 identifier tokens, which is the soft target KD needs. We agree this does not test reasoning, and the title claimed otherwise; it is changed (GR 4).

T1 now enables thinking: one sampled trace of at most 4,096 tokens, followed by the same constrained scoring. All 18,088 records completed and none reached the cap. The students never see the reasoning. T1 was more accurate than T0 (0.877 vs 0.857; headroom -4.7 points [-5.5, -3.8], 95%) but near one-hot (training-split mean entropy 0.004 vs 0.297 nats). KD(T1) was 0.73 points less accurate than KD(T0) [-1.27, -0.19], and no scaled NLL difference was detected. KD(T1c) was worse than KD(T0) after scaling (Section 5.2, Table 1).

Q3 (main novelty). No new loss or training method is claimed.

  • Setting. Run-time candidate sets with an explicit none option.
  • Projection. Hinton et al.'s (2015) dustbin coarsening, adapted so that none is correct whenever the gold label is not listed.
  • Measurement. A matched-control test of what LLM-teacher probabilities add beyond label smoothing, a permuted teacher and temperature scaling, across thinking off and on, two teacher sizes and two student sizes.

Section 2 discusses the closest studies, each of which covers only part of this combination. See also our reply to YGfS W1.

Reviewer_QdXGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the distillation of LLM decision making into a much smaller “system one” cross-encoder decision model that scores a runtime candidate set. A Qwen 3.8-27B model is used as a teacher by scoring the full label space and using its probability mass on omitted labels to a “none” option. The work features a ModernBert cross encoder student that is trained on the CLINIC150 and BoolQ datasets, providing an experimental protocol and empirical assessment of distilling compared to ground-truth controls.

Critical strengths
  1. The controls introduced are useful and meaningfully analysed where distillation contributes. Label smoothing and temperature-scaling baselines provide useful information compared to only cross-entropy as a baseline. For example, the observed reduction of most of the raw NLL advantage after temperature scaling is a useful finding. Additionally, KD-PERM with gold and no label options is an interesting adaptation of DKPP.

  2. The paper is honest about negative results and statistical rules. The authors acknowledge key negative results such as the accuracy deficit relative to label smoothing, failure of information criterion, prior test-set exposure and failed headline claim. Additional limitations such as the contaminated headroom reference and the limited teacher settings are stated.

Critical weaknesses
  1. The paper lacks a clear narrative structure connecting the research question, experiments and conclusions. Most of the main text is directed towards providing an inventory of controls, protocol rules, numerical results and qualifications while the findings are insufficiently prioritized. This is particularly evident in the abstract, where long, semicolon-linked sentences pack together experimental details, results and caveats making the central contribution difficult to identify. Overall, the presentation requires substantial effort from the reader to discern a relatively modest scientific contribution.

  2. The experiments performed do not resolve the central claim in the title and abstract around distilling frontier reasoning models into decision models. The paper claims frontier reasoning models but the teacher is a Qwen 3.8-27B model with its thinking disabled and one option order under a limited 3 seeds during training. The results therefore concern transferring its direct output probabilities rather than reasoning capabilities. BoolQ additionally does not provide convincing results that corroborate CLINIC15 findings because of weak and unstable controls. The teacher in this case inherently underperforms the label-trained students providing a narrower finding of distillation improving probability quality in this particular setting but leaves the broader question as to whether the approach can transfer useful capabilities from frontier reasoning models unanswered.

Questions for authors
  1. Have you tried enabling reasoning even on a small subset or different weaker/stronger teachers?

  2. Which single finding do the authors consider the main contribution and can the narrative be better structured around that narrative?

  3. The paper is dense with text and tables, could the authors add a few figures to make the findings easier to follow? For example, a schematic of the decision construct would prove helpful.

2 comments

Thank you for the detailed review. The general response (GR) lists the shared changes.

W1 (narrative structure). Accepted. The paper is now organized around one finding (see Q2).

  • Abstract. Gives the question, design, finding and boundary in about 200 words, without semicolon chains.
  • Introduction. Motivates the headroom question.
  • Section 5.1. States the main finding with Figure 2.
  • Sections 5.2-5.4. Each answers one question: thinking, teacher size, student size.
  • Appendices. Protocol rules and amendments are in Appendix H, the tuning-cohort headroom table in Appendix D, and the T0 test and difference tables in Appendix E.

W2 (title versus experiments). Accepted; the title overclaimed. With thinking disabled, the teacher's full answer distribution comes from the identifier-token probabilities of one forward pass. The submission therefore measured what the teacher's direct judgment transfers, not its reasoning. The new title drops "frontier reasoning": "What Does an LLM Teacher Transfer Without Accuracy Headroom? Distilling Probabilities into Small Candidate-Set Decision Models". A new condition, T1, adds reasoning before the same constrained scoring. T1 was more accurate than T0 (0.877 vs 0.857) but near one-hot (training-split mean entropy 0.004 vs 0.297 nats). Neither KD(T1) nor the tempered KD(T1c) improved on KD(T0) (Section 5.2). The students never see the reasoning, and we do not claim to transfer reasoning capabilities. Section 7 discloses one option order per teacher; we do not test independence of irrelevant alternatives. Every method uses three seeds. Our intervals cover test sampling; seed variation enters through the three-seed average and the requirement that all three seeds agree in sign.

W3 (BoolQ). Agreed. GT-CE is below the majority-class rate (0.595 against 0.622). Against GT-LS, KD's NLL advantage is unresolved and disappears after temperature scaling. BoolQ is now descriptive only and supports no claim. Its per-seed results are in Appendix E, and it received no new runs.

W4 (teacher underperforms the students). Agreed; this was the central limitation. The submitted teacher's test accuracy was 0.857, against 0.910 (GT-CE) and 0.924 (GT-LS). We now measure test-set headroom with 95% intervals for every teacher, and none had headroom. Against the better base student, T1 was -4.7 points [-5.5, -3.8] and T2 -36.0 [-37.2, -34.6], and every teacher was below both ModernBERT-large students. The paper therefore says only that headroom was negative for every tested teacher. It does not claim that headroom bounds the gain. Whether a teacher with real headroom transfers accuracy remains untested (Section 6).

Q1 (enabling reasoning; weaker or stronger teachers). Yes, on all CLINC150 rows rather than a subset.

  • T1 (thinking on). All 18,088 rows (training split, tuning cohort, test) completed, and none reached the 4,096-token cap (median reasoning length on test: 150 tokens). One sampled trace per input is used; Section 7 lists this as a limitation.
  • T1c. Adds one fitted teacher temperature (tau = 3.18), with its own permuted control.
  • T2. The weaker Qwen3-4B. After scaling, its student was worse than KD(T0) by 0.038 nats [0.030, 0.047].
  • Stronger teachers. The larger open model of the same generation is released only in FP8 and did not fit in memory on two GPUs (Appendix B). Closed models and teachers above 27B were not tested.

Results: Sections 5.2-5.3, Table 1, Appendix I.

Q2 (single main finding). Without accuracy headroom, distilling an LLM teacher's probabilities transferred calibrated probability structure, not accuracy, to a ModernBERT-base candidate-set student, and how much transferred depended on the teacher. The title, abstract, introduction, Figure 1 and Figure 2 are all built around this finding.

Q3 (figures). We added two figures.

  • Figure 1. A schematic of the decision setup. The teacher scores the full label set plus none once per training input. The projection keeps the listed labels' probabilities and moves the rest to none. The cross-encoder student scores each candidate and none.
  • Figure 2. Three panels for each teacher and backbone: (a) teacher headroom, (b) KD minus the better ground-truth student in accuracy, (c) the same difference in temperature-scaled dynamic NLL.

Thanks for the response and additional experiments. I appreciate that you have resolved several of my concerns specifically the title to evidence mismatch and provided results with thinking. The presentation is clearer and additional figures are added but the text is still difficult to read with lots of hedging and procedure details throughout the text. Additionally, while the revisions are appreciated they do limit the scope of what I consider an already modest contribution. The best result is specific to a single teacher/student/dataset configuration and is not consistent with other teacher/student setups when compared to temperature baselines. I therefore leave my review of this paper unchanged.

Reviewer_YGfSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether one can distill large reasoning models into smaller classification models for quick decision making. They find that distillation does not help accuracy compared to supervised learning of a classification head, but that it does help calibration. However, they find that through methods like temperature scaling, these classification methods trained from scratch can approach the distilled calibration performance. They evaluate other qualities like latency, but note that this is not specific to their distilled method.

Critical strengths
  1. The authors are honest about their results. They provide transparent evaluation, have many experiments, and are honest about the advantages and limitations of their method.
Critical weaknesses
  1. Limited novelty: the paper uses distillation of a larger reasoning model into a classification head which offers limited novelty. They do not have strong empirical results to back up this limited novelty.

  2. Limited experiments: the authors base these conclusions off a limited selection of models, student backbones, and datasets. For this to be a scientific investigation, I would expect a much more comprehensive suite of models, experiments, to provide insights into the method.

Questions for authors

How sensitive are the results to the choice of teacher models and student backbones?

1 comment

Thank you for the review. The general response (GR) lists the shared changes; this reply answers your points.

Note on the summary. Temperature scaling narrows the gap but does not close it. It removes about 74% (GT-CE) and 73% (GT-LS) of KD's raw dynamic-NLL advantage. KD remains better after scaling, by 0.021 and 0.025 nats, at the 99% family level with all seeds agreeing in sign (Section 5.1; tables in Appendix E). The submitted teacher was scored with thinking disabled, so the submission measured distillation of an LLM's direct answer distribution, not of reasoning. The revised title no longer claims reasoning distillation (GR 4).

W1 (limited novelty). We agree that the training method is not new, and the revision no longer presents it as new. KD is standard distillation with no new loss. Our projection is the dustbin coarsening of Hinton et al. (2015), adapted so that none is correct whenever the gold label is not listed. The contribution we claim is a measurement. With run-time candidate sets and a none option, it asks what an LLM teacher's probabilities add beyond controls that each remove one source of benefit:

  • label smoothing: softening without a teacher;
  • a permuted teacher: same gold and none mass and entropy, no structure over wrong labels;
  • temperature scaling: a pure scale effect.

The revision runs this with thinking off and on, two teacher sizes and two student sizes. Section 2 discusses the closest studies, each of which covers only part of this. The evidence supports one finding: without accuracy headroom, the teacher's probabilities transferred calibrated probability structure, not accuracy, to ModernBERT-base, and how much transferred depended on the teacher (Section 1; Section 5.1).

W2 (limited experiments). We accept this for teachers and backbones. We added T1 (thinking on), T1c (T1 with a fitted teacher temperature and its own permuted control), T2 (Qwen3-4B) and a ModernBERT-large student with all controls. Each uses the submitted recipe and three seeds (GR 1). We did not add datasets. The question needs candidate sets where the correct answer can be a listed label, an unlisted known label or an out-of-scope input. In our setup only CLINC150 yields both known-absent and out-of-scope-absent views. BoolQ subset views hold one option, so BoolQ stays descriptive. The abstract and Section 7 state the single confirmatory dataset as the main boundary.

Q1 (sensitivity to teacher and backbone). Headroom intervals are 95%; comparisons are 99% grouped intervals with the seed-sign rule.

  • T1 (thinking on). Headroom was -4.7 points [-5.5, -3.8]. KD(T1) had lower dynamic NLL than GT-LS, both raw (-0.080 [-0.090, -0.069]) and scaled (-0.020 [-0.027, -0.012]), but was less accurate (-1.58 points [-2.18, -0.97]). Against GT-CE its NLL intervals were below zero but not seed-consistent. Compared with KD(T0), it was 0.73 points less accurate [-1.27, -0.19], and no scaled NLL difference was detected.
  • T1c. After scaling, KD(T1c) was lower than GT-LS (-0.012 [-0.019, -0.005]) but not seed-consistently lower than GT-CE. It was worse than KD(T0) by 0.013 nats [0.006, 0.020], and no difference was detected against its permuted control. Tempering did not restore the T0 gain.
  • T2. Headroom was -36.0 points [-37.2, -34.6]. After scaling, KD(T2) was worse than KD(T0) (+0.038 [0.030, 0.047]) and than GT-LS (+0.014 [0.004, 0.023]).
  • ModernBERT-large. KD had lower raw dynamic NLL than GT-CE (-0.035 [-0.050, -0.022]), GT-LS (-0.094 [-0.101, -0.087]) and KD-Perm (-0.013 [-0.019, -0.007]). After scaling it stayed lower than GT-LS and KD-Perm, but no difference from GT-CE was detected (-0.005 [-0.011, 0.002]), so the main finding is restricted to ModernBERT-base. Large KD was about 1.2 points more accurate than both controls (+1.18 and +1.21). This is a single-backbone observation under reused base learning-rate candidates.

The results are therefore sensitive to the teacher: the smaller teacher transferred less, and thinking did not help. They are partly sensitive to the backbone.

Locations: Sections 5.2-5.4, Figure 2, Table 1, Appendix I.

Comments on the paper

1 comment

Thank you for the three reviews. They agree on four points, and we accept all four: (a) the title promised "frontier reasoning", but the teacher was scored with thinking disabled; (b) the submitted teacher had no accuracy headroom over label-trained students; (c) one teacher and one student backbone limit generality; (d) the paper lacked a single main finding and was hard to read. The individual replies refer to this response as "GR".

1. New experiments. All use CLINC150, the submitted recipe, the frozen splits and views, and seeds 17, 29 and 43.

  • T1, thinking on. Qwen3.8-27B samples one reasoning trace (at most 4,096 tokens); the same constrained scoring then gives an answer distribution conditional on it. All 18,088 records completed; none reached the cap.
  • T1c. T1 with one teacher temperature (tau = 3.18), fitted on the teacher's own tuning-cohort NLL, with its own permuted-teacher control.
  • T2. Qwen3-4B with thinking off, the second pre-specified teacher candidate.
  • ModernBERT-large students (GT-CE, GT-LS, KD, KD-Perm), reusing the base model's learning-rate candidates, so cross-backbone differences are descriptive.

Headroom is now measured on the test set against both ground-truth students of each backbone (95% grouped bootstrap intervals). The new comparisons were designed after the reviews, with outcome rules fixed before any production run (Appendix I). They lie outside the pre-registered family and are reported with 99% grouped intervals and the three-seed sign rule, without multiplicity adjustment.

2. Main finding. Without accuracy headroom, the thinking-off teacher's probabilities transferred calibrated probability structure, not accuracy, to the ModernBERT-base student, and how much transferred depended on the teacher (Sections 5.1-5.4, Figure 2).

  • No headroom. Against the better base student (GT-LS), T0 was -6.7 points [-7.6, -5.8], T1 -4.7 [-5.5, -3.8] and T2 -36.0 [-37.2, -34.6]. Every teacher was also below both ModernBERT-large students.
  • Teacher size. After temperature scaling, KD(T2) was worse than KD(T0) by 0.038 nats [0.030, 0.047].
  • Thinking. T1 was more accurate than T0 (0.877 vs 0.857) but near one-hot (mean entropy 0.004 vs 0.297 nats); neither KD(T1) nor KD(T1c) improved on KD(T0).
  • ModernBERT-large. The raw ordering reproduced, but the scaled advantage over GT-CE was not detected (-0.005 [-0.011, 0.002]), so the main finding is stated for ModernBERT-base. Its KD student was about 1.2 points more accurate than both large controls: a secondary, single-backbone observation, not a claim of accuracy transfer.
  • Scope. CLINC150 is the only confirmatory dataset, and its test set was used in an earlier study. Closed models and teachers larger than 27B were not tested (Section 7).

3. Submitted results (unchanged). With T0, KD had lower dynamic NLL than GT-CE (-0.082 [-0.102, -0.064]), GT-LS (-0.092 [-0.099, -0.084]) and KD-Perm (-0.027 [-0.035, -0.020]) at the 99% family level, all three seeds agreeing in sign. Temperature scaling removed about 74% and 73% of the gaps to GT-CE and GT-LS (residuals 0.021 and 0.025 nats). The pre-registered information claim was not met: KD was non-inferior to GT-CE in accuracy (+0.55 points), but not shown non-inferior to GT-LS (-0.85 points [-1.44, -0.26]).

4. Presentation and summary of changes (revision round 1)

  • Title: "frontier reasoning" removed; now "What Does an LLM Teacher Transfer Without Accuracy Headroom? Distilling Probabilities into Small Candidate-Set Decision Models".
  • Abstract: rewritten around the main finding and its boundary.
  • Section 1: defines teacher headroom and calibrated probability structure (a scale effect plus a scaled advantage that survives temperature scaling); states the main finding; no new loss is claimed.
  • Figures (new): Figure 1 (setup); Figure 2 (headroom, accuracy and scaled NLL per teacher and backbone).
  • Sections 2-4: students never see the teacher's reasoning, and teacher-calibration work is cited (2); plain-language setup (3); plain role names, mapped in Appendix A (4.1); new conditions and test-set headroom (4.3, new); pre-registration rules moved to Appendix H (4.6).
  • Section 5: main finding (5.1); thinking teacher, teacher size and larger student, with Table 1 (5.2-5.4, new); cost unchanged, duplicate table removed (5.5).
  • Sections 6-8: discussion extended with the new readings; three limitations added (one T1 trace per input, ModernBERT-large learning-rate reuse and micro-batch size, no multiplicity adjustment).
  • Appendices: tuning-cohort headroom (D) and T0 tables with descriptive BoolQ (E), moved from the main text; future work reduced to several reasoning traces per input (G); post-submission design, costs, outcome rules and results (I, new).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.