acceptodds
Under review as a conference paper at ICLR 2027

BWB-KD: Domain Knowledge Distillation via Knowledge Injection, Reasoning Organization, and Activation

Abstract

Transferring domain expertise from black-box teachers while preserving general capabilities remains a challenge in domain adaptation. Direct supervised adaptation can improve domain performance at the cost of broader capabilities, while subsequent reinforcement learning does not reliably resolve this trade-off. We propose BWB-KD, a stage-coordinated distillation framework that connects knowledge injection, reasoning organization, and outcome activation through complementary reasoning modes. Non-thinking supervised fine-tuning first builds a domain expert from black-box demonstrations. Thinking-mode on-policy distillation then uses the frozen expert to guide a base-initialized student along student-generated reasoning trajectories. Finally, non-thinking reinforcement learning optimizes the transferred student using outcome feedback. This off–on–off design places expert-guided reasoning organization between knowledge acquisition and task optimization. Experiments on Qwen3.6-27B across three biomedical tasks show improved domain performance over supervised fine-tuning followed by reinforcement learning, with larger gains in non-thinking inference. General-capability evaluations further show better non-thinking retention. Comparisons with offline distillation, alternative reinforcement learning methods, and different stage orders and reasoning modes support the value of coordinating how domain knowledge is acquired, transferred, and reinforced.

Primary area:
foundation or frontier models, including LLMs
Keywords:
black-box knowledge distillation, domain adaptation, on-policy distillation, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_33coOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BWB-KD trains a non-thinking SFT expert, distills it into a fresh base-initialized student using thinking-mode on-policy trajectories, then applies non-thinking GRPO. Three biomedical adaptations show improved domain performance, while the MedXpert branch retains substantially more non-thinking general capability than SFT-based training paths.

Critical strengths

The MedXpert OFF gain over SFT+GRPO is substantial: +5.37 points, with a question-paired 95% CI of [2.82, 8.02] (Table 6). General capability also improves markedly over that path (Table 2). Offline distillation, stage-order, mode, and alternative-RL comparisons probe the design beyond a simple SFT+RL baseline (Tables 3–5). They support separating expert acquisition from final student training.

Critical weaknesses

The advantage over stronger alternatives is less decisive. OFF MedXpert gains are only 1.04 points over FKL+GRPO and 0.91 over Base+DAPO, with lower general means (Tables 3, 5). Comparable uncertainty and end-to-end training-cost comparisons are missing. Evidence is concentrated in one backbone, with general-retention tests only on the MedXpert branch. Reported bootstrap intervals measure question sampling, not training variance (Appendices D–E); the wider robustness of the schedule remains uncertain.

Questions for authors

Do gains over FKL+GRPO and Base+DAPO persist across training seeds and comparable total budgets, including expert acquisition and OPD?

1 comment
AuthorsOpenReview ↗

Performance Against Stronger Baselines, Training Cost, and Cross-Task Retention

Thank you for highlighting offline distillation and direct DAPO as strong alternatives. We added three-seed comparisons and extended RL budgets.

Given strong offline FKL and Base+DAPO baselines, is BWB-KD worthwhile?

The main experiments use 100 RL updates. We extended FKL+GRPO to 200 and SFT+GRPO to 400, while keeping BWB-KD and the other methods at their original budgets.

The following results are from Table 14(a). Accuracy is mean ± sample standard deviation over three training seeds (%). Each run selects its checkpoint by OFF validation accuracy and evaluates that model in both modes.

MethodRL updatesEstimated total GPU-hoursMedXpert OFFMedXpert ON
SFT expert—2.6940.16 ± 0.1947.53 ± 0.47
OPD student—75.3344.09 ± 0.3847.01 ± 0.89
Base+GRPO10076.3041.64 ± 0.2743.23 ± 0.47
SFT+GRPO400161.1241.74 ± 0.5347.94 ± 0.82
FKL+GRPO200163.2646.21 ± 0.2347.58 ± 0.30
Base+DAPO100148.3444.88 ± 0.7144.83 ± 1.06
BWB-KD100150.8146.79 ± 0.3048.81 ± 0.42

Costs include expert acquisition, distillation, teacher services, and RL, excluding loading, saving, and independent evaluation. Offline FKL+GRPO uses a different GPU configuration but has similar output lengths, so its RL cost uses BWB-KD's reference step time, scaled to 200 updates. DAPO uses timing measurements from the 32-GPU configuration.

Extended FKL+GRPO costs more than BWB-KD, yet BWB-KD remains ahead by 0.58 points in OFF and 1.23 in ON. Extending SFT+GRPO from 100 to 400 updates improves OFF accuracy by only 0.52 points and ON by 0.29, leaving gaps of 5.05 and 0.87 points. Giving the additional computation to RL does not replace expert-guided transfer on student-generated trajectories.

Both routes retain OFF general ability well. Their difference in ON reasoning appears before RL: offline FKL scores 70.50 on BBEH, below Base's 83.25, while the OPD student reaches 83.50. After RL, BWB-KD exceeds the original 100-update FKL+GRPO baseline by 2.88 points in ON general mean, including 14.00 points on BBEH.

Offline FKL fits fixed teacher trajectories. OPD instead provides expert guidance along the student's current reasoning trajectories. The results support this form of transfer as a way to integrate domain knowledge while preserving the student's general reasoning ability.

At similar total cost, BWB-KD exceeds Base+DAPO by 1.91 points in OFF and 3.98 in ON across three seeds. In the main experiment, it also generates 43.0% fewer tokens in OFF and 28.2% fewer in ON.

Thus, using part of the budget for expert-guided transfer before RL gives stronger domain performance and shorter outputs than direct RL. Appendix G, Tables 13–14, reports stage-wise costs and the repeated-run results.

General capabilities after adaptation to other tasks

We evaluated all five adapted training paths from each task, together with Base, in both inference modes. To reduce API costs, ArenaHard uses DeepSeek V4.1 Flash instead of GPT-5.4, and Base is reevaluated as well. All methods share the judge and scoring procedure.

The columns group models by adaptation task and inference mode. Each entry is the general-benchmark mean over AIME25, ArenaHard, BBEH, and IFBench. Appendix J, Tables 18–19, gives individual scores. The judge budget is 32K and the ON BBEH context is 128K.

MethodBioASQ OFFBioASQ ONMedCaseReasoning OFFMedCaseReasoning ON
Base64.0875.5164.0875.51
SFT57.6675.2956.3376.84
OPD student63.7275.2363.8974.89
Base+GRPO64.9675.9565.7875.36
SFT+GRPO57.7476.0055.8575.79
BWB-KD62.9575.4766.3875.28

The SFT paths lose OFF general performance on both tasks, and subsequent GRPO does not recover it. BWB-KD improves over SFT+GRPO by 5.21 points after BioASQ adaptation and 10.53 after MedCaseReasoning adaptation. Its mean across the two branches is 64.66 in OFF and 75.38 in ON, close to Base's 64.08 and 75.51. ON performance is broadly similar across methods; the main difference is in OFF.

Retention is already present before RL. The OPD students score 63.72 and 63.89 in OFF, both close to Base. Combined with the domain gains, this shows that expert-guided transfer followed by RL preserves general ability while improving domain performance. Base+GRPO also retains general ability, but its domain results are below BWB-KD. The retention finding therefore extends beyond the MedXpert branch.

Reviewer_MfawOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BWB-KD, a three-stage pipeline for distilling domain expertise from a black-box teacher (GPT-5.4) into a hybrid-reasoning open model (Qwen3.6-27B) while keeping its general capabilities. Stage 1 runs thinking-OFF SFT on filtered teacher demonstrations to build a frozen domain expert. Stage 2 initializes a fresh student from the base model and runs thinking-ON on-policy distillation (top-32 reverse KL) against that expert. Stage 3 applies thinking-OFF GRPO with binary outcome rewards.

Critical strengths

The paper proposes a simple, useful idea. Turning black-box demonstrations into a white-box expert, then distilling on-policy into a fresh student, gives dense token-level supervision without teacher logits. It also avoids SFT's damage to the deployed model. This is easy to adopt.

Critical weaknesses
  1. The headline claims ("particularly clear gains in non-thinking inference", "better non-thinking retention") are all made in OFF mode. The paper never says why OFF is the target, such as serving cost or latency, or how much a hybrid model is actually used without thinking. With thinking ON, all six training paths score 71.12–73.01 on the general suite (Table 2). The SFT expert (72.81) even beats Base (71.93).

  2. The baseline's OFF collapse looks partly self-inflicted. OFF SFT on short teacher explanations cuts OFF output length from 1,614 to 285 tokens and AIME25 (OFF) from 79.31 to 51.72. The ON-SFT ablation shows the same damage in the ON mode (ON general 72.81 → 61.09). So the observation is that SFT damages whichever mode it trains. The SFT+RL baseline was set up to train exactly the mode being evaluated.

  3. Weak generalizability. All results use Qwen3.6-27B, GPT-5.4 and biomedical QA.

Questions for authors
  1. How do the main tables change if each method's checkpoint is picked by validation accuracy in the mode being reported, rather than always OFF?

  2. How exactly are demonstrations formatted for ON SFT? Is the teacher's <reasoning> placed inside <think>, left as visible output, or something else?

  3. How does BWB-KD compare with simpler retention methods: lower-learning-rate SFT, SFT with general-data replay, LoRA SFT, or SFT followed by OPD toward the base model? Please also report total compute per method.

2 comments
AuthorsOpenReview ↗

The Value of OFF Inference, Broader Applicability, and ON-Based Model Selection (Part 1/2)

Thank you for the detailed assessment. We address your questions below.

Why focus on OFF rather than simply use ON?

The methods have similar ON general scores. BWB-KD improves domain performance at lower generation cost and recovers the OFF general ability lost by SFT-based training. This matters for services constrained by response time and throughput.

BWB-KD OFF scores 46.45% on MedXpert, close to SFT+GRPO ON at 47.44%, with 45.3% fewer tokens. Comparing both in OFF, the paired domain gain is 5.37 points and the general mean rises from 49.20 to 61.54.

Serving measurements use the same 100 questions, one response each, one H200 per service, TP=1, and identical sampling. Loading and warm-up are excluded. RT is mean response time; throughput is in requests/s.

ConcurrencySFT+GRPO ON RT (s)BWB OFF RT (s)Throughput: SFT / BWB
145.5924.450.0219 / 0.0409
547.4727.030.1024 / 0.1619
1051.3025.730.1750 / 0.3041

BWB-KD reduces response time by 43.1%–49.8% and delivers 1.58–1.86 times the throughput. Time per token is similar; shorter outputs drive the improvement.

Changing the final RL stage to ON increases OFF length from 1,541.8 to 2,377.6 tokens but lowers accuracy from 46.45 to 45.58. ON distillation followed by OFF RL thus uses the transferred knowledge more effectively. See Section 5.1 and Appendix H.

Does the method apply to other models, teachers, and tasks?

We added Qwen3.5-4B experiments using the same MedXpert data and training configuration.

MethodMedXpert OFFMedXpert ON
Base21.5825.35
SFT expert22.7028.34
OPD student23.7428.29
SFT+GRPO23.4428.69
BWB-KD25.0830.16

BWB-KD again achieves the highest accuracy in both modes. It exceeds SFT+GRPO by 1.64 points in OFF and 1.47 in ON, and Base by 3.50 and 4.81 points. The OPD student's ON score is close to the SFT expert's before RL, but BWB-KD reaches a higher score after RL. As in the 27B results, the middle stage improves the benefit obtained from subsequent RL.

Our teacher choice reflected performance and cost. We compared Gemini 3.6 Flash, GPT-5.2, GPT-5.4, and Claude Opus 5 in preliminary evaluations and selected GPT-5.4 for the full experiments, not because other teachers were unusable. BWB-KD acquires demonstrations without relying on GPT-5.4-specific interfaces or internal information. The resulting open-weight expert supplies token distributions for OPD.

The three tasks also impose different requirements: MedXpert uses multiple-choice questions, BioASQ covers several types of biomedical knowledge QA, and MedCaseReasoning asks for diagnoses from clinical cases. Each task is adapted independently, and BWB-KD improves domain performance across all three. In future versions, we will consider benchmarks outside medicine to examine additional domains and task types.

Do the conclusions change when checkpoints are selected by ON validation accuracy?

We retrained five MedXpert paths, selecting the best ON-validation checkpoint at every stage. The training modes remain unchanged.

This applies to the whole pipeline. The ON-selected SFT expert supplies supervision for a fresh Base-initialized OPD student, and BWB-KD's final RL starts from the ON-selected OPD checkpoint. Each final model is tested in both modes.

MethodMedXpert OFFMedXpert ON
SFT expert40.0747.94
OPD student43.0948.29
Base+GRPO41.1243.46
SFT+GRPO40.7749.15
BWB-KD45.4350.07

BWB-KD remains the best in both modes, exceeding SFT+GRPO by 4.66 points in OFF and 0.92 in ON. ON scores generally increase and OFF scores decline slightly relative to the original runs, but the main comparison is unchanged. The domain advantage therefore does not depend on always selecting checkpoints by OFF validation performance.

AuthorsOpenReview ↗

ON-SFT Formatting and Comparisons with Simpler Retention Methods (Part 2/2)

How are ON-SFT demonstrations placed in the template?

ON SFT uses the same teacher explanations and answers as OFF SFT. The explanation is placed inside the thinking channel rather than left in the visible response.

We extract the text inside <reasoning>...</reasoning> into the assistant's reasoning_content field. The content field holds the answer. With enable_thinking=true, the chat template renders:

<think>
{original teacher reasoning text}
</think>

<answer>
opt_X
</answer>

The original reasoning tags are removed; the explanation, question, and options are unchanged. The user instruction requests internal thinking followed by only the final option inside <answer> tags.

The reasoning text, closing </think> marker, and answer receive the SFT loss. User tokens and the assistant generation prefix, including the opening <think>, are masked. This compares learning the same explanation as visible text with learning it in the thinking channel. Appendix F.4 gives the exact instruction, field mapping, and loss mask; Section 4.4 points to it.

Does SFT degrade because of its configuration, and can simpler retention methods replace BWB-KD?

We added lower-learning-rate SFT, LoRA, general-data replay, and SFT followed by OPD toward Base. SFT variants use the same medical demonstrations and OFF prompts. Replay uses 50% general examples. New models are selected by OFF validation accuracy and tested in both modes.

To reduce API costs, ArenaHard uses DeepSeek V4.1 Flash, including reevaluation of reference models. All methods share the judge and scoring procedure; general scores therefore differ from the original main table.

MethodMedXpert OFFMedXpert ONOFF tokensOFF generalGPU-hours
Base39.8243.241,613.664.080.00
SFT40.2947.92285.149.752.69
SFT, LR 1e-636.8346.36174.459.632.13
SFT, LR 2e-735.4042.63369.464.452.12
LoRA, r16, LR 2e-537.1845.19285.462.882.30
LoRA, r64, LR 2e-641.0742.591,564.263.832.27
Tulu SFT replay40.0848.57339.027.217.44
Nemotron SFT replay38.0447.88390.360.1938.51
Tulu mixed OPD replay39.7746.62283.747.7788.65
Nemotron mixed OPD replay39.8648.01285.449.4287.93
SFT → Base OPD40.7348.44291.456.0260.15
OPD student43.6746.101,408.064.8975.33
BWB-KD46.4548.351,541.862.12150.81

Scores are percentages; general is the four-benchmark mean. Original domain scores and lengths follow the paper. Estimated GPU-hours include expert training and teacher services, excluding independent evaluation and demonstration API charges. Nemotron uses math/STEM data.

Lower-learning-rate SFT and LoRA do improve retention. Rank-64 LoRA retains a general mean of 63.83 and generates 1,564 OFF tokens, close to BWB-KD's 1,542. Yet its domain scores are 41.07 in OFF and 42.59 in ON, compared with BWB-KD's 46.45 and 48.35. Restoring output length and general ability does not by itself produce the same domain gains.

Replay also improves some results. Nemotron SFT reaches a 60.19 general mean but only 38.04 OFF domain accuracy. Some alternatives match BWB-KD in ON, but lag in OFF domain accuracy and general retention.

BWB-KD uses the same SFT expert as the baseline, but transfers its guidance to a separate Base-initialized student.

Mixed OPD replay continues the SFT model, with expert guidance on medical queries and Base guidance on general queries. SFT → Base OPD instead uses only general queries and Base as teacher. It raises the general mean to 56.02, with OFF domain accuracy of 40.73.

BWB-KD's OPD student reaches 43.67 OFF domain accuracy and a 64.89 general mean before RL. RL then raises domain accuracy to 46.45.

Low-learning-rate SFT and LoRA are cheaper retention options. BWB-KD invests more compute to obtain stronger domain gains while retaining general ability close to Base. Its value is therefore not merely reducing forgetting. Appendix I gives the full configurations, dual-mode results, and costs.

Reviewer_3JjHOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The goal is to adapt a model to a domain using a proprietary teacher without losing its general ability, in a model that has separate thinking and non-thinking modes. The recipe is three stages with thinking off, on, then off. SFT on filtered GPT-5.4 demonstrations produces a domain expert, which is then frozen. A separate student, starting from the same base weights, is trained to match that expert's top-32 token distributions along its own thinking-mode rollouts. GRPO with binary outcome rewards finishes, again with thinking off. Everything runs on Qwen3.6-27B over three biomedical tasks, with four general benchmarks for retention. Against SFT followed by RL, the domain mean improves by about 2.5 points without thinking and 0.6 with it; the larger effect is on general ability, where plain SFT costs more than fourteen points of non-thinking mean and this pipeline keeps most of it. Comparisons include offline distillation, two other RL choices, and the other orderings of the three stages.

Critical strengths
  1. The retention result is large and does not run through an LLM judge: non-thinking SFT costs 14.4 points of general mean, and this pipeline keeps almost all of it.

  2. The mode grid in Table 4 lets the schedule claim be checked directly, token counts included, rather than taken on trust.

Critical weaknesses
  1. One model occupies three roles: GPT-5.4 writes the demonstrations, supplies the RL reward for BioASQ and MedCaseReasoning, and judges those same two tasks at test time, with no human check or second judge. Those are exactly the thin margins (50.91 against 50.67; 51.03 against 49.25). The rule-scored task, MedXpert, gives a credible 5.38 points without thinking but only 0.91 with it, on an interval that includes zero.

  2. The extra stage brings extra compute and nothing separates the schedule from the budget: only the query pool is shared, no arm gives the baseline the additional updates, and the number of retained demonstrations is never reported. Single runs throughout, while 0.14 to 1.0 point differences are discussed as findings. Reasoning organization, the middle stage's whole purpose, is never measured.

Questions for authors
  1. How many demonstrations survived filtering per task? I could not find the number anywhere.
  2. What are the token and rollout costs of BWB-KD against SFT then RL, and does the baseline close the 2.46-point gap if given the same budget?
  3. With a judge other than GPT-5.4, or human adjudication on a sample, do BioASQ 50.91 vs 50.67 and MedCaseReasoning 51.03 vs 49.25 hold?
  4. Over three seeds, what is the spread of MedXpert accuracy and of the non-thinking general mean?
  5. What does the frozen non-thinking expert's distribution look like inside the student's thinking spans, and does an expert trained with thinking enabled change Table 3?
2 comments
AuthorsOpenReview ↗

Demonstration Counts, Training Budgets, and Alternative-Judge Results (Part 1/2)

Thank you for the feedback. We address your questions below.

How many demonstrations remain after filtering?

The Train and Validation columns in Table 7 give the final counts after demonstration filtering and splitting. The corresponding candidate pools are larger:

TaskCandidate questionsSFT demonstrationsValidation
MedXpert2,0171,19763
BioASQ5,3891,812200
MedCaseReasoning13,0927,671500

The test sets are held out for final evaluation and are not used for filtering or training. We have clarified these counts in Table 7 and the demonstration-construction description in Appendix B.

Do the gains come from the method or the additional compute?

To test whether more RL can close the gap, we extended SFT+GRPO from 100 to 400 updates and offline FKL+GRPO from 100 to 200. BWB-KD and the other methods retain their original budgets.

The table below combines the supplementary results in Table 14. Accuracy is mean ± sample standard deviation over three training seeds (%). Each run selects its checkpoint by OFF validation accuracy, then tests that checkpoint in both modes. General mean averages the four general benchmarks.

MethodRL updatesGPU-hoursMedXpert OFFMedXpert ONOFF general mean
Base————64.48 ± 1.40
SFT expert—2.6940.16 ± 0.1947.53 ± 0.4750.50 ± 1.07
OPD student—75.3344.09 ± 0.3847.01 ± 0.8965.17 ± 0.25
Base+GRPO10076.3041.64 ± 0.2743.23 ± 0.4764.94 ± 0.71
SFT+GRPO400161.1241.74 ± 0.5347.94 ± 0.8251.24 ± 0.55
FKL+GRPO200163.2646.21 ± 0.2347.58 ± 0.3065.10 ± 0.60
Base+DAPO100148.3444.88 ± 0.7144.83 ± 1.0667.09 ± 1.10
BWB-KD100150.8146.79 ± 0.3048.81 ± 0.4264.11 ± 1.77

GPU-hours estimate the cost per run, including expert training, distillation, teacher services, and RL. Loading, saving, and independent evaluation are excluded. Offline FKL+GRPO uses the manuscript's reference step-time estimate; DAPO uses timing measurements from the 32-GPU configuration.

SFT+GRPO's OFF accuracy rises from 41.22 at 100 updates to 41.74 at 400; ON accuracy rises from 47.65 to 47.94. Even with four times as many RL updates and a higher total cost, it remains 5.05 points behind BWB-KD in OFF and 0.87 in ON. The additional RL does not recover BWB-KD's domain gains.

The stronger alternatives lead to the same conclusion. Extended FKL+GRPO costs more than BWB-KD but remains 0.58 points behind in OFF and 1.23 in ON. At a similar cost, Base+DAPO trails by 1.91 and 3.98 points. These comparisons show the benefit of transferring expert guidance on student-generated trajectories before RL, rather than allocating the additional computation entirely to RL.

For rollout costs, Table 8 specifies 32 responses per OPD update and 256 per GRPO update. The cost-comparison budgets therefore allocate 6,400 responses to 200 OPD updates and 25,600 to 100 GRPO updates. BWB-KD uses a combined budget of 32,000 responses, compared with 102,400 for 400-step SFT+GRPO. The per-response token limits are 1,500 for OPD and 8,192 for MedXpert GRPO. Stage-wise costs are in Appendix G, Tables 13–14.

Do the domain gains hold with another judge?

We generated new OFF and ON responses from all six methods on BioASQ and MedCaseReasoning, then scored every method with DeepSeek V4.1 Flash.

MethodBioASQ OFFBioASQ ONMedCaseReasoning OFFMedCaseReasoning ON
Base46.8447.1139.4641.58
SFT expert51.7551.0639.2742.16
OPD student51.6051.7240.6741.91
Base+GRPO49.6149.2641.4742.06
SFT+GRPO52.3452.9141.1742.69
BWB-KD53.0153.3242.0543.71

BWB-KD achieves the highest accuracy on both tasks in both modes. Compared with SFT+GRPO, the gains on BioASQ are 0.67 points in OFF and 0.41 in ON; those on MedCaseReasoning are 0.88 and 1.02 points. The domain improvement therefore persists with another judge. Appendix L reports the complete results, and Section 4.2 summarizes the finding.

AuthorsOpenReview ↗

Three-Seed Stability and Expert Guidance During Thinking (Part 2/2)

Are the results stable across three seeds?

Table 14 reports three-seed MedXpert accuracy and OFF general performance, summarized in Part 1. Each run selects its checkpoint by OFF validation accuracy and evaluates the same checkpoint in both modes.

The repeated runs retain the stage-wise improvement: OFF accuracy increases from 40.16 for the SFT expert to 44.09 for the OPD student and 46.79 after RL. BWB-KD's OFF standard deviation is 0.30, versus 0.53 for extended SFT+GRPO. Its 5.05-point advantage is therefore not confined to the originally reported run.

General retention also persists. BWB-KD reaches 64.11 ± 1.77, close to Base's 64.48 ± 1.40, whereas SFT and SFT+GRPO score 50.50 ± 1.07 and 51.24 ± 0.55. To reduce API costs, ArenaHard uses DeepSeek V4.1 Flash for every method in this evaluation. Base's variability comes from three evaluations without retraining.

Together, the repeated runs reproduce stronger domain performance and general retention. The ON domain advantage is smaller, consistent with the larger benefit in OFF. Table 14(a) reports domain results and costs; Table 14(b) gives the individual general scores. Section 4.2 summarizes these findings.

What demonstrates "reasoning organization," and can an OFF expert guide an ON student?

1. What guidance does the OFF expert provide inside thinking spans?

“OFF expert” refers to the expert's SFT training mode. Its dedicated thinking channel is disabled during SFT, but the training targets still contain teacher explanations and answers. During OPD, the frozen expert reads the student's actual ON prefix and predicts the next token. Teacher and student distributions are compared on the same context.

We sampled two ON trajectories for each of the 577 MedXpert test questions and examined the expert at identical prefix positions. On Base-generated thinking prefixes, its top-32 candidates contain 99.92% of its probability mass. The actual next token is among these candidates at 99.99% of sampled positions. Its mean probability of the thinking-end marker is 1.06 × 10^-6. Thus, OFF training does not make the expert consistently ask the student to stop thinking. Table 20 includes the corresponding Base and ON-expert distributions.

2. How does reasoning organization develop during training?

We recorded the first 200 OPD updates and rescored a fixed set of thinking prefixes at five checkpoints. Keeping the prefixes fixed lets us examine changes in the student's predictions without changing the sampled content.

OPD updateStudent–expert KL ↓Top-32 candidate overlap ↑
00.111280.28%
500.051684.56%
1000.036886.66%
1500.031187.41%
2000.029187.70%

This table uses fixed Base prefixes. KL is computed with the teacher's top-32 support and the training temperature. Candidate overlap rises by 7.42 points, with a question-paired bootstrap 95% interval of [7.29, 7.54]. The same pattern holds on fixed OPD prefixes: KL falls from 0.1104 to 0.0312, while overlap rises from 80.49% to 87.46% (Figure 5).

Each checkpoint's newly generated trajectories show a similar trend. Student entropy over the full vocabulary rises from 0.3128 to 0.3802, approaching the expert's 0.3820 on the same prefixes (Figure 6). The student is therefore learning the expert's conditional predictions during reasoning, not simply becoming more confident. Figure 7 gives the per-update curves. These process measurements, together with the stage-order and mode ablations, support the reasoning-organization stage. Section 5.3 and Appendix K present the analysis.

We also tested whether the expert can continue these prefixes. On 64 randomly selected questions, each model generated four continuations from the same student thinking prefix. Base, the OFF expert, and the ON expert scored 42.58%, 43.36%, and 43.75%, respectively, with parseable answers throughout (Table 21). This confirms that the OFF-trained expert can continue the student's ON context.

3. What changes with an ON-trained expert?

Table 4 compares the two expert training modes while keeping student OPD in ON:

Expert SFT modeStudent OPD modeMedXpert OFFMedXpert ON
OFFON43.6746.10
ONON42.5946.53

The ON-trained expert does not improve the student's OFF accuracy, and the ON scores are close. Together with the prefix measurements, this supports the division of roles in BWB-KD: OFF SFT acquires domain knowledge, ON OPD applies expert guidance during student reasoning, and RL turns the transferred knowledge into task gains. Section 4.4 now discusses this comparison explicitly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

BWB-KD Rebuttal Summary: Transferring Domain Expertise While Preserving General Capabilities

Dear Area Chair,

We thank you and the reviewers for your time and thoughtful feedback.

Learning a domain expert’s answers and incorporating that expertise into a broadly capable student are different problems. In our experiments, imitating a black-box teacher produces a medical expert, but subsequent RL does not recover the general ability lost in non-thinking inference. BWB-KD instead uses that expert as a teacher. It coordinates knowledge injection through non-thinking SFT, reasoning organization through expert guidance on a separate student’s own thinking trajectories, and knowledge activation through outcome-based RL with thinking switched off. The original submission showed stronger domain performance than direct RL or SFT followed by RL across three biomedical tasks, with the largest gains in non-thinking inference.

During the rebuttal, we added experiments to answer three questions raised by the reviewers: what the student learns during distillation, whether simpler methods can match the gains, and how well the findings carry over to other settings.

What does the middle stage contribute? The original ablations tested its placement and reasoning mode. We tracked what the student predicts on the same thinking prefixes as distillation proceeds. Over 200 updates, overlap between the student’s and expert’s top-32 next-token candidates rose from 80.28% to 87.70%. The student thus learns to use the expert’s guidance along its own reasoning trajectories.

Can simpler methods achieve the same balance? Across three seeds, BWB-KD reached 46.79% non-thinking MedXpert accuracy versus 41.74% for SFT+GRPO, even after quadrupling the baseline’s RL updates pushed its estimated cost above BWB-KD’s. BWB-KD’s general mean remained close to Base (64.11 vs. 64.48). Rank-64 LoRA also preserved general ability, but scored 5.38 points below BWB-KD on MedXpert despite nearly identical output length.

How broadly do the findings hold? The original retention tests covered MedXpert-trained models. After BioASQ and MedCaseReasoning adaptation, BWB-KD also exceeded SFT+GRPO’s non-thinking general mean by 5.21 and 10.53 points, respectively. With fresh responses scored by a second judge, BWB-KD ranked first on both domain tasks in both modes. Its MedXpert advantage over SFT+GRPO also held on a 4B model. Together, these results show why the expert is most useful not as the final policy, but as a teacher that helps the student acquire domain expertise while retaining its broader abilities.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.