acceptodds
Under review as a conference paper at ICLR 2027

Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation

Abstract

Specialist distillation effectively transfers domain expertise to student models via teacher-generated reasoning trajectories. However, when these specialists are trained solely on question–answer pairs without explicit reasoning supervision, what governs the trajectories they generate? In this work, we show that specialist optimization implicitly selects from this latent trajectory space. To isolate and observe this latent distribution, we leverage student distillation not as a downstream goal, but as an agnostic probe—since students inherit no parameterization or optimization constraints from the specialist, inheriting only the sampled trajectories themselves. Through this probe, our empirical analysis unveils a tight governing relationship: across specialist–student pairings, their specialization–generalization profiles correlate exceptionally strongly. Crucially, explicitly controlling the specialist's distributional drift systematically shifts both the teacher and its distilled student along a controllable trade-off between domain precision and general-capability retention. Across chemistry, physics, and multilingual settings, distilled students systematically reflect these specialist-induced profiles, even across divergent model families. Our findings establish a new view of specialist training: when gold reasoning is absent, tuning choices directly control the latent supervision passed to downstream models.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Large Language Models, Specialist Distillation, Knowledge Distillation, Reasoning, Domain Adaptation, General-Capability Retention, Chain-of-thought Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_aCU8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The setting is specialist distillation when you only have question-answer pairs and no reasoning traces. The authors fine-tune Qwen3-8B on answers only, using full fine-tuning, LoRA, or a layer-selective method. They then sample reasoning traces from that specialist and train a fresh student on them. Across chemistry, physics and translation, the student ends up with roughly the same strengths and weaknesses as its teacher. The authors argue that how far the specialist drifts from the base model is what controls this, and they back it up with a KL-anchored fine-tuning sweep.

Critical strengths

Good question, and the setup is controlled where it matters. Every student is trained the same way on the same amount of data, so differences between the students can be attributed fairly directly to the supervision generated by their corresponding specialists. The KL-anchoring sweep was the most convincing part for me.

Critical weaknesses

I think there is a simpler explanation for the results than the one emphasized in the paper. Full fine-tuning on answer-only targets produces very short and mostly structurally incomplete reasoning traces. Training a student on those traces could explain much of the degradation on harder reasoning tasks. LST moves the model less and preserves much more origin-like reasoning traces, so its student gets better supervision. This seems consistent with a more ordinary distribution-shift or forgetting explanation, without requiring the stronger interpretation of specialist optimization as "trajectory selection." I did not find an experiment that cleanly separates these explanations. The theory section does not resolve this for me either, since the actual training procedure does not marginalize over reasoning trajectories.

I am also not sure how much of some of the smaller differences in Table 2 is robust across training runs. The training appendix says experiments use a fixed seed unless multiple runs are otherwise required, and I could not find repeated training runs for the main specialist/student comparisons. Some reported gaps are under half a point. The 0.957 specialist-student correlation is interesting, and the blocked permutation test helps, but it still pools 27 measurements from only nine specialist-student pairs. Per-domain or profile-level correlations would make it easier to judge how broadly the relationship holds. The comparison to zero-shot Qwen3-32B is also less informative about the proposed distillation mechanism because the 8B specialists receive task-specific fine-tuning.

Questions for authors
  1. What exactly does the specialist's answer-only training target look like? Is it just the final answer, or does it include an empty or otherwise formatted think block? If FFT is partly learning to omit reasoning because of the target format, that would change how I interpret the trajectory-collapse result.

  2. Could you train a student on LST-generated traces truncated to approximately the same length distribution as the FFT traces? If the student still inherits the LST capability profile, that would make the trajectory-selection interpretation substantially more convincing.

Smaller point: an 8B model scoring around 90 on the physics in-task benchmark is striking, especially because roughly 85% of the physics sub-answers fall back to an LLM judge. I also had trouble reconciling the appendix's description of the PHYSICS benchmark with the cited Feng et al. dataset. Please clarify the exact dataset version, subset construction, and evaluation pipeline.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer aCU8 [1/2]

We thank the reviewer for the careful review and the concrete suggestions. We are also glad the reviewer found the KL-anchoring sweep convincing. We address each of your point below.

W1: “A simpler explanation for FFT results”

We agree that, in terms of FFT, truncated or incomplete trajectories are the primary driver of student model degradation. However, we are afraid that this seemingly "simpler" explanation fails to address two issues:

  1. Why trajectories differ so significantly, given the same data and the same objectives, when different training methods are employed: (Also in our response to Reviewer MZeB W2/Q4) With the same data, loss, and hyperparameters, LoRA and LST specialists reach nearly the same teacher-forced fit on unseen SMol queries (HR@1 = 0.86 vs. 0.85, see Fig 4) and comparable in-task accuracy (28.57 vs. 29.61, see Tab 2), yet LST keeps complete reasoning in 92.5% of samples versus 41.4% for LoRA, with traces close to the origin model’s length (2334 vs. 1378 words, see Tab 3). In other words, two specialists (LST/LoRA) that fit the answers (nearly) equally well can reason very differently, so the degree of fit to (q,a) cannot be the one and only factor that decides the specialization-generalization profile. Instead, how the update is constrained, not just how well the answers are fit, decides how much reasoning survives.
  2. How this degradation can bypass filters based on answer correctness, subsequently being propagated to a new "student" model that has not yet experienced any drift.

W2: “Fixed Seed”

The training appendix says experiments use a fixed seed unless multiple runs are otherwise required, and I could not find repeated training runs for the main specialist/student comparisons.

We did only perform the training once for each specialist/student, using a single fixed seed 42. As you appreciated in Critical Strengths, the experiments in our paper are extremely computationally intensive, requiring a vast number of GPU hours, which makes it unfeasible for us to rerun all experiments with different seeds. Maybe a fixed seed could explain the ~1pp performance fluctuation, however, we do not believe that using a fixed seed would bridge the substantial performance gaps between the various training methods presented in Table 2.

The mention of "multiple runs" primarily concerns the evaluation phase. We mainly use OpenCompass for evaluation, a framework that automatically performs multiple runs using different seeds for some predefined benchmarks. For instance, the results for AIME are calculated by taking the average across 32 independent runs (L1836).

W3: “Per-domain or profile-level correlations”

That is an excellent suggestion. In the appendix of our next version, we will include a breakdown of the scores for each benchmark ( IT (14/1/16), ID (4/4/16), and OOD (5) ), which will provide more data for calculating correlation coefficients. Unfortunately, we have too much raw data to include in the rebuttal, but this addition will certainly not affect the conclusions in the main text, if anything, it will only strengthen them. Thank you again for your suggestion.

W4: “Zero-shot Qwen3-32B is less informative”

Our primary intention here is to demonstrate that the first stage, QA-only specialist-training on a smaller model yields better results than intuitively direct distillation from a larger, superior model. This is also precisely the approach taken by frontier models: processes like those used for DeepSeek-V3.2 involve training multiple domain-specific specialists from a single base model, and then distilling the data they generate back into a main model of the same size.

AuthorsOpenReview ↗

Reply to Reviewer aCU8 [2/2]

Q1

We train with LLaMA-Factory’s qwen3_nothink template, which adds no <think> block or no thinking tokens, so the loss covers only the answer, as in the anonymous repository. Each training example has a system prompt asking for a direct answer (“Provide direct answers to the user’s question without showing your reasoning process. Do not use <think> and </think> tags.”), the task instruction and question, and the gold answer as the assistant turns. When sampling trajectories for distillation, we switch to the reasoning system prompt in Fig. 8 with Qwen3’s thinking mode on. However, this format does push all three specialists toward answering without reasoning, and LoRA is clearly affected (complete traces fall from 87.9% for the origin model to 41.4%, Tab 3). The training data alone cannot tell the model whether to answer directly only when told not to think, or for every question whatever the prompt says. Both fit the training set equally well. They only diverge at sampling time, when we ask the model to think. The further a method moves the model from its origin (KL: FFT 3.05 > LoRA 2.00 > LST 1.43, Fig 2b), the more the specialist ignores that request. This is not a matter of trainable parameter count: LST updates roughly 20× more parameters than our LoRA setup, yet it keeps reasoning best.

Q2

This is a great idea for a controlled experiment. While we highly value your suggestion, we are unfortunately unable to carry it out due to limitations in computational resources and the short duration of the rebuttal period. Nevertheless we the authors thank you for your valuable advice.

Q3

Thank you for catching this. Our citation is indeed wrong: we used the PHYSICS dataset of [1]^ not Feng et al. (2025). In D.2.2, we believe we authentically describe the subset construction and evaluation pipeline for PHYSICS despite the citation error.

[1] Shenghe Zheng, Qianjia Cheng, Junchi Yao, Mengsong Wu, Haonan He, Ning Ding, Yu Cheng, Shuyue Hu, Lei Bai, Dongzhan Zhou, Ganqu Cui, Peng Ye: Scaling Physical Reasoning with the PHYSICS Dataset. NeurIPS 2025.

^ Actually we use the version from https://huggingface.co/datasets/desimfj/PHYSICS

Reviewer_n5tdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies specialist distillation in which a specialist model is first fine-tuned on an existing QA dataset, then it is distilled to a student model of the same size. The paper makes several claims, for example that distilled models have a similar performance profile as the specialist model.

Critical strengths
  • The experiments cover a good breadth of domains including Chemistry, physics and multilingual translation.
  • The paper includes extensive experiments.
Critical weaknesses

The motivation for studying the proposed pipeline is a bit unclear. In the main experiments and the intro/motivation for the paper, the specalist model is first fine-tuned on a dataset, and then another student model is distilled from this specialist model. Why not just fine-tune the final model using the original dataset? The paper doesn't provide sufficient justification for why this is an interesting training recipe to study.

The paper needs to be clearer about what new insights it is providing beyond what is already accepted knowledge / practice. For example the insight about students inheriting the generalization profile of their teacher doesn't seem particularly surprising.

Overall the paper could be written more clearly to lay out a coherent logical argument.

Questions for authors

What motivates training an answer-only specialist and then distilling that to another model of the same size?

Reviewer_MZeBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies specialist distillation when the domain data contain only question–answer pairs, with no reference reasoning trajectories. The authors argue that answer-level supervision leaves the specialist’s reasoning trajectories underdetermined, so the choice of specialist training method implicitly determines the supervision passed on to the student.

The pipeline has three stages:

A general-purpose reasoning model (Qwen3-8B) is fine-tuned on the question–answer pairs to obtain a specialist, using full fine-tuning (FFT), LoRA, or layer-selective tuning (LST). In Section 5.2, ASFT, which adds an explicit KL penalty toward the original model, is also evaluated. The specialist generates reasoning traces and answers. Only samples with a correct answer and a complete reasoning block are retained (rejection sampling). A student, initialised from the same base model, is fully fine-tuned wit each specialist.

Experiments cover three domains: chemistry (SMol), university-level physics (MegaScience), and low-resource multilingual translation (OPUS). Each is evaluated on in-task, in-domain, and out-of-domain benchmarks. The authors report the following:

  • The training method strongly affects reasoning behaviour. The FFT specialist largely stops producing reasoning (8.9% complete traces), the LoRA specialist does so partially (41.4%), and the LST specialist preserves it (92.5%).

  • Specialist and student performance profiles are strongly correlated (Spearman ρ = 0.957 over 27 measurements), including across model families.

  • Drift trades specialisation against generalisation. FFT, which drifts furthest from the base model, yields large in-task gains in chemistry but substantial losses in-domain and out-of-domain. LST yields more moderate in-task gains while retaining general capabilities. ncreasing the KL anchoring strength in ASFT shifts both specialists and students from specialisation toward retention of general capabilities.

Critical strengths

This work proposed an interesting way to study the specialise and generalise trade off.

Critical weaknesses
  1. The tight link between teacher and student trained on it is pretty much the expected behaviour.
  2. The stronger fit to the (q,a) pair in the specialised domain, the less the model would reason - this is defined by loss and optimisation. This is not a new observation
  3. Reasoning “quality” is measured only by structure, and reasoning doesn’t explain in-task gains
  4. The “probe” only covers an easy case - this claim may not be correct when the teacher-student model capability gap to large – this is actually what motivates OPD from the off-policy distillation
Questions for authors
  1. What does the inheritance result add beyond standard distillation? The student is trained to minimise the KL to the specialist, so inheriting its profile is expected.

  2. Does preserving reasoning help domain performance, or only reduce forgetting? The FFT specialist and its student achieve the best Chemistry in-task scores despite collapsed reasoning, and “quality” is measured only by completeness and length.

  3. Do the conclusions hold for a stronger teacher? All main results distil 8B into 8B, the easiest setting for off-policy distillation, and one where distillation brings little practical benefit.

  4. Is this trajectory selection (as stated in contribution 1), or answer-only fine-tuning simply suppressing thinking

2 comments
AuthorsOpenReview ↗

Reply to Reviewer MZeB

Thank you for recognizing our investigation on specialization-generalization trade-off. We would like to take this opportunity to address your concerns here.

W1/Q: “That students resemble their teachers is expected”

Taking our training domain Chemistry as an example, we would like to draw reviewer’s attention that the intuition on standard distillation would only imply that “a student will resemble its teacher on chemistry domain”, it is not implied that “the teacher’s performance degradation on out-of-domain (OOD) tasks like AIME (math) or LiveCodeBench (coding) would also be inherited by the student.” Yet, even though the student never encountered OOD data (one sees only chemistry trajectories that yielded correct answers), the OOD degradation was nonetheless transferred: the 8B FFT-trained student saw its OOD performance drop to 25.2 (Tab 2 in Sec 4.1), while the 14B version dropped to 3.46 (Tab 5 in App. B.5). This effect persists even across different model families (e.g., from Intern-S1-mini to Qwen3, see Tab 4 in App B.2). This demonstrates that the common rejection sampling, which focuses solely on whether an answer is correct, fails to capture the underlying locations of "breakdowns" in the teacher model. This is a practically significant conclusion that cannot be derived from the framework of "standard distillation." If you are aware of any existing literature that already discusses how/why a student model can inherit its teacher’s OOD performance profile without sampling OOD data from the teacher, please do let us know. We are quite willing to read and cite such papers, even though doing so ends up narrowing the scope of our own innovation.

W2/Q4: “Stronger QA-only fitting leads to less reasoning, nothing new”

We agree that answer-only SFT can suppress reasoning, and we do not claim that this phenomenon is something new or innovative. However, our results show the specialization-generalization profile is not determined by the fit to (q,a) alone. With the same data, loss, and hyperparameters, LoRA and LST specialists reach nearly the same teacher-forced fit on unseen SMol queries (HR@1 = 0.86 vs. 0.85, see Fig 4) and comparable in-task accuracy (28.57 vs. 29.61, see Tab 2), yet LST keeps complete reasoning in 92.5% of samples versus 41.4% for LoRA, with traces close to the origin model’s length (2334 vs. 1378 words, see Tab 3). In other words, two specialists (LST/LoRA) that fit the answers (nearly) equally well can reason very differently, so the degree of fit to (q,a) cannot be the one and only factor that decides the specialization-generalization profile. Instead, how the update is constrained, not just how well the answers are fit, decides how much reasoning survives. Our contribution is to show that this choice changes what the distillation data carries: the effect passes through answer-correctness filtering and is inherited by students (Sec 4.2, 5.2).

W3/Q2: “Reasoning “quality” is measured only by structure”

We agree that in terms of IT, FFT-specialist and distilled students are indeed the best (59.65 / 40.55 as in Tab 2), but its reasoning capabilities have essentially collapsed. However, our paper’s claim was originally about the specialization-generalization trade-off, not that “reasoning improves IT” or that “QA-only fitting improves IT”.

W4/Q3: “Only covered 8B→8B easy case”

We would like the Reviewer to read our appendices. In B.5, we used Qwen3-14B as an origin. Due to limited computational resources, we are unable to cover larger models. In B.3, we implemented 14B→8B cases. In B.2, we covered the Intern-S1-mini model as a teacher and Qwen3-8B as a student, i.e., cross-model transferability. Given our computational budget, we believe we have done our utmost to demonstrate the generalizability of our findings and their potential to scale up.

Thanks to the authors for the rebuttal. After carefully reviewing the rebuttal and comments from the other reviewers, I would keep my initial score.

First, it is unclear whether the student's OOD degradation is specifically inherited from the teacher or is a consequence of domain-specific fine-tuning. The current results establish a strong correlation, but do not fully distinguish these explanations.

Second, and perhaps more importantly, I would like to understand how this observation informs the methodology design. Even if the teacher's OOD degradation is transferred to the student, what new insights does this provide for designing a better specialist distillation strategy?

Reviewer_n42gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates specialist-to-student distillation when specialists are trained only on question–answer pairs without gold reasoning trajectories. It frames specialist optimization as implicitly shaping an underdetermined distribution of reasoning trajectories and uses controlled student distillation as an agnostic probe: students inherit only the specialists’ generated and filtered trajectories, rather than their parameters or optimization constraints. The paper shows that specialists’ specialization–generalization profiles are strongly reflected in their distilled students, and identifies distributional drift from the base model as a controllable axis of this trade-off. Layer-Selective Tuning provides implicit drift control, while KL-anchored SFT provides explicit control, enabling both specialists and students to trade target-domain gains against retention of general capabilities. These findings are evaluated across chemistry, physics, and low-resource multilingual tasks using FFT, LoRA, and LST, with additional ASFT experiments on chemistry and multilingualism to study explicit drift control. Additional analyses examine trajectory integrity, KL drift, data scaling, larger models, and cross-family and cross-scale transfer.

Critical strengths

The experimental perspective of using "student distillation as a probe" is notably innovative. Rather than viewing distillation merely as a method to enhance student performance, the paper leverages the fact that students do not inherit the parameters or optimization constraints of the expert—receiving only the trajectories generated by the expert—to examine how the supervisory differences resulting from various expert training approaches propagate to downstream models. While this design may not yet fully isolate the underlying causal mechanisms, it nonetheless provides a valuable and actionable analytical framework for investigating the relationship between "QA-only" expert training and the quality of distilled data.

The experimental scope is extensive, revealing a consistent correspondence between the capabilities of the expert and the student. The paper compares FFT, LoRA, LST, and ASFT across domains such as chemistry, physics, and low-resource multilingual tasks, while further extending the analysis to cover different model families, varying model scales, data scaling, and the integrity of the generated trajectories.

Critical weaknesses
  1. There is a discrepancy between the theoretical framework presented in the paper and the actual training procedure. The paper reformulates "answer likelihood" as a marginal probability over latent trajectories and derives a "trajectory-weighted gradient" based on this. In practice, however, the training of the "specialist model" is simply standard supervised fine-tuning (SFT) using "teacher-forcing" on (q,a)(q, a) pairs; no sampling or marginalization of latent trajectories occurs during optimization. Consequently, I argue that this derivation serves merely as an intuitive explanation rather than a formal proof that "QA-only supervised fine-tuning" filters trajectories in the specific manner claimed by the authors. This distinction is critical, as many of the paper’s interpretations rely on this assumption.

  2. The current experiments fail to isolate the effect of "trajectory filtering" from other factors inherent in training "specialist models." Methods such as FFT, LoRA, LST, and ASFT differ in their parameter update constraints and optimization dynamics, leading to specialist models that vary in terms of accuracy, output length, trajectory completeness, and the distribution of generated data. Any of these differences could influence the "student model." In particular, the observed correlation between the teacher model and the student model might simply stem from the fact that a better-performing (or less prone to forgetting) teacher model produces more useful distillation samples. While the experimental results are interesting, they do not prove that the "latent trajectory distribution" is the fundamental driver behind the specific "specialization-generalization" characteristics exhibited by the models.

  3. The specialists also differ sharply in trajectory properties before filtering: for example, only 8.9% of FFT-generated chemistry trajectories are structurally complete, compared with 92.5% for LST. Since final retention additionally depends on answer correctness and task-specific filtering rules, reporting only an equal final data budget is insufficient to establish that the resulting datasets are comparable.

  4. Furthermore, I am not convinced that "KL drift" has been established as a definitive, critical variable. The observed ranking of KL divergence values ​​across FFT, LoRA, and LST is merely an empirical observation. While ASFT serves as a valuable supplementary experiment—by intervening in "anchoring strength"—adjusting this coefficient not only alters the quality of the expert model but also influences which generated samples pass the filtering process. Moreover, this analysis covers only a limited range of domains. A more rigorous approach would be to compare models generated via different methods while keeping KL divergence levels roughly equivalent, or to examine whether the "student effect" persists after controlling for the specific characteristics of the "accepted trajectories." Without such controls or mediation analysis, while "KL drift" may serve as a useful diagnostic tool and a hyperparameter for tuning, it is insufficient to serve as a definitive explanation for the phenomenon of "profile inheritance."

Questions for authors

How comparable are the filtered distillation datasets generated by different "specialist" models? Are the training questions used for the "student" models—FFT, LoRA, and LST—identical? Or are the training sets for each derived from different question pools, consisting of subsets of comparable size that have passed filters for "answer correctness" and "trajectory completeness"? Please report the answer correctness rate, structural completeness rate, and final retention rate for each method, along with the distributions of question difficulty, trajectory length, and task composition. Such details are crucial for determining whether the observed differences among the "student" models stem from the generated trajectories themselves or from selection bias introduced during the filtering process.

Can student performance be predicted equally well by simpler quantities, such as specialist answer accuracy, trajectory acceptance rate, or the length and completeness of retained trajectories? Results using matched question sets or controlled trajectory properties would help determine whether the proposed trajectory-distribution account explains anything beyond these observable measures of teacher and data quality.

In the ASFT experiments, does adjusting the "anchoring coefficient" merely alter the behavioral KL divergence (KL drift), or does it also change the content of the final, filtered dataset? Please provide statistics—such as answer correctness, completeness, trajectory length, and retention rate—for the trajectories generated under different coefficients. If these attributes change significantly, could the authors clarify what evidence suggests that shifts in student model performance are indeed driven by KL divergence, rather than by changes in the composition or quality of the filtered training sets?

1 comment
AuthorsOpenReview ↗

Reply to Reviewer n42g

We sincerely thank the Reviewer n42g who recognizes our experimental perspective of "student distillation as a probe" and respective experiment scopes.

W1

Thank you for this constructive observation. Indeed, we agree that Eqs. 1–3 are an interpretive characterization of how teacher-forced QA-only SFT behaves, not a formal proof. The paper states at L073–074, L154–156 and L235–237 that these eqs are “characterization”. But we admit some phrases may be overstating, so we will revise them accordingly. However, our main claims do not rely on this abstraction. Eqs. 4–5 are the student’s actual objective, and profile inheritance, measured KL drift, ASFT anchoring and trajectory integrity are all empirical. We will separate the exact statements from the interpretive ones in Sec 3 and App. A.

W2

We admit that FFT/LoRA/LST/ASFT specialists differ in many dimensions, and we do not claim the trajectory distribution is the one and only fundamental driver. Instead, our claim is that the specialists’ effects reach students only through the generated data. There are two observations in our paper that could argue against a simple “better teacher → better data” explanation. On LRM, the FFT and LST specialists have nearly identical IT scores (35.93 vs. 35.68), yet their students differ sharply on ID/OOD (27.78/15.08 vs. 43.87/41.23). What transfers is the profile, not overall teacher strength. Self-Distill and Self-Rationalize also share the same base model (πθ0π_{θ_0}) but differ only in how trajectories are produced, and they still yield very different students. With the same data, loss, and hyperparameters, Chemistry LoRA and LST specialists reach nearly the same teacher-forced fit on unseen Chem SMol queries (HR@1 = 0.86 vs. 0.85, see Fig 4) and comparable in-task accuracy (28.57 vs. 29.61, see Tab 2), yet LST keeps complete reasoning in 92.5% of samples versus 41.4% for LoRA, with traces close to the origin model’s length (2334 vs. 1378 words, see Tab 3). In other words, two specialists (LST/LoRA) that fit the answers (nearly) equally well can reason very differently, so the degree of fit to (q,a) cannot be the one and only factor that decides the specialization-generalization profile. Instead, how the update is constrained, not just how well the answers are fit, decides how much reasoning survives.

W3

We have controlled the data budget, training method (FFT), and hyperparameters for all student models; this is likely the best we could possibly do. Firstly, the "thinking completeness" of a specialist, as well as its length, is determined by its upstream training methodology, we cannot control that. Secondly, if we need to exercise precise control over the training data for the students, i.e., ensure that the set of problems remains identical (the question IDs are the same, with only the answers varying based on which specialist generated them), this would be impractical. For example, After filtering for answer correctness and trajectory completeness on 534,805 Chemistry candidates, the retained rates are 10.50% (FFT), 8.13% (LoRA), and 9.81% (LST); the FFT–LoRA–LST intersection is below 0.75% of the original pool. Training at this intersection would introduce a severe data-scarcity confound.

W4

Our evidence shows that KL drift tracks the inherited specialization–generalization profile and can be used to adjust it, while we do not claim KL-drift as a definitive causal explanation. We agree that varying λ\lambda in ASFT changes which samples pass the answer filter, so the composition of the accepted trajectories may mediate part of the effect. However, with the same set of specialist-training-data, different tuning methods naturally reach different KL levels, which is the very difference we observe. Matching KL-drift across FFT/LoRA/LST-specialists therefore requires changing some other factors (say hyperparameters or epochs) that used to be the same in our paper's setting of specialist training, which in turn introduces a new difference between the specialists being compared.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.