acceptodds
Under review as a conference paper at ICLR 2027

TangentMix: Recovering Specialized Language Models through State-Conditioned Data Allocation

Abstract

Specialized language models can acquire strong domain capabilities while developing uneven regressions across others. In on-policy distillation (OPD), controlled interventions show that source exposure can harm its matching capability and the same mixture can reverse its effect as the specialist evolves. These failures expose a mismatch between common data-value signals and the multi-capability effect of the complete update received by the current learner. We propose TangentMix, a state-conditioned data-allocation method that jointly adapts source exposure and complete recovery–preservation update selection. TangentMix evaluates candidate updates from the current model and optimizer state, commits the selected update, and uses independently measured post-selection utility to update future source allocation. Across four recovery settings spanning 1.7B and 4B students and a broad capability suite covering world knowledge, compositional reasoning, multilingual comprehension, mathematics, code generation, and instruction following, TangentMix improves 92.9% of evaluated capability axes over their specialist initialization. On the three 4B specialists, it closes 79.1% of the aggregate recovery gap and achieves the highest Macro-7 in every profile; across all four settings, it raises the average Macro-7 by 4.2% relative to standard OPD. It further improves competition-level AIME24/25 reasoning by 1.88 points over OPD and achieves 2× data efficiency over fixed-mixture MOPD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Multi-Task Learning, Model Post-Training, Data Valuation, Data Selection, Catastrophic Forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hHs6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TangentMix, a state-aware data allocation method for recovering the capabilities of specialized language models while preserving their existing strengths. The authors argue that the effectiveness of training data depends on the current model state, making fixed data mixtures potentially suboptimal. TangentMix evaluates candidate optimizer updates and uses feedback to dynamically adjust data allocation during on-policy distillation. Experiments across multiple recovery settings demonstrate improvements over existing baselines.

Critical strengths
  1. The paper investigates an important problem in on-policy distillation, focusing on recovering weakened capabilities while preserving the strengths of specialized language models.

  2. The proposed method introduces state-aware data allocation by considering the effects of actual optimizer updates rather than relying solely on static data selection criteria.

  3. The paper provides extensive evaluations across multiple capability domains and compares TangentMix with several existing OPD baselines.

Critical weaknesses
  1. The fairness of the runtime comparison is not sufficiently clear. Although TangentMix is reported to be faster than OPD and other baselines, it is unclear whether all methods are evaluated under identical hardware conditions and with equally optimized implementations. A fair comparison of complete end-to-end training costs using the same hardware and software environment would be necessary to substantiate the claimed efficiency advantages.

  2. The performance improvements are relatively modest in several settings, often within 2 percentage points compared with the strongest baselines. Although the authors report results averaged over three runs, standard deviations and confidence intervals are not provided in the main results. Without further statistical analysis, it is difficult to assess whether these improvements are statistically significant.

  3. The proposed framework introduces considerable complexity relative to its performance gains, including multiple candidate updates, counterfactual optimizer steps, calibration panels, and adaptive data allocation. However, the ablation results suggest that adaptive allocation alone accounts for a substantial portion of the improvement, while the additional benefit of update selection is relatively small. It remains unclear whether a simpler adaptive source allocation approach could achieve comparable performance with significantly lower computational overhead.

Questions for authors

See weaknesses!

2 comments
AuthorsOpenReview ↗

Response to Reviewer hHs6 (Part 1/2)

We thank the reviewer for recognizing the importance of the problem and that TangentMix evaluates actual optimizer updates rather than static selection criteria. Our claim is simple: specialist recovery should be judged on the receiving model's complete capability profile, so a meaningful gain is measured by how much of each capability gap is closed and how little is lost elsewhere. Your three concerns—runtime, effect size, and update selection—all ask what this recovery costs and delivers, and we answer them in that order.

W1. Do all methods in the runtime comparison use identical hardware and equally optimized implementations?

Yes: all methods are timed on the same hardware (2 RTX 4090 GPUs) and settings, and equalizing implementations changes little. Adding sequence packing and response-only scoring to OPD and MOPD reduces their step time by only 4% and 18%. End-to-end wall-clock runtime per update (Appendix E.6, Table 12, Figure 11; full account in iUih Q3):

MethodOPDMOPDTIDECaMOPDDIEM-OPDGKDOPSDTangentMix
Seconds / update ↓12.9513.9914.4215.9127.6631.0357.017.94
Relative runtime1.63×1.76×1.82×2.00×3.49×3.91×7.18×1.00×
  • Aligned settings: All methods use 2 RTX 4090 GPUs (Appendix D.1), the same student, LoRA learner, teacher, prompt/response lengths (512/256), 8 committed trajectories per update (§5.1), and timed unit (one complete update).
  • Equally optimized baselines: Beyond packing and response-only scoring (4% and 18% faster for OPD and MOPD), the remaining speed advantage is structural: only the commit writes state, so independent operators overlap within and across stages (max⁡(Trollpipe,Tlearnpipe)\max(T_{\mathrm{roll}}^{\mathrm{pipe}},T_{\mathrm{learn}}^{\mathrm{pipe}}) vs. the same-engine synchronized serial sum Trollseq+Tlearnseq=13.31 sT_{\mathrm{roll}}^{\mathrm{seq}}+T_{\mathrm{learn}}^{\mathrm{seq}}=\mathbf{13.31\text{ s}}, including 12.79 s of operators and 0.52 s of host overhead), while candidate evaluation stays exact at MtM_t. TangentMix does more work per update (KK temporary steps plus panel scoring, linear in KK, no extra rollout), yet runs at 7.94 s/update (~26.5 min for 200u; see iUih Q3 above).

W2. Gains are often within 2 points of the strongest baseline, without standard deviations or confidence intervals.

The gains are well outside seed variation: three seeds (42/43/44) on Dominant (fZJt W1) give standard deviations of 0.08 Macro-7 and 0.010 ProfileScore, with 95% t-interval half-widths (t0.025,2=4.303t_{0.025,2}=4.303) of ±0.19 and ±0.024—far below the +1.78 Macro-7 and +0.145 ProfileScore margin over MOPD.

Effect size. In Table 1, the top two baselines differ by only 0.15–0.47 Macro-7 in every setting, while TangentMix leads the best baseline by 0.38–3.95. Since the teacher sits only 8–11 points above each 4B specialist, a 2-point margin on a 79-point scale is a large share of the recoverable gap. Normalizing Macro-7 by the gap each result could close ([R17], (x−xInit)/(xT−xInit)(x-x^{\mathrm{Init}})/(x^{\mathrm{T}}-x^{\mathrm{Init}})) alongside ProfileScore (%, §5.1):

Setting (200u)Macro-7 gap closedProfileScore (%)Gain over Init, relative to best baseline
Complementary 4B89.1% vs. 81.3%74.2% vs. 72.0%+9.7%
Dominant 4B74.0% vs. 57.4%67.2% vs. 52.7%+28.9%
Retention 4B80.8% vs. 76.7%71.5% vs. 68.5%+5.4%
Cross-scale 1.7B36.0% vs. 23.1%41.3% vs. 32.0%+55.9%

Downside. A mean lets a large loss on one axis be paid for by gains elsewhere. Following continual-learning and forgetting metrics [R18, R19], we report raw per-axis changes against Init (no clipping, 4 settings × 7 axes = 28 cells, Δc,k=xc,k−xc,kInit\Delta_{c,k}=x_{c,k}-x_{c,k}^{\mathrm{Init}}, regression rc,k=max⁡(−Δc,k,0)r_{c,k}=\max(-\Delta_{c,k},0)): improved cells (CIC), settings with no axis <−5<-5 (ZCC), worst forgetting MR=max⁡c,krc,k\mathrm{MR}=\max_{c,k}r_{c,k}, mean forgetting RB=128∑c,krc,k\mathrm{RB}=\frac{1}{28}\sum_{c,k}r_{c,k}, worst-two-axes mean CVaR-2=14∑c12(Δc,(1)+Δc,(2))\mathrm{CVaR\text{-}2}=\frac{1}{4}\sum_{c}\frac{1}{2}(\Delta_{c,(1)}+\Delta_{c,(2)}), and downside-adjusted recovery DAR=14∑c(17∑kΔc,k−17∑krc,k2)\mathrm{DAR}=\frac{1}{4}\sum_{c}\big(\frac{1}{7}\sum_k\Delta_{c,k}-\sqrt{\frac{1}{7}\sum_k r_{c,k}^2}\big):

MethodCIC ↑ZCC ↑MR (pp) ↓RB (pp/cell) ↓CVaR-2 (pp) ↑DAR (pp) ↑
OPD23/283/46.990.45−0.72+4.30
GKD22/282/47.860.80−2.04+4.02
MOPD23/283/425.781.16−2.40+0.75
TIDE23/283/48.920.75−1.58+4.34
CaMOPD22/283/424.491.34−4.09−0.42
TangentMix26/284/44.290.20+1.61+7.95

TangentMix is the only method with a positive worst-two-axes mean (CVaR-2 +1.61 vs. −0.72 to −4.09), the smallest worst regression (MR 4.29 vs. 6.99–25.78), and the lowest regression burden (0.20 vs. 0.45–1.34). At 1.7B, MOPD and CaMOPD lose 25.78 and 24.49 on GSM8K, while TangentMix improves Code (+16.33), MATH500 (+21.00), GSM8K (+22.89), and IFEval (+5.91) in the same model.

AuthorsOpenReview ↗

Response to Reviewer hHs6 (Part 2/2)

W3. Given the framework's complexity and that adaptive allocation accounts for much of the ablation gain, could a simpler adaptive allocator achieve comparable performance with lower overhead?

We thank the reviewer for this question. It does not: every simpler state-conditioned controller recovers less of the capability profile, and the full method is also the fastest per update among all methods.

Vt(pt,σt)=∑s∈Spt,sE[Ut(σt(At,s))]  (34),Ct=UB(at⋆)−UB(a0)  (32).V_t(p_t,\sigma_t)=\sum_{s\in\mathcal{S}}p_{t,s}\mathbb{E}\big[U_t\big(\sigma_t(\mathcal A_{t,s})\big)\big]\ \ (34),\qquad C_t=U_B(a_t^\star)-U_B(a_0)\ \ (32).

Theory. In the Dyna loop (§4.1 [R8]), allocation ptp_t (search control), selection σt\sigma_t (update evaluation, Eq. (7)), and panel-B credit CtC_t (learning, updating memory via tanh⁡(Ct/cfb)\tanh(C_t/c_{\mathrm{fb}}), Eq. (33)) jointly optimize Vt(pt,σt)V_t(p_t,\sigma_t) (App. C.7.3). By Eq. (4) and Lemma 6 (Eq. (19)), update value −∇Φt(θt)⊤Δt(a)-\nabla\Phi_t(\theta_t)^\top\Delta_t(a) depends on the realized step Δt(a)\Delta_t(a) and recovery–preservation interaction: ptp_t exposes candidate sources, and σt\sigma_t extracts the complete update.

Allocation-only is the simpler allocator: without selection (at⋆=a0a_t^\star=a_0), Ct≡0C_t\equiv 0 and allocation reduces to the Eq. (35) demand prior [R11], driven by gap and progress without learning which sources yield good updates at MtM_t.

Experiments.

(1) Four-arm ablation & allocation trajectory (§5.4, 100u, Tables 2(b) & 14(b)).

Variant (100u, Table 2(b))AllocationSelectionMacro-7Gain
Fixed––76.594–
Selection only–✓76.673+0.079
Allocation only✓–77.096+0.502
TangentMix (allocation + selection)✓✓77.487+0.893

Selection adds +0.079 under fixed allocation but +0.391 under adaptive allocation (4.9×4.9\times larger); the interaction (+0.312) is 35% of the joint gain. Across settings, selection replaces a0a_0 in 74.0%–80.0% of steps, with 89.4%/62.8% cross-panel sign agreement vs. 50% random (Figure 3c–d). Mixture weights ptp_t (%) show how CtC_t reshapes allocation:

Update / Variantreasoningcomp_mathknowledgecodemultilingual_qa
Steps 1–20 (shared prior)17.8817.8917.884.318.55
Step 60: Alloc-only / Full12.71 / 26.348.67 / 8.1212.18 / 22.6610.14 / 12.0129.82 / 7.45
Step 100: Alloc-only / Full9.19 / 10.087.20 / 16.4223.71 / 11.777.89 / 8.7721.12 / 24.73

From the shared prior, Allocation-only shifts prematurely to multilingual_qa at step 60 (29.82%) and collapses comp_math/code to ~7.2% by step 100, while Full uses CtC_t to sustain reasoning, knowledge, and code at step 60 before shifting to multilingual_qa (24.73%) and comp_math (16.42%) at step 100.

(2) Simpler state-conditioned controllers (Dominant, 200u, Table 13(b); full table in fZJt W1).

Method (Dominant, 200u)Macro-7ProfileScoreTyDiQA
TangentMix (full method, 3 seeds)76.22 ± 0.080.672 ± 0.01055.69
Progress router (no preservation) [R11]75.410.55651.70
Same-panel feedback (no panel B)74.970.56352.33
OPD (uniform sources)73.790.51250.44

Every simpler controller trails the full method on Macro-7/ProfileScore and loses >3 points on TyDiQA: the progress router [R11] over-allocates to math/reasoning (51.8% at step 50, compressing code to 7.4%–7.8% and late multilingual_qa to 6.7%, −3.99 TyDiQA vs. full); same-panel feedback rewards the panel-A winner (−3.36 TyDiQA); and removing source memory (75.34 / 0.614) chases short-term reasoning gains (31.2% at step 50, −3.48 TyDiQA).

(3) Cost (Table 12, Figure 11). Only the commit writes state, so candidate evaluation overlaps across stages, cutting same-engine step time from 13.31 s (synchronized serial, 12.79 s bare operator sum) to 7.94 s/update (fastest of all methods; see W1, iUih Q3).

Takeaway & Revisions. Each component protects a different part of the capability profile while running faster per update than every baseline. Revisions update Table 12, Figure 11, and App. E.6 runtime decomposition (W1); add 3-seed 95% t-intervals (§5.2, Table 13(b)) and Table 14(a,c) downside/gap tables (W2); and update §5.4 and App. E.7 (Tables 13(b), 14(b)) with the 4-arm interaction, Eq. (35) reduction, trajectories, and simpler-controller ablations (W3).

References.

[R8] Sutton, Dyna, an Integrated Architecture for Learning, Planning, and Reacting, SIGART'91.

[R11] Sun et al., D$^3$-MOPD: Dynamic Domain Scheduling for Efficient Multi-Teacher Distillation, arXiv'26.

[R17] Burns et al., Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, ICML'24.

[R18] Lopez-Paz & Ranzato, Gradient Episodic Memory for Continual Learning, NeurIPS'17.

[R19] Harmon et al., Mapping Post-Training Forgetting in Language Models at Scale, ICLR'26.

Reviewer_fZJtOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes TangentMix for recovering weakened capabilities of specialized language models while preserving their existing strengths. The method treats data utility as dependent on the current model and optimizer state. At each step, it constructs several recovery-preservation candidate batches, temporarily applies their optimizer updates from a shared snapshot, selects an update using calibration panel A, and uses independent feedback from panel B to adapt future source allocation. Experiments cover four Qwen3 specialist-recovery settings, multiple capability benchmarks, and several OPD-based baselines.

Critical strengths

S1: Important Problem and Relatively Strong Empirical Results. Recovering regressed capabilities without sacrificing a specialized language model's acquired strengths is an important problem for multi-stage post-training. The evaluation spans multiple recovery profiles, model scales, capability dimensions, and competitive baselines, and TangentMix reports relatively strong aggregate results in the evaluated settings.

Critical weaknesses

W1: Overly Complex Method without a Clear Methodological Motivation. The paper begins from a reasonable observation that data utility depends on the current learner state, but does not explain why this requires the full TangentMix pipeline. The method combines dynamic allocation, recovery-preservation candidate construction, counterfactual optimizer steps, two calibration panels, a predictive-Fisher penalty, source memory, and periodic reallocation. The necessity of this elaborate design is not established, and the ablation does not show that a much simpler state-aware method could not achieve comparable gains.

W2: Obscure Presentation and Inadequate Related-Work Positioning. The core method is difficult to follow because it is presented through dense notation and interdependent definitions, with essential execution details deferred to the appendix. Figure 2 is also difficult to understand. Moreover, the main text does not provide a clear related-work discussion or adequately position TangentMix against prior work on dynamic data mixing, sample selection, and capability recovery.

Questions for authors

Q1: Can the authors identify the minimal algorithmic design required for state-conditioned allocation and demonstrate why TangentMix's additional components are necessary relative to a substantially simpler learner-aware allocation or update-selection baseline?

2 comments
AuthorsOpenReview ↗

Response to Reviewer fZJt (Part 1/2)

We thank the reviewer for recognizing that recovering regressed capabilities without sacrificing acquired strengths is an important problem, and that data utility depends on the learner state (Section 3). Our claim is simple: specialist recovery should be judged on the receiving model’s complete capability profile, so the unit TangentMix values is the complete update the specialist receives at its current state. The reviewer’s two concerns, whether every component is needed and whether the method and its positioning are clear, both concern this unit, and we answer them in that order.

W1 / Q1. The method combines source allocation, candidate updates, a predictive-Fisher term, two evaluation panels and source memory. Is all of this necessary, and would a simpler learner-state-aware method suffice?

We thank the reviewer for asking whether a simpler learner-state-aware method suffices. Each component is necessary: we show why theoretically and verify it experimentally.

(1) allocation: st∼pt(2) selection: at⋆=arg⁡max⁡a∈AtJtA(a) (Eq. 7)\text{(1) allocation: } s_t\sim p_t\qquad \text{(2) selection: } a_t^\star=\arg\max_{a\in\mathcal A_t}J_t^A(a)\ \text{(Eq. 7)}

(3) feedback: Ct=UB(Mt,at⋆)−UB(Mt,a0),pt+1=Update(pt,Ct) (§4.4)\text{(3) feedback: } C_t=U_B(M_t,a_t^\star)-U_B(M_t,a_0),\qquad p_{t+1}=\mathrm{Update}(p_t,C_t)\ \text{(§4.4)}

Theory. Following the Dyna decomposition of Section 4.1 [R8], each update makes three decisions (above).

  • Search control: allocation (1) decides which capability gap to visit.
  • Update evaluation: selection (2) decides which complete update the student receives, judged on recovery and preservation together.
  • Learning: feedback (3) measures on an independent panel B how much the chosen update helped over a0a_0, and sets the next allocation. By Eq. (4), an update’s value −∇Φt(θt)⊤Δt(a)-\nabla\Phi_t(\theta_t)^{\top}\Delta_t(a) depends on the displacement the optimizer actually produces and on the recovery–preservation interaction (Lemma 6, Eq. (19)).

Each simpler method misses one of these: Per-example scores ignore Δt(a)\Delta_t(a), progress routers have no preservation term, and same-panel feedback is biased toward the winner. The predictive-Fisher term, source memory and the bound on UU in JtAJ_t^A keep (2) and (3) stable.

Experiments. Section 5.4 (Table 2(b)) ablates allocation and selection; following the reviewer's request, we add simpler state-aware alternatives and single-component removals:

Method (Dominant, 200 updates)Macro-7ProfileScoreTyDiQAIFEval
TangentMix (full method, 3 seeds)76.22 ± 0.080.672 ± 0.01055.6982.26
(a) Simpler state-aware alternatives
OPD (uniform sources) [R2]73.790.51250.4482.44
Static selector: IFD [R13]74.720.57653.7682.26
Static selector: Gap [R4]74.500.50752.7980.78
Static selector: LESS-style [R14]75.480.61159.9880.22
Static selector: DSIR [R12]74.610.55152.7182.81
Progress router, no preservation branch [R11]75.410.55651.7080.78
Same-panel feedback74.970.56352.3379.85
(b) Removing one component
No predictive-Fisher term76.010.66355.0882.62
No source memory75.340.61452.2181.89
Unbounded utility U74.760.56155.3281.52

Read against the complete capability profile, every variant is below the full method on both Macro-7 and ProfileScore:

  • (i) w/o complete-update evaluation. OPD and static selectors score data without the displacement the optimizer actually produces (Eq. (4)), so they cannot see interference across capabilities. In the Section 3 diagnostic, all four selectors stay below Random (Figure 1d). On Dominant the strongest, LESS-style [R14], reaches higher TyDiQA (59.98) only with lower IFEval (−2.04 vs. full), Code (−3.68 vs. full, Table 7), and overall recovery (75.48 / 0.611 vs. 76.22 / 0.672).
  • (ii) w/o preservation branch. A progress router [R11] over-allocates to math/reasoning (51.8% at step 50, compressing code to 7.4%–7.8% and late-stage multilingual_qa to 6.7%), recovering the targeted axes (MMLU 78.24, GSM8K 92.49, MATH500 76.20) while finishing 3.99 points below the full method on TyDiQA (51.70 vs. 55.69) and dropping to 0.556 ProfileScore.
  • (iii) w/o independent feedback. Crediting sources on the selection panel rewards the update that already won selection; it finishes 3.36 lower on TyDiQA and 2.41 lower on IFEval than the full method.
  • (iv) w/o stabilizers. The predictive-Fisher term limits output drift on non-targeted capabilities. Source memory stabilizes allocation (3); without it, the controller chases short-term reasoning/comp_math gains (reasoning 31.2% at step 50), crowding out multilingual_qa and finishing 3.48 lower on TyDiQA than the full method. Removing the bound on U in selection (2) gives the largest drop in part (b) (−1.46 / −0.111).

Takeaway: each component protects a different part of the receiving model’s capability profile, which is why we keep these components.

AuthorsOpenReview ↗

Response to Reviewer fZJt (Part 2/2)

W2. Section 4 and Figure 2 are hard to follow, and the positioning against related work is unclear.

We thank the reviewer for the concrete feedback on Section 4, Figure 2 and related work. We clarify all three around the paper’s core object: the complete update the specialist receives at its current state. Below we restate the method in these terms and position related work by what each method values.

The method as one loop. Following the Dyna decomposition of Section 4.1 [R8] (Appendix C.1), each OPD update has three modules:

pt→st→(At,a0)⏞search control (§4.2) ⟶ at⋆=arg⁡max⁡a∈AtJtA(a)⏞update evaluation (§4.3) ⟶ Mt+1=OptStep(Mt,gt(at⋆)),  Ct→pt+1⏞learning (§4.4)\overbrace{p_t\rightarrow s_t\rightarrow(\mathcal A_t,a_0)}^{\text{search control (§4.2)}}\ \longrightarrow\ \overbrace{a_t^\star=\arg\max_{a\in\mathcal A_t}J_t^A(a)}^{\text{update evaluation (§4.3)}}\ \longrightarrow\ \overbrace{M_{t+1}=\mathrm{OptStep}\big(M_t,g_t(a_t^\star)\big),\ \ C_t\rightarrow p_{t+1}}^{\text{learning (§4.4)}}

  • Search control picks a recovery source by ptp_t and builds candidate complete updates At\mathcal A_t with a reference a0a_0.
  • Update evaluation plays the role of planning. Instead of a learned model, it takes the exact counterfactual step Mt+(a)M_t^+(a) from a saved parameter–optimizer snapshot, scores it on panel A (Eqs. (6)–(7)) and rolls it back.
  • Learning commits only the selected update, once, and its gain CtC_t on the independent panel B sets the next allocation.

Each module answers one question: which data to consider, which complete update to apply, and how much that choice was worth.

Related work, by what each method values:

  • Dynamic mixture controllers (DoReMi [R9], DoGE [R10], D$^3$-MOPD [R11]) optimize source exposure;
  • data-selection methods (DSIR [R12], IFD [R13], LESS [R14], MATES [R15]) value individual examples with static or offline influence signals;
  • OPD variants (GKD [R1], OPD [R2], TIDE [R5], DIEM-OPD [R6], OPSD [R7]) change the token-level distillation loss;
  • capability-recovery and multi-domain OPD methods (MOPD [R3], CaMOPD [R4], Nemotron-Cascade 2 [R16]) fix or calibrate a static or stage-wise mixture.

TangentMix values the complete update the specialist receives now, judged on recovery and preservation together, as an outer controller on top of these losses and optimizers.

Revisions. Section 4 clarifies the Dyna loop and the soft-constraint link between Eq. (6) and Eq. (3) (with Algorithm 1 in Appendix C.2 and the notation reference in Table 3, matching the module flow pt→At→at⋆→Mt+1,Ctp_t\rightarrow\mathcal A_t\rightarrow a_t^\star\rightarrow M_{t+1},C_t in Figure 2). Appendix A adds the related-work positioning paragraph above together with a capability-coverage table comparing the evaluation axes across prior work, and Appendix E.7 adds the two-part component-ablation table (Table 13(b), W1).

References.

[R1] Agarwal et al., On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, ICLR'24.

[R2] Lu & Lab, On-Policy Distillation, Thinking Machines Lab'25.

[R3] Ma et al., MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training, arXiv'26.

[R4] Chen et al., Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation, arXiv'26.

[R5] Yu et al., Mismatch Matters: On-Policy Distillation Beyond Token Agreement, arXiv'26.

[R6] Tan et al., Dynamic Important Example Mining for Reinforcement Finetuning, arXiv'26.

[R7] Zhao et al., Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models, ICML'26.

[R8] Sutton, Dyna, an Integrated Architecture for Learning, Planning, and Reacting, SIGART'91.

[R9] Xie et al., DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining, NeurIPS'23.

[R10] Fan et al., DOGE: Domain Reweighting with Generalization Estimation, ICML'24.

[R11] Sun et al., D$^3$-MOPD: Dynamic Domain Scheduling for Efficient Multi-Teacher Distillation, arXiv'26.

[R12] Xie et al., Data Selection for Language Models via Importance Resampling, NeurIPS'23.

[R13] Li et al., From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning, NAACL'24.

[R14] Xia et al., LESS: Selecting Influential Data for Targeted Instruction Tuning, ICML'24.

[R15] Yu et al., MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models, NeurIPS'24.

[R16] Yang et al., Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation, arXiv'26.

Reviewer_iUihOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

TangentMix is designed to regain the abilities that have been lost while at the same time retaining the specialist's original strengths. It applies several training updates to the same saved state, uses one calibration panel to select an update and another to direct subsequent data sampling. Experiments on the Qwen3-4B and 1.7B models show that they achieve higher average capability scores and exhibit greater efficiency when measured by the committed training trajectories.

Critical strengths

Section 3 shows that more code data can hurt code performance, and the same mixture can help at one checkpoint and hurt at another. These observations give a good reason to adapt the mixture to the current model state.

Critical weaknesses
  1. The averaged utility in Eq. 6 does not enforce the retention constraints in Eq. 3 On Retention, TyDiQA falls from 72.37 to 68.08, v.s. 71.10 for OPD and 72.31 for MOPD. Here, the higher Macro-7 may comes at a greater cost to the specialist's multilingual strength.

  2. The budgets do not seem to line up. Section 5.1 specifies 200 updates, but TangentMix's full Dominant vector in Table 5 matches its 500-update row in Table 10. Its 200-update Macro-7 is 74.18, not 75.78. TIDE and CaMOPD match their 200-update rows, so I am unsure whether the main comparison uses a common budget.

Questions for authors
  1. Which checkpoints and trajectory counts were used in Tables 1 and 5? Do Tables 9 and 10 use the same TangentMix configuration?

  2. How does TangentMix compare with MOPD at similar TyDiQA retention? Reporting Macro-7 and TyDiQA across recovery weights in the Retention profile would help.

  3. Did OPD and MOPD also use sequence packing and response-only scoring? Comparing equally optimized implementations would help clarify the actual overhead of candidate evaluation.

2 comments

Response to Reviewer iUih (Part 1/2)

We thank the reviewer for recognizing that Section 3 gives a good reason to adapt the mixture to the current model state. Our claim is simple: specialist recovery should be judged on the receiving model's complete capability vector under a fixed update budget. The reviewer's concerns—retention on one axis, the training budget, and implementation fairness—are all questions about that vector and that budget, and we answer them in that order.

W1 / Q2. The averaged utility in Eq. (6) does not enforce the retention constraints of Eq. (3). On Retention, TyDiQA falls from 72.37 to 68.08 (vs. 71.10 for OPD and 72.31 for MOPD), so the higher Macro-7 may come at a cost to multilingual strength. How does TangentMix compare with MOPD at similar TyDiQA retention, across recovery weights?

We thank the reviewer for this question and for asking for the matched-retention comparison. Retention in TangentMix is a soft constraint: Eq. (6) carries the retention requirement of Eq. (3) into every update as a weighted term, and the recovery weight λ\lambda sets how strongly it is enforced. Concretely, each update distills recovery trajectories toward the teacher and preservation trajectories toward the frozen specialist (Eq. (5)), and every candidate update is scored by its recovery gain, its drift from the specialist and its output change (Eq. (6)); a lower λ\lambda makes drift from the specialist more expensive. The recovery–retention trade-off is therefore directly controllable, and we test it on Retention at 200 updates:

Method (Retention)UpdatesTyDiQAMacro-7ProfileScore
Specialist (Init)072.3769.770.143
MOPD20072.3173.700.462
TangentMix (recovery weight 0.35)20072.8674.510.511
TangentMix (recovery weight 0.50)20072.5575.460.581
TangentMix (recovery weight 0.65)20071.9575.420.584
TangentMix (default weight 0.80)20068.0877.240.715

Across the three recovery weights, TyDiQA stays at 71.95–72.86 (MOPD: 72.31), while Macro-7 (74.51–75.46) and ProfileScore (0.511–0.584) all exceed MOPD (73.70 / 0.462). At matched or higher TyDiQA (72.86 and 72.55 vs. 72.31), TangentMix leads MOPD by +0.81 and +1.76 Macro-7 (+0.049 and +0.119 ProfileScore), so its Macro-7 lead holds without any multilingual cost. The default weight (0.80) sits further along the same curve. The teacher scores only 57.70 on TyDiQA, so recovering toward it from λ=0.50\lambda=0.50 to 0.800.80 trades 4.47 TyDiQA points for +1.78 Macro-7 and +0.134 ProfileScore, while six of seven axes improve over the specialist. Section 4.3 states explicitly that Eq. (6) is the soft, per-update weighted form of Eq. (3), and Appendix Table 13(a) reports this sweep.

W2 / Q1. The Dominant TangentMix entry appears to come from a longer trajectory than the 200-update budget used for the baselines. Which checkpoints, and how many committed trajectories, underlie Tables 1 and 5?

We thank the reviewer for checking this carefully. In the submitted version, the Dominant TangentMix row in Tables 1 and 5 (75.78 / 0.617) was inadvertently copied from the 500-update trajectory endpoint in Appendix Tables 9 and 10. We have replaced it in Tables 1 and 5 with the actual 200-update checkpoint under the main 200-update protocol used for every baseline; all baseline rows are unchanged. At 200 updates, TangentMix reaches 76.22 Macro-7 / 0.672 ProfileScore on Dominant, against 74.44 / 0.527 for the strongest baseline (MOPD), and improves all seven axes over Dominant Init (MMLU +4.14, BBH +1.48, TyDiQA +6.04, GSM8K +6.67, MATH500 +23.80, Code +7.16, IFEval +6.29).

Q1 (checkpoints and trajectories). Every entry in Tables 1 and 5 is the 200-update checkpoint. Every method runs for 200 optimizer updates and commits bt=8b_t=8 trajectories per update (1,600 committed trajectories in total, Section 5.1); in TangentMix, the four candidates (4+4,6+2,6+2,7+1)(4{+}4, 6{+}2, 6{+}2, 7{+}1) each contain 8 trajectories and share a reusable gradient union of 8 recovery and 4 preservation trajectories per update (Appendix C.8; the 2,400 committed trajectories in Figure 10 come from the 300-update trace in Appendix E.5.2: 300×8300\times 8). Tables 1/5 (main 200-update benchmark) and Appendix Tables 9/10 (long-horizon diagnostic sweeps) share the same codebase and infrastructure but use horizon-specific learning rates and schedules, so the 200-update slice of Tables 9/10 is not a duplicate of Tables 1/5.

AuthorsOpenReview ↗

Response to Reviewer iUih (Part 2/2)

Q3. Did OPD and MOPD also use sequence packing and response-only scoring? Comparing equally optimized implementations would clarify the actual overhead of candidate evaluation.

We thank the reviewer for asking for equally optimized implementations. With sequence packing and response-only scoring added, OPD and MOPD run only 4% and 18% faster per update; the remaining gap comes from intra-stage concurrency and asynchronous pipelining, not from less computation. All methods share the same hardware (2 RTX 4090 GPUs, Appendix D.1), student, LoRA learner, teacher, prompt/response lengths (512/256) and eight committed trajectories per update (Section 5.1). TangentMix reuses candidate-side computation across the 8+4 candidate gradient union (Eq. (29)), and its update dependencies allow independent operators to overlap within and across stages:

Troll=Tgen+Ttch.T_{\mathrm{roll}}=T_{\mathrm{gen}}+T_{\mathrm{tch}}.

Tbase≈Troll+Tfb.T_{\mathrm{base}}\approx T_{\mathrm{roll}}+T_{\mathrm{fb}}.

Tlearn≈Tfb+K[O(∣θLoRA∣)+Tscore(nA)]+Tscore(2nB)⏟candidate evaluation and feedback+Tcommit.T_{\mathrm{learn}}\approx T_{\mathrm{fb}}+\underbrace{K\big[\mathcal{O}(|\theta_{\mathrm{LoRA}}|)+T_{\mathrm{score}}(n_A)\big]+T_{\mathrm{score}}(2n_B)}_{\text{candidate evaluation and feedback}}+T_{\mathrm{commit}}.

TTMseq≈Trollseq+Tlearnseq+Thost.T^{\mathrm{seq}}_{\mathrm{TM}}\approx T^{\mathrm{seq}}_{\mathrm{roll}}+T^{\mathrm{seq}}_{\mathrm{learn}}+T_{\mathrm{host}}.

TTMpipe≈max⁡(Trollpipe,Tlearnpipe).T^{\mathrm{pipe}}_{\mathrm{TM}}\approx\max\big(T^{\mathrm{pipe}}_{\mathrm{roll}},T^{\mathrm{pipe}}_{\mathrm{learn}}\big).

Here, the host term accounts for batching and cache overhead outside the timed operators.

Algorithmically, TangentMix does more work per update than a single-stage baseline: the learning stage adds K temporary optimizer steps and panel scoring, which grow linearly in K and require no additional rollout.

Why the design pipelines naturally. Every operator reads either the frozen learner state Mₜ or the selected winner, and only the commit writes learner state.

(i) Teacher scores are computed per trajectory, so teacher scoring proceeds concurrently while rollout generation continues, shortening the rollout stage.

(ii) The K candidates are counterfactual steps from the same snapshot and are rolled back. Their panel-A scoring and predictive-Fisher terms depend only on Mₜ and the candidates, so these computations run concurrently across the two GPUs.

(iii) Panel-B feedback compares the winner with the reference a₀ at Mₜ, so it runs alongside the commit, shortening the learning stage.

(iv) In pipelined deployment, the next rollout prefetches from θₜ while update t is being evaluated. This introduces a one-step bounded actor–learner lag (Appendix E.6), while candidate evaluation remains exact at Mₜ.

Because the commit is the only sequential barrier, we can separate kernel-level optimization from concurrency on the same engine (Table 12, Figure 11):

  1. Kernel/packing optimization (pure serial, 26.57 s → 13.31 s, 2.00×): Adding sequence packing, response-only scoring, and 1-position Fisher while disabling all concurrency (torch.cuda.synchronize() after every operator) reduces pure-serial step time from 26.57 s to 13.31 s/update (median 13.15 s). The bare sum of the 14 CUDA/HTTP operators is 12.79 s, plus 0.52 s of host batching/cache overhead. This is comparable to the equally optimized OPD (12.43 s) and MOPD (11.47 s) baselines despite evaluating K counterfactual candidates.

  2. Multi-stream and dual-GPU overlap (13.31 s → 7.94 s, 1.68×): Enabling overlap on this identical engine hides calibration rollout (4.02 s serial / 1.95 s merged), teacher HTTP scoring (1.34 s), and GPU-1 replica probes (0.85 s across panels A and B). The slower pipelined stage determines wall-clock time, yielding 7.94 s/update, or approximately 26.5 minutes for 200 updates. The revised Table 12 caption, Figure 11, and Appendix E.6 state the aligned settings, the equally optimized OPD/MOPD timings, and this two-step cost decomposition.

Revisions. Corrected 200-update Dominant row (Tables 1 and 5), unified 200-update / 8-trajectory-per-update checkpoint specification (Appendix D.1/D.3), Retention trade-off and sweep (Section 5.2, Table 13(a)), and Table 12 caption, Figure 11 pipeline schematic, and cost decomposition (Appendix E.6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.