- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TangentMix, a state-aware data allocation method for recovering the capabilities of specialized language models while preserving their existing strengths. The authors argue that the effectiveness of training data depends on the current model state, making fixed data mixtures potentially suboptimal. TangentMix evaluates candidate optimizer updates and uses feedback to dynamically adjust data allocation during on-policy distillation. Experiments across multiple recovery settings demonstrate improvements over existing baselines.
Critical strengths
-
The paper investigates an important problem in on-policy distillation, focusing on recovering weakened capabilities while preserving the strengths of specialized language models.
-
The proposed method introduces state-aware data allocation by considering the effects of actual optimizer updates rather than relying solely on static data selection criteria.
-
The paper provides extensive evaluations across multiple capability domains and compares TangentMix with several existing OPD baselines.
Critical weaknesses
-
The fairness of the runtime comparison is not sufficiently clear. Although TangentMix is reported to be faster than OPD and other baselines, it is unclear whether all methods are evaluated under identical hardware conditions and with equally optimized implementations. A fair comparison of complete end-to-end training costs using the same hardware and software environment would be necessary to substantiate the claimed efficiency advantages.
-
The performance improvements are relatively modest in several settings, often within 2 percentage points compared with the strongest baselines. Although the authors report results averaged over three runs, standard deviations and confidence intervals are not provided in the main results. Without further statistical analysis, it is difficult to assess whether these improvements are statistically significant.
-
The proposed framework introduces considerable complexity relative to its performance gains, including multiple candidate updates, counterfactual optimizer steps, calibration panels, and adaptive data allocation. However, the ablation results suggest that adaptive allocation alone accounts for a substantial portion of the improvement, while the additional benefit of update selection is relatively small. It remains unclear whether a simpler adaptive source allocation approach could achieve comparable performance with significantly lower computational overhead.
Questions for authors
See weaknesses!
2 comments
Response to Reviewer hHs6 (Part 2/2)
W3. Given the framework's complexity and that adaptive allocation accounts for much of the ablation gain, could a simpler adaptive allocator achieve comparable performance with lower overhead?
We thank the reviewer for this question. It does not: every simpler state-conditioned controller recovers less of the capability profile, and the full method is also the fastest per update among all methods.
Theory. In the Dyna loop (§4.1 [R8]), allocation (search control), selection (update evaluation, Eq. (7)), and panel-B credit (learning, updating memory via , Eq. (33)) jointly optimize (App. C.7.3). By Eq. (4) and Lemma 6 (Eq. (19)), update value depends on the realized step and recovery–preservation interaction: exposes candidate sources, and extracts the complete update.
Allocation-only is the simpler allocator: without selection (), and allocation reduces to the Eq. (35) demand prior [R11], driven by gap and progress without learning which sources yield good updates at .
Experiments.
(1) Four-arm ablation & allocation trajectory (§5.4, 100u, Tables 2(b) & 14(b)).
| Variant (100u, Table 2(b)) | Allocation | Selection | Macro-7 | Gain |
|---|---|---|---|---|
| Fixed | – | – | 76.594 | – |
| Selection only | – | ✓ | 76.673 | +0.079 |
| Allocation only | ✓ | – | 77.096 | +0.502 |
| TangentMix (allocation + selection) | ✓ | ✓ | 77.487 | +0.893 |
Selection adds +0.079 under fixed allocation but +0.391 under adaptive allocation ( larger); the interaction (+0.312) is 35% of the joint gain. Across settings, selection replaces in 74.0%–80.0% of steps, with 89.4%/62.8% cross-panel sign agreement vs. 50% random (Figure 3c–d). Mixture weights (%) show how reshapes allocation:
| Update / Variant | reasoning | comp_math | knowledge | code | multilingual_qa |
|---|---|---|---|---|---|
| Steps 1–20 (shared prior) | 17.88 | 17.89 | 17.88 | 4.31 | 8.55 |
| Step 60: Alloc-only / Full | 12.71 / 26.34 | 8.67 / 8.12 | 12.18 / 22.66 | 10.14 / 12.01 | 29.82 / 7.45 |
| Step 100: Alloc-only / Full | 9.19 / 10.08 | 7.20 / 16.42 | 23.71 / 11.77 | 7.89 / 8.77 | 21.12 / 24.73 |
From the shared prior, Allocation-only shifts prematurely to multilingual_qa at step 60 (29.82%) and collapses comp_math/code to ~7.2% by step 100, while Full uses to sustain reasoning, knowledge, and code at step 60 before shifting to multilingual_qa (24.73%) and comp_math (16.42%) at step 100.
(2) Simpler state-conditioned controllers (Dominant, 200u, Table 13(b); full table in fZJt W1).
| Method (Dominant, 200u) | Macro-7 | ProfileScore | TyDiQA |
|---|---|---|---|
| TangentMix (full method, 3 seeds) | 76.22 ± 0.08 | 0.672 ± 0.010 | 55.69 |
| Progress router (no preservation) [R11] | 75.41 | 0.556 | 51.70 |
| Same-panel feedback (no panel B) | 74.97 | 0.563 | 52.33 |
| OPD (uniform sources) | 73.79 | 0.512 | 50.44 |
Every simpler controller trails the full method on Macro-7/ProfileScore and loses >3 points on TyDiQA: the progress router [R11] over-allocates to math/reasoning (51.8% at step 50, compressing code to 7.4%–7.8% and late multilingual_qa to 6.7%, −3.99 TyDiQA vs. full); same-panel feedback rewards the panel-A winner (−3.36 TyDiQA); and removing source memory (75.34 / 0.614) chases short-term reasoning gains (31.2% at step 50, −3.48 TyDiQA).
(3) Cost (Table 12, Figure 11). Only the commit writes state, so candidate evaluation overlaps across stages, cutting same-engine step time from 13.31 s (synchronized serial, 12.79 s bare operator sum) to 7.94 s/update (fastest of all methods; see W1, iUih Q3).
Takeaway & Revisions. Each component protects a different part of the capability profile while running faster per update than every baseline. Revisions update Table 12, Figure 11, and App. E.6 runtime decomposition (W1); add 3-seed 95% t-intervals (§5.2, Table 13(b)) and Table 14(a,c) downside/gap tables (W2); and update §5.4 and App. E.7 (Tables 13(b), 14(b)) with the 4-arm interaction, Eq. (35) reduction, trajectories, and simpler-controller ablations (W3).
References.
[R8] Sutton, Dyna, an Integrated Architecture for Learning, Planning, and Reacting, SIGART'91.
[R11] Sun et al., D$^3$-MOPD: Dynamic Domain Scheduling for Efficient Multi-Teacher Distillation, arXiv'26.
[R17] Burns et al., Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, ICML'24.
[R18] Lopez-Paz & Ranzato, Gradient Episodic Memory for Continual Learning, NeurIPS'17.
[R19] Harmon et al., Mapping Post-Training Forgetting in Language Models at Scale, ICLR'26.
Response to Reviewer hHs6 (Part 1/2)
We thank the reviewer for recognizing the importance of the problem and that TangentMix evaluates actual optimizer updates rather than static selection criteria. Our claim is simple: specialist recovery should be judged on the receiving model's complete capability profile, so a meaningful gain is measured by how much of each capability gap is closed and how little is lost elsewhere. Your three concerns—runtime, effect size, and update selection—all ask what this recovery costs and delivers, and we answer them in that order.
Yes: all methods are timed on the same hardware (2 RTX 4090 GPUs) and settings, and equalizing implementations changes little. Adding sequence packing and response-only scoring to OPD and MOPD reduces their step time by only 4% and 18%. End-to-end wall-clock runtime per update (Appendix E.6, Table 12, Figure 11; full account in iUih Q3):
The gains are well outside seed variation: three seeds (42/43/44) on Dominant (fZJt W1) give standard deviations of 0.08 Macro-7 and 0.010 ProfileScore, with 95% t-interval half-widths (t0.025,2=4.303) of ±0.19 and ±0.024—far below the +1.78 Macro-7 and +0.145 ProfileScore margin over MOPD.
Effect size. In Table 1, the top two baselines differ by only 0.15–0.47 Macro-7 in every setting, while TangentMix leads the best baseline by 0.38–3.95. Since the teacher sits only 8–11 points above each 4B specialist, a 2-point margin on a 79-point scale is a large share of the recoverable gap. Normalizing Macro-7 by the gap each result could close ([R17], (x−xInit)/(xT−xInit)) alongside ProfileScore (%, §5.1):
Downside. A mean lets a large loss on one axis be paid for by gains elsewhere. Following continual-learning and forgetting metrics [R18, R19], we report raw per-axis changes against Init (no clipping, 4 settings × 7 axes = 28 cells, Δc,k=xc,k−xc,kInit, regression rc,k=max(−Δc,k,0)): improved cells (CIC), settings with no axis <−5 (ZCC), worst forgetting MR=maxc,krc,k, mean forgetting RB=281∑c,krc,k, worst-two-axes mean CVaR-2=41∑c21(Δc,(1)+Δc,(2)), and downside-adjusted recovery DAR=41∑c(71∑kΔc,k−71∑krc,k2):
TangentMix is the only method with a positive worst-two-axes mean (CVaR-2 +1.61 vs. −0.72 to −4.09), the smallest worst regression (MR 4.29 vs. 6.99–25.78), and the lowest regression burden (0.20 vs. 0.45–1.34). At 1.7B, MOPD and CaMOPD lose 25.78 and 24.49 on GSM8K, while TangentMix improves Code (+16.33), MATH500 (+21.00), GSM8K (+22.89), and IFEval (+5.91) in the same model.