- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes DynaMiCS, a dynamic data-mixture method for improving target-domain performance during LLM fine-tuning while preserving specified capabilities. It uses short, single-domain probes to estimate cross-domain effects, then periodically selects mixture weights through a local linear model with explicit performance constraints. Across 50 scenario configurations and models from 3B to 12B, it reports lower target perplexity and higher constraint feasibility than fixed-mixture baselines.
Critical strengths
- The 50 scenarios thoughtfully vary target domains, auxiliary training data, and preservation constraints, providing a broad assessment of different adaptation–retention trade-offs.
- Ablations on update schedules and probe lengths help characterize the method's sensitivity to these choices.
- The paper reports target improvement, constraint feasibility, and the computational overhead of probing.
Critical weaknesses
- The adaptive baselines tested are largely from 2023–24; newer dynamic SFT methods, such as DynamixSFT, are discussed but not compared experimentally.
- Direct comparisons with other adaptive methods are largely confined to the 3B model, leaving their relative performance at 8B–12B unclear.
- All main fine-tuning runs stop at 2,048 steps, leaving it unclear whether the target gains and preservation constraints remain stable over longer training.
- The contribution appears somewhat incremental relative to recent adaptive multi-domain SFT methods such as IDEAL and VersaTune, and the added benefit of explicit preservation constraints is not clearly isolated.
Questions for authors
- Given the similarities to recent adaptive SFT mixture methods, what is the core methodological novelty of DynaMiCS?
- Could the authors compare against recent dynamic-mixture baselines on the 8B and 12B models under matched compute, and run substantially longer fine-tuning experiments (e.g., 8,192 steps) to assess whether target gains and constraint satisfaction persist?
3 comments
Rebuttal to reviewer SZ3K
W4: "The contribution appears somewhat incremental relative to recent adaptive multi-domain SFT methods such as IDEAL and VersaTune" Q1: "what is the core methodological novelty of DynaMiCS?"
The core novelty of DynaMiCS is making capability preservation an explicit optimization requirement rather than an indirect objective. Existing adaptive mixture methods optimize training efficiency, aggregate performance, or proxies for forgetting, but do not let practitioners specify which capabilities must be preserved and how much degradation is acceptable. DynaMiCS instead formulates mixture selection as a constrained optimization problem, using local cross-domain measurements to improve target performance while respecting user-defined thresholds expressed directly in loss or benchmark accuracy (l. 93–100, 159–172). We will further clarify this positioning and add IDEAL to Section 2 and Appendix A.3.
W4 (cont.): "the added benefit of explicit preservation constraints is not clearly isolated."
Appendix N isolates it. Table 10 runs DynaMiCS with and without constraints, with the same estimate, schedule and solver (l. 2108–2124). Without constraints, target-perplexity reduction rises from 9.5% to 16.7%, but only 11/50 scenarios preserve every protected capability. With constraints, 40/50 do. The constraints are the only difference.
Thanks for the clarification. I appreciate the explanation of the explicit constraints, but I still have concerns about the lack of comparisons with recent methods and the limited training horizon. I would have liked to see more experimental evidence on these points, so I'll keep my original score.
Rebuttal to reviewer SZ3K
We thank reviewer SZ3K for the thoughtful comments and suggestions.
Explicit preservation constraints are a pillar of our contribution. Our paper clarifies this premise up front: existing methods "optimize unconstrained performance objectives: none lets a practitioner name a capability that must be preserved" (l. 75–77). Our first contribution is the constrained formulation (l. 93–95). The work builds on two hypotheses: such constraints matter to practitioners, and available methods ignore them. We ask the reviewer to weigh the baselines against this premise. We will make it explicit in the introduction: "Our premise is that practitioners need explicit preservation guarantees, which current mixing methods do not provide."
The importance of these constraints is supported empirically. DoReMi, Skill-It, and Aioli (ICLR 2025, our closest methodological predecessor) were carefully tuned for our setting (Appendix M), yet satisfy all constraints in only 9–16 of 50 scenarios, compared with 40 for DynaMiCS (Table 9). Crucially, removing the constraints from DynaMiCS itself reduces feasibility from 40/50 to 11/50 (Table 10). As scenarios become more constrained, fixed-weight baselines also deteriorate substantially, while DynaMiCS remains comparatively stable (Fig. 4).
With this context, we believe our results establish two important points. First, our extensive experiments demonstrate that popular mixture-selection methods without explicit preservation constraints increasingly fail as scenarios become more complex. Second, we expect these failures to extend to newer methods that share the same structural limitation. DynamixSFT, which the reviewer cites, adjusts mixture weights using loss-reduction rewards, but cannot explicitly enforce a specified capability-preservation threshold. Given the breadth of our existing comparisons, we believe evaluating another method with the same limitation would just add another empirical data point, rather than change any conclusion.
Our evaluation already includes cross-model comparisons: gradient alignment, our most directly comparable estimator-only baseline, is evaluated on all three model families and consistently underperforms DynaMiCS (Fig. 17, l. 1939–1949). The extensive fixed-weight grid is also evaluated across all models (Fig. 2).
Together, these experiments already provide evidence across model scales. As discussed in W1, the remaining adaptive methods lack explicit preservation constraints, a structural limitation that does not depend on model scale. Given the breadth of our evaluation, we believe extending these comparisons would not change the conclusions of our work.
2,048 steps already involves substantial repetition. A 300-example dataset sampled at weight 0.5 is seen about 27 times, making forgetting and overfitting relevant within this budget.
DynaMiCS's advantage persists throughout training. We examined every evaluation checkpoint. DynaMiCS satisfies all constraints 3.7–5.2× more often than fixed weights during steps 64–1,024, and 5.0–6.7× more often during steps 1,088–2,048. Target improvements also persist. Overall, results show no erosion of DynaMiCS's toward the end of the horizon we evaluated.
Finally, our protocol selects the best feasible checkpoint rather than the final one (l. 1166–1168), so extending training cannot invalidate previously found feasible solutions.