acceptodds
Under review as a conference paper at ICLR 2027

DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixtures

Abstract

Multi-domain fine-tuning of large language models requires improving performance on target domains while preserving previous capabilities, such as general knowledge, instruction following, or safety evaluations. Existing data mixing strategies rely on fixed heuristics or adaptive rules that cannot explicitly enforce preservation of such capabilities. We propose DynaMiCS, a dynamic mixture optimizer that casts multi-domain fine-tuning as a constrained optimization problem. At each update, DynaMiCS performs short domain-specific probing runs to estimate a slope matrix of local cross-domain effects, capturing how training on each fine-tuning dataset affects each evaluation domain. These estimates are then used to compute mixture weights through optimization over the probability simplex, with the objective of improving target-domain performance while keeping constrained-domain metrics within a specified tolerance of reference levels. Because these effects are measured by finite differences rather than gradients, targets and constraints need not be differentiable, or present in the fine-tuning data, and can be specified directly as benchmark accuracies. Across scenarios with varying numbers of target and constrained domains, and with loss- or accuracy-based objectives, DynaMiCS achieves stronger target-domain improvements and higher constraint satisfaction than static, dynamic, similarity-based and probing-based alternatives, without a reference model, per-example scoring, or manually tuned weights.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data mixtures, fine-tuning, constrained optimization, forgetting, LLMs, data selection, mixture weights, knowledge preservation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SZ3KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes DynaMiCS, a dynamic data-mixture method for improving target-domain performance during LLM fine-tuning while preserving specified capabilities. It uses short, single-domain probes to estimate cross-domain effects, then periodically selects mixture weights through a local linear model with explicit performance constraints. Across 50 scenario configurations and models from 3B to 12B, it reports lower target perplexity and higher constraint feasibility than fixed-mixture baselines.

Critical strengths
  • The 50 scenarios thoughtfully vary target domains, auxiliary training data, and preservation constraints, providing a broad assessment of different adaptation–retention trade-offs.
  • Ablations on update schedules and probe lengths help characterize the method's sensitivity to these choices.
  • The paper reports target improvement, constraint feasibility, and the computational overhead of probing.
Critical weaknesses
  • The adaptive baselines tested are largely from 2023–24; newer dynamic SFT methods, such as DynamixSFT, are discussed but not compared experimentally.
  • Direct comparisons with other adaptive methods are largely confined to the 3B model, leaving their relative performance at 8B–12B unclear.
  • All main fine-tuning runs stop at 2,048 steps, leaving it unclear whether the target gains and preservation constraints remain stable over longer training.
  • The contribution appears somewhat incremental relative to recent adaptive multi-domain SFT methods such as IDEAL and VersaTune, and the added benefit of explicit preservation constraints is not clearly isolated.
Questions for authors
  • Given the similarities to recent adaptive SFT mixture methods, what is the core methodological novelty of DynaMiCS?
  • Could the authors compare against recent dynamic-mixture baselines on the 8B and 12B models under matched compute, and run substantially longer fine-tuning experiments (e.g., 8,192 steps) to assess whether target gains and constraint satisfaction persist?
3 comments
AuthorsOpenReview ↗

Rebuttal to reviewer SZ3K

We thank reviewer SZ3K for the thoughtful comments and suggestions.

W1: "The adaptive baselines tested are largely from 2023–24; newer dynamic SFT methods, such as DynamixSFT, are discussed but not compared experimentally."

Explicit preservation constraints are a pillar of our contribution. Our paper clarifies this premise up front: existing methods "optimize unconstrained performance objectives: none lets a practitioner name a capability that must be preserved" (l. 75–77). Our first contribution is the constrained formulation (l. 93–95). The work builds on two hypotheses: such constraints matter to practitioners, and available methods ignore them. We ask the reviewer to weigh the baselines against this premise. We will make it explicit in the introduction: "Our premise is that practitioners need explicit preservation guarantees, which current mixing methods do not provide."

The importance of these constraints is supported empirically. DoReMi, Skill-It, and Aioli (ICLR 2025, our closest methodological predecessor) were carefully tuned for our setting (Appendix M), yet satisfy all constraints in only 9–16 of 50 scenarios, compared with 40 for DynaMiCS (Table 9). Crucially, removing the constraints from DynaMiCS itself reduces feasibility from 40/50 to 11/50 (Table 10). As scenarios become more constrained, fixed-weight baselines also deteriorate substantially, while DynaMiCS remains comparatively stable (Fig. 4).

With this context, we believe our results establish two important points. First, our extensive experiments demonstrate that popular mixture-selection methods without explicit preservation constraints increasingly fail as scenarios become more complex. Second, we expect these failures to extend to newer methods that share the same structural limitation. DynamixSFT, which the reviewer cites, adjusts mixture weights using loss-reduction rewards, but cannot explicitly enforce a specified capability-preservation threshold. Given the breadth of our existing comparisons, we believe evaluating another method with the same limitation would just add another empirical data point, rather than change any conclusion.

W2: "Direct comparisons with other adaptive methods are largely confined to the 3B model" Q2: "Could the authors compare against recent dynamic-mixture baselines on the 8B and 12B models [...]"

Our evaluation already includes cross-model comparisons: gradient alignment, our most directly comparable estimator-only baseline, is evaluated on all three model families and consistently underperforms DynaMiCS (Fig. 17, l. 1939–1949). The extensive fixed-weight grid is also evaluated across all models (Fig. 2).

Together, these experiments already provide evidence across model scales. As discussed in W1, the remaining adaptive methods lack explicit preservation constraints, a structural limitation that does not depend on model scale. Given the breadth of our evaluation, we believe extending these comparisons would not change the conclusions of our work.

W3: "All main fine-tuning runs stop at 2,048 steps, [...]." Q2 (cont.): "[...] run substantially longer fine-tuning experiments"

2,048 steps already involves substantial repetition. A 300-example dataset sampled at weight 0.5 is seen about 27 times, making forgetting and overfitting relevant within this budget.

DynaMiCS's advantage persists throughout training. We examined every evaluation checkpoint. DynaMiCS satisfies all constraints 3.7–5.2× more often than fixed weights during steps 64–1,024, and 5.0–6.7× more often during steps 1,088–2,048. Target improvements also persist. Overall, results show no erosion of DynaMiCS's toward the end of the horizon we evaluated.

Finally, our protocol selects the best feasible checkpoint rather than the final one (l. 1166–1168), so extending training cannot invalidate previously found feasible solutions.

AuthorsOpenReview ↗

Rebuttal to reviewer SZ3K

W4: "The contribution appears somewhat incremental relative to recent adaptive multi-domain SFT methods such as IDEAL and VersaTune" Q1: "what is the core methodological novelty of DynaMiCS?"

The core novelty of DynaMiCS is making capability preservation an explicit optimization requirement rather than an indirect objective. Existing adaptive mixture methods optimize training efficiency, aggregate performance, or proxies for forgetting, but do not let practitioners specify which capabilities must be preserved and how much degradation is acceptable. DynaMiCS instead formulates mixture selection as a constrained optimization problem, using local cross-domain measurements to improve target performance while respecting user-defined thresholds expressed directly in loss or benchmark accuracy (l. 93–100, 159–172). We will further clarify this positioning and add IDEAL to Section 2 and Appendix A.3.

W4 (cont.): "the added benefit of explicit preservation constraints is not clearly isolated."

Appendix N isolates it. Table 10 runs DynaMiCS with and without constraints, with the same estimate, schedule and solver (l. 2108–2124). Without constraints, target-perplexity reduction rises from 9.5% to 16.7%, but only 11/50 scenarios preserve every protected capability. With constraints, 40/50 do. The constraints are the only difference.

Thanks for the clarification. I appreciate the explanation of the explicit constraints, but I still have concerns about the lack of comparisons with recent methods and the limited training horizon. I would have liked to see more experimental evidence on these points, so I'll keep my original score.

Reviewer_dpj4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

DynaMiCS chooses data mixtures for multi-domain fine-tuning while explicitly constraining the preservation of existing capabilities. At each scheduled update, it runs short single-source probes from the current model and optimizer state, estimates a signed matrix of local cross-domain effects, and combines a linear predictor with a squared-hinge penalty to select the next mixture. This allows designated target domains to improve while requiring each protected domain to remain at its base-model reference. Figures 1 and 3 illustrate that evaluation domains need not supply training data and that targets can be measured by accuracy. The experiments cover 50 scenarios and three model families, with three seeds for the main configurations. Figures 2 and 4 primarily compare DynaMiCS with fixed-weight sweeps and examine how performance changes with model scale and the number of constraints.

Critical strengths
  1. Protecting specific capabilities is a useful goal that can be lost when preservation is reduced to an aggregate objective. DynaMiCS instead gives each protected capability its own constraint and base-model reference, as illustrated in Figure 1 and developed in Sections 3.1 through 3.3. I also find it valuable that the finite-difference construction does not require target and constrained evaluations to match the available training sources. Figure 3 demonstrates this flexibility with an unseen benchmark measured by accuracy.

  2. The evaluation is substantial, and accounting for measurement costs makes the gains more informative. Section 4 covers 50 scenarios with 1–3 targets and 3–10 constraints on Qwen2.5-3B, Qwen3-8B, and Gemma3-12B, using three seeds for the main configurations. Including probing and reduced evaluations, Section 4.5 puts the total cost at about 1.4–1.7 times that of one fixed run. Against this cost, the dense-warmup results in Figure 2 are encouraging. Mean test perplexity reduction reaches 11.3% on Qwen3-8B, compared with 3.5% for two fixed attempts, and 22.3% on Gemma3-12B, compared with 18.3% for the complete fixed grid.

  3. The gains appear to depend on revising the mixture during training, not just on finding a good initial constrained solution. In the frozen-mixture ablation over all 50 scenarios in Section 5, feasibility is identical, but dynamic updates raise mean test perplexity reduction from 2.2 ± 3.6% to 10.1 ± 9.7%. Figure 4 also tracks performance as the number of constraints increases from 3 to 10. Together, these analyses help explain the value of repeated adaptation and show how the method behaves as preservation becomes more demanding.

Critical weaknesses
  1. It is still difficult to judge whether DynaMiCS improves on the closest adaptive mixture methods, rather than on fixed weights. Section 2 identifies Skill-It and Aioli as especially close, and Section 4.4 lists five additional mixture methods, yet Figure 2 compares only against the fixed-weight grid. The paper's primary comparison does not show the closest adaptive estimators using the same constrained objective, update schedule, measurement budget, seeds, and selection rule. The results in Figures 1 through 4 are therefore less conclusive about superiority over adaptive mixture optimization.

  2. I am not sure whether the final preservation check is independent of the evaluations that guide training and checkpoint selection. Section 4.2 evaluates models every 64 steps on 200 batches and selects the best feasible checkpoint after training, while Section 4.5 uses reduced evaluations of 50 batches per domain for the slope probes. Section 4.6 explicitly measures target perplexity on a held-out test set, but does not make the same distinction for the final constraint verdict. It remains unclear whether that verdict is recomputed on examples never queried during probing, mixture selection, or checkpoint selection. Because feasibility is judged against the exact base-model reference, finite-sample noise could affect the outcome after repeated evaluations and post-hoc checkpoint selection.

  3. The aggregate improvement scores make it hard to tell whether DynaMiCS finds better feasible solutions, finds them more often, or does both. Figure 2 assigns 0% improvement to each infeasible run, as stated in its caption and the accompanying text. Its means and boxplots consequently combine the quality of feasible checkpoints with the frequency of finding one. The main figures do not separately show feasibility rates or target improvement conditional on feasibility. That distinction is especially important for interpreting Figure 4 as the number of constraints increases.

Questions for authors

Could you compare DynaMiCS with Skill-It, Aioli, or the closest linear-law estimator using the same constrained objective, update schedule, measurement budget, seeds, and checkpoint-selection rule? How much of the gain remains when only the estimator or probing strategy changes?

2 comments
AuthorsOpenReview ↗

Rebuttal to reviewer dpj4

We thank reviewer dpj4 for their thoughtful questions and comments, and for their support for the acceptance of our paper.

W1: "Figure 2 compares only against the fixed-weight grid. The paper's primary comparison does not show the closest adaptive estimators using the same constrained objective, update schedule, measurement budget, seeds, and selection rule." Q: "How much of the gain remains when only the estimator or probing strategy changes?"

The requested comparisons are already included in the paper. We agree that they deserve greater visibility and will move Table 9 to the main text, leveraging the additional page now available to authors. Section 5 explicitly references them (l. 415–417), and Appendix M provides a detailed evaluation of five alternative mixture-selection methods, summarized in Table 9 (l. 2084–2105). While Figure 2 focuses on fixed-weight baselines, the broader comparison is therefore already part of our empirical evaluation.

For a fair comparison, we evaluated all methods on the same 50 scenarios, matching training budgets, update schedules, evaluation protocols, and checkpoint-selection rules wherever applicable, while preserving each method's defining characteristics (Appendix M).

The comparison covers two complementary groups:

  • Alternative estimators within our constrained optimizer: gradient alignment and embedding similarity replace the estimation of cross-domain effects while retaining our optimization objective (l. 2085–2087). Both substantially underperform DynaMiCS, achieving 3.3% and 2.3% mean PPL reduction, respectively, versus 9.5% for DynaMiCS. Gradient alignment is further evaluated across all three model families (Fig. 17).
  • Established mixture optimization methods: Skill-It, Aioli, and DoReMi are evaluate on the same scenarios, matching training budgets and evaluation protocols of DynaMiCS while preserving each method's defining characteristics.. Skill-It is additionally granted oracle selection over 24 configurations per scenario, favoring the baseline (l. 2030–2049). Yet these methods achieve only 2.6–3.5% mean PPL reduction, compared with 9.5% for DynaMiCS. Unlike DynaMiCS, they do not explicitly model preservation constraints (l. 2084–2088).

Overall, Table 9 shows that DynaMiCS achieves both substantially higher target improvement (9.5% versus at most 3.5%) and higher constraint satisfaction (40/50 versus at most 19/50 feasible scenarios).

W2: "It remains unclear whether that verdict is recomputed on examples never queried during probing, mixture selection, or checkpoint selection."

We follow the standard validation/test separation: select on validation, report on test. We select the best feasible checkpoint using validation data, then report its target improvement on the held-out test set (l. 362–365). The test data is not used for training, mixture selection, or checkpoint selection, avoiding an optimistic estimate from evaluating on the same data used to select the checkpoint.

We additionally verified that the constraint verdict does not change on test data. We recomputed constraint feasibility for the validation-selected checkpoints on the test splits. All feasible DynaMiCS checkpoints remain feasible on the test split (Qwen2.5-3B, r=32).

W3: "The aggregate improvement scores make it hard to tell whether DynaMiCS finds better feasible solutions, finds them more often, or does both."

Our primary metric deliberately reflects the improvement a practitioner obtains from a single run: if no checkpoint satisfies the constraints, the practitioner must retain the original model, hence the 0% improvement (Section 4.6, l. 362–367). As discussed in the main text (l. 405–407), this metric captures both solution quality and feasibility. Feasibility rates are already reported separately in Appendix F (Fig. 6). We agree that disentangling these factors is informative, and computed the following additional statistics from our existing runs (Qwen2.5-3B, r=32):

  • Higher feasibility: DynaMiCS achieves substantially higher run-level feasibility (~80% versus ~20% for individual fixed-weight runs).
  • Comparable solution quality: In the 26 scenarios where both approaches find feasible solutions, DynaMiCS achieves 12.0% target-PPL reduction, compared with 12.3% for the oracle best fixed-weight configuration selected retrospectively from the entire grid.
  • Broader feasibility: DynaMiCS finds feasible solutions in 14 scenarios where none of the tested fixed mixtures succeeds, achieving 10.9% PPL reduction.

Thus, DynaMiCS's main advantage is finding feasible solutions substantially more reliably, while matching the quality of the oracle best fixed-weight solutions when both succeed. We will include this decomposition in the revision.

Thank you to the authors for their response, which partially addressed my concerns. I will keep my score unchanged.

Reviewer_EdyeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies domain mixture weighting in fine-tuning. The setup is to improve given target domains without losing specified capabilities like general knowledge or safety. The authors frame mixture selection as a constrained problem: lower the target losses while keeping the losses on “constrained” domains below their starting values. Their method, DynaMiCS, periodically pauses training and briefly fine-tunes on each dataset alone to measure how it moves every target and constraint loss.

The method is tested on 50 scenarios with Qwen2.5-3B, Qwen3-8B and Gemma3-12B, mostly with LoRA. The authors report larger target improvements and more cases meeting the constraints than fixed-weight baselines and five other mixing methods.

Critical strengths

Strength 1: Comprehensive of experiments The author has done a lot of experiments, a full suite across 50 different scenarios and models of different sizes. There are also different LoRA ranks tested.

Strength 2: reasonable setup and problem setting The setting of improving certain capabilities with a few constraints is a reasonable practical scenario. This is a natural constrained optimization setup and the authors have turned the (local) computation into solvable convex problems by measuring the slopes.

Critical weaknesses

Weakness 1: Limited novelty, and the method seems to reduce to a simple heuristic.

Many pieces of of the method are from prior methods, as the authors also acknowledge. The novel part is adding a penalty for constraints. However, to make optimization possible, a linear model is chosen, and the solutions end up using only one or two datasets at a time. Looking at the weight plots, the method mostly trains on the target data first and then shifts to FineWeb or Wikipedia once a constraint gets close. This may be an hidden bias of the method. That is something a practitioner could do with a simple rule, I find it hard to tell how much the probing and optimization actually contribute.

Weakness 2: The method seems to come with many knobs, which can potentially make it harder to use. There seems to be a lot of hyper parameter to be chosen, such as the update schedule, number of samples each dataset is probed, lambda, epsilon. The paper’s own results show that two of these matter a lot. When the mixture is updated only every 512 steps, DynaMiCS does worse than the fixed-weight baselines. I am worried that some of these knobs are sensitive to datasets. Further, it seems that there is no clear guidelines on how a practitioners can set for their own scenarios. In some sense, we did not choose the weights directly, but we need to work with another set of knobs.

Questions for authors

Q1: How would a simple heuristic rule work?

In the examples shown, DynaMiCS mostly trains on the target data first and then moves weight to a generic corpus like FineWeb. Would this simple approach become a strong baseline and I wonder how it performs. We can measure the constraints across the runs and switch to the generic dataset when the constraints are close to break.

Q2: How much do the constraints will break with other baselines?

This will help us quantify how much the method would help "protecting" the constraints.

4 comments
AuthorsOpenReview ↗

Rebuttal to reviewer Edye

We thank reviewer Edye for recognizing the relevance of our constrained fine-tuning formulation and the extensive experimental evaluation.

W1: "the method mostly trains on the target data first and then shifts to FineWeb or Wikipedia once a constraint gets close [...] That is something a practitioner could do with a simple rule" Q1: "How would a simple heuristic rule work?"

The target-then-generic data is not a rule. We observe this pattern qualitatively (l. 457–460). It holds in 28% of runs, which start on a target and end on FineWeb or Wikipedia. However, in 37% of runs, the final mixture is led by a domain-specific dataset. We will clarify this in the caption of Figure 20. Nothing in Eq. 4 favors generic data: weight moves to whichever dataset the probes show helps the constraints.

In general, DynaMiCS draws data from several sources. A linear objective favors few datasets, and the squared-hinge penalty mixes them when a constraint is active (l. 208–215). In 36% of updates, two or more datasets each receive more than 5% weight. The weights also change from one update to the next. In 98% of runs, at least three datasets receive more than 5% weight at some point, and most runs use four or five.

A heuristic needs knowledge that is often missing. The rule the reviewer proposes must know which corpus protects the constraints, and the numbers above show it is not necessarily a generic one (e.g., scenario 20 has no generic corpus at all). Most importantly, DynaMiCS can steer fine-tuning toward a target that has no training data (l. 258–265, l. 419–433). The proposed rule cannot be applied in this setting.

The heuristic struggles when generic data is not the right choice. We have tested the heuristic on 18 scenarios that include both target training data and a generic corpus (Qwen2.5-3B, r=32). Every 64 steps, we switch from target data to FineWeb (or Wikipedia) when a constraint comes within 0.01 nats of its reference, and back once all constraints recover. In 8 scenarios, DynaMiCS favors domain-specific rather than generic data, succeeding in 6 scenarios compared with 4 for the heuristic. In S25 and S44, the heuristic violates constraints at every checkpoint, while DynaMiCS remains feasible and achieves 12.7% and 10.7% target-PPL reduction. In the other 10 scenarios, DynaMiCS mostly ends on FineWeb, as in Figure 20. Here the rule performs comparably, because it is given in advance the corpus that DynaMiCS finds with its probes. A fixed rule works only when its fixed choice happens to be right. The pattern in the weight plots cannot replace measuring the effects. We will add this experiment to the revision.

W2: "There seems to be a lot of hyper parameter to be chosen [...] When the mixture is updated only every 512 steps, DynaMiCS does worse than the fixed-weight baselines."

DynaMiCS does not require scenario-specific hyperparameter tuning: the settings mentioned are either selected automatically or fixed across experiments.

  • λ and ε are selected automatically by the algorithm. At each update, DynaMiCS evaluates (λ, ε) pairs on a fixed grid and selects the feasible solution with the lowest target objective (l. 216–220, 1175–1182). No manual selection is required.
  • The schedule and probe length follow one fixed recipe. We study it once and apply it unchanged across setups and models, accuracy-target experiments included (l. 429–433).
  • The schedule reflects one main finding: update early. Section 5 compares schedules and shows that early updates are important (l. 441–445). This is the schedule we use in all main results. Appendix G is a sensitivity analysis that removes early updates and uses fixed intervals instead (l. 1354–1361). At H = 512, the first update comes only at step 512, and the gains fall below the baselines. This confirms the finding. The guidance for a practitioner is simply to front-load updates.

In practice, DynaMiCS uses a single set of defaults across scenarios, whereas selecting competitive fixed mixtures typically requires exploring multiple weight configurations. We will make the recommended default settings more explicit in Section 5.

Q2: "How much do the constraints break with other baselines?"

Every baseline breaks the constraints more often than DynaMiCS. Table 9 counts the scenarios where no checkpoint satisfies all constraints. Out of 50:

  • the adaptive baselines break them in 31 to 41 scenarios,
  • the best fixed weights, chosen per scenario after the fact, in 21,
  • DynaMiCS in 10.

The gap grows with the number of constraints. As constraints are added, the baselines degrade steadily while DynaMiCS stays largely stable (Figure 4, l. 447–453).

When both fail, the baselines fail by more. In scenarios where no method is feasible, DynaMiCS has a lower worst-case violation than the best fixed weights in every model configuration (Figure 14, l. 1676–1686).

Thanks for the rebuttal.

W1 and Q1 Thanks for the new experiment. I think that these add more understanding to the method. It seems that if the general dataset will help keep the constraint, then it will be chosen. There are cases that another dataset can fit the constraints, where the method will find them. It does seem that the method tends to concentrate the weights on a few datasets. I believe this is due to a tendency of the linear model, at each update:

  1. When constraints are not active (i.e., no penalty is active), then the method will focus on the dataset that can optimize the loss.
  2. When constraints are active, then the method will land someone in between the constraint dataset(s) and target dataset.

This weakness is partly addressed and clarified. DynaMics seems to work better in more complex cases, such as multiple datasets are needed to protect the constraints. These are worth stating in the paper.

W2 Yes λ\lambda and ϵ\epsilon are found automatically. What's more important though, is that the schedule and probe length affect the final results. It is still unclear how to set them in a practical setup. For example, setting the probes too short may cause constraint violation or missing target loss.

Q2 Thanks for clarifying that other baselines break the constraints more often. However, my question is "how much". Since many other baselines are not aware of the constraints, it is a good test to see how much the constraints are violated. This would show how important the method actually matters. Could the authors report the size of the violations across the scenarios?

Overall, thanks for the clarifications. I still intend to keep my score, the advantage over simple rules is only shown in a few scenarios. And more importantly, the results remain sensitive to the schedule and probe length, which may make the method harder to apply in practice.

We thank the reviewer for engaging with the rebuttal.

W1/Q1: "This weakness is partly addressed and clarified. DynaMics seems to work better in more complex cases, such as multiple datasets are needed to protect the constraints. These are worth stating in the paper."

We thank the reviewer for recognizing that this point is addressed, and we will expand the paper accordingly.

  • Section 3.3, after l. 220: "Without an active constraint, the solution concentrates on the dataset that lowers the target loss fastest. Once a constraint becomes active, the penalty mixes in the datasets that protect it."
  • Appendix P, l. 2259–2267: we will add the heuristic experiment, and state that DynaMiCS matters most when several datasets, or non-generic ones, are needed to protect the constraints.

W2: "the schedule and probe length affect the final results. It is still unclear how to set them in a practical setup."

Every training setting affects the results; what matters is whether it must be retuned. The learning rate is the clearest example. Practitioners routinely tune it for each new model, and this is standard practice, not a weakness of fine-tuning. We did the same, with a six-candidate sweep per model (l. 279–281). The schedule and probe length need even less: we fixed them once (l. 286–300) and never retuned them, for any model. The same values run on all 50 scenarios, three models, three LoRA ranks, full fine-tuning and the accuracy targets (Section 5, Appendices H–I).

Q2: "my question is 'how much' [...] Could the authors report the size of the violations across the scenarios?"

The submission already reports how much constraints are violated, in the hardest case. Appendix K takes the scenarios where neither DynaMiCS nor any baseline is feasible, and plots each method's constraint violation (Figure 14, l. 1676–1686). DynaMiCS violates less than the best baseline in every model configuration, so it degrades gracefully even when no feasible solution exists. The main text states this result (l. 451–454).

As requested, we now report how much each method violates the constraints across all scenarios (Qwen2.5-3B, r=32).

MethodMedian violation90th pct.
Fixed weights2.7%5.5%
Gradient alignment1.9%5.0%
Embedding similarity1.0%3.6%
DoReMi3.3%5.4%
Aioli2.9%5.7%
DynaMiCS0.06%1.4%

Overall: "I still intend to keep my score, the advantage over simple rules is only shown in a few scenarios."

The simple rule only works if one already knows DynaMiCS's answer. The rule we tested at the reviewer's request trains on the targets and switches to FineWeb or Wikipedia whenever a constraint nears its reference. It was derived from the weight plots in Figure 20, that is, from what DynaMiCS chose in those scenarios. A practitioner does not have these plots before training, and usually does not know which dataset will protect which capability. Finding this out is exactly what DynaMiCS does. When FineWeb happens to be the right choice, the rule does about as well; when it is not, the rule fails. Comparing DynaMiCS with a rule built from its own results is therefore circular.

We believe we have now addressed every point the reviewer raised, with evidence from the paper or from new experiments. We would be grateful if the reviewer took these responses into account in the final assessment.

Thanks for the authors for the follow-up. The new results help make some clarification, answering Q2 and W2. The heuristic part of W1 is also clarified since the rule needs to be know advance, in complex cases the method can help find the datasets that protects the constraints.

In W1, my remaining concern is novelty. The method mainly combines existing methods for a new problem setting: probing is similar to skill-it, and the linear method is from Aioli. The main contribution here is the constrained formulation. In fact, DynaMiCS's benefit mainly comes from constraint preservation.

I appreciate the rebuttal and agree that the empirical results of combining these methods is good. However, the novelty is still limited by combining existing techniques with a constraint penalty.

Comments on the paper

1 comment

Dear Area Chair,

Overall, the reviews raised questions about the experimental evidence, the contribution of explicit preservation constraints, and the practical applicability of DynaMiCS. We believe we have addressed these concerns with concrete evidence, both from the submission and from additional analyses conducted during rebuttal:

  • Adaptive baselines (dpj4 W1, SZ3K W1–W2): We clarified that Table 9 already compares five alternative mixture-selection methods, including controlled estimator ablations. We will move these results to the main text.
  • Practical applicability (Edye W1–W2): We implemented the reviewer's proposed heuristic and explained when and why it fails. We also clarified that DynaMiCS requires no scenario-specific hyperparameter tuning.
  • Constraint preservation (Edye Q2, dpj4 W2–W3): We provided additional constraint-violation statistics and verified feasibility on test splits.
  • Training stability (SZ3K W3): We provided checkpoint-level analyses showing that DynaMiCS's advantage persists throughout training. Moreover, since our protocol selects the best feasible checkpoint rather than the final one, extending training cannot invalidate previously found solutions. We therefore believe the proposed longer runs would offer no additional insight into the practical effectiveness of our method.

We respectfully disagree that the paper lacks sufficient experimental evidence. Our evaluation already spans over 8,500 runs, 50 scenarios, multiple model families, and five alternative mixture-selection methods. We believe baseline selection should be driven by the methodological questions being tested, not simply by the availability of newer methods (not even tailored for the same problem). Adding another baseline that cannot express preservation constraints would not address any missing aspect of our contribution. Given the extensive evidence already provided, we do not believe the additional computational cost of these comparisons would be justified.

We hope the final assessment will take into account both the breadth of the evaluation and the additional evidence provided in our responses to all reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.