acceptodds
Under review as a conference paper at ICLR 2027

Drift-Constrained Optimization: Only Direction Matters in Fine-Tuning Instruct Models

Abstract

Fine-tuning instruct models often improves target performance while inducing behavioral drift that can degrade existing capabilities. Rather than treating drift as an uncontrolled consequence, we impose a drift budget and ask how to maximize target performance within it. Locally, behavioral drift defines a geometry around the reference model, where the budget constrains distance, leaving direction as the remaining degree of freedom. Directional efficiency thus reformulates fine-tuning as a direction-selection problem, predicting that changes in accessible directions can qualitatively alter outcomes. We test this prediction in a stringent QA-only setting, where strong instruct models are trained only on final answers yet must generate multi-step reasoning at inference. A coarse layer-selective probe reverses QA-only fine-tuning failure, with neighboring configurations improving target performance while preserving reasoning and general capabilities. Across Qwen3 models, these directions substantially improve scientific reasoning and multilingual translation. Over 100+ languages, the resulting models match or outperform dedicated translation systems and provide a stronger initialization for reinforcement learning. Our results suggest that fine-tuning is not just about how much a model changes, but how that change is spent.

Primary area:
optimization
Keywords:
Behavioral Drift, Fine-Tuning, Drift-Constrained Optimization, Update Direction, Fisher Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_T9soOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to improve the target-task performance of instruction-tuned language models while preserving their existing capabilities. The authors propose a drift-constrained perspective on fine-tuning, where behavioral changes from the reference model are measured using KL divergence. Under a local Fisher approximation, they formulate fine-tuning as a direction-selection problem, arguing that the effectiveness of an update depends not only on its magnitude but also on its direction. To investigate this, the authors introduce Layer-Selective Tuning (LST), which restricts fine-tuning to selected transformer layers. Experiments on scientific reasoning and multilingual translation show that appropriate layer configurations can substantially improve target-task performance under QA-only supervision while preserving general capabilities.

Critical strengths
  1. The paper provides a useful geometric interpretation of fine-tuning, emphasizing the importance of update directions under constrained behavioral drift.
  2. The experiments demonstrate substantial improvements over conventional fine-tuning methods while largely preserving existing model capabilities. And the matched-drift comparisons, layer-configuration studies, and Pareto frontier analysis provide convincing evidence that update structure matters. The results on multilingual translation and subsequent RL training demonstrate the potential usefulness of the proposed approach beyond the initial fine-tuning setting.
  3. The paper evaluates fine-tuning under QA-only supervision, where models receive no reasoning trajectories during training but are still expected to perform multi-step reasoning at inference time. This provides a convincing stress test of the proposed approach and makes the empirical findings more compelling.
Critical weaknesses
  1. The paper formulates fine-tuning as a constrained optimization problem in Equation 3. But the actual experiments seem to train different layer configurations and measure their resulting KL divergence, without explicitly enforcing the same drift constraint during optimization. Although the authors provide comparisons between configurations with similar KL values, these are primarily post-hoc comparisons and do not necessarily show how the proposed approach behaves under a fixed drift budget. I think it would be useful to evaluate different fine-tuning methods under explicitly controlled drift budgets, for example through KL-based early stopping or adaptive constraint enforcement. This would provide a more direct validation of the proposed drift-constrained formulation and help distinguish the effects of update direction from differences in the amount of behavioral drift.

  2. The proposed approach relies on probing different layer subsets to identify effective update directions. In Section 4.3, the authors explicitly use a coarse search over structured layer configurations rather than directly optimizing the proposed directional-efficiency objective. I think in this way, searching for effective configuration for a new model or downstream task without performing multiple fine-tuning runs would be very challenging, makeing this method hard to adapt. The transfer results on InternS1-mini-8B and the consistent performance of neighboring configurations are good, but they do not fully establish whether the same layer-selection strategy can be applied reliably across different tasks and architectures. It would be useful to provide a more systematic selection criterion, or at least quantify the computational cost of searching for effective configurations.

Questions for authors

See the weaknesses.

1 comment

We thank the reviewer for the careful reading and for recognizing the value of the geometric perspective and the QA-only experiments. The two concerns highlight an important distinction between our drift-constrained analysis and its current layer-selective instantiation.

1. Explicit control of behavioral drift

Thank you for this suggestion. Explicit control of behavioral drift in this geometry remains an open question. Adding a KL penalty to the loss does not directly provide such geometric control, and our experiments in Section 5.2 illustrate the limitations of relying on this penalty alone.

We therefore use layer selection as a controlled test-bed to study the properties of update directions. It allows us to keep the data, objective, and optimization setup fixed while varying the accessible direction families. This includes the matched-drift comparisons in Figure 3(a), which examine the effect of direction at closely matched KL values. Figure 3(b) additionally controls the number of trainable layers.

These experiments test the predictions of our geometric perspective through controlled comparisons. We will clarify the distinction between these comparisons and explicit drift control in the geometry, and describe the drift-matching procedure more clearly.

2. Layer selection and the computational cost of exploration

Thank you for raising this concern. Our exploration aims to identify structural patterns that transfer across tasks. The analysis on MegaScience covers all layer-selection configurations in the low-drift region of our experimental grid. It reveals a stable split structure, with several configurations, including b4t8, b4t16, and b4t20, working well. This pattern also holds in the translation and scientific-task experiments.

The transferable finding is the split structure itself, so applying it does not require a new search for the optimal configuration on each downstream task. The success of multiple configurations also shows that the finding does not depend on one carefully selected layer subset. We will clarify this point and report the computational cost of the exploration.

Reviewer_dY7iOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies supervised fine-tuning (SFT) from the perspective of behavioral drift, defined as the KL divergence between the output distributions of a fine-tuned model and its reference model. Using a local Fisher approximation, the authors reformulate fine-tuning under a fixed drift budget as a direction-selection problem and introduce directional efficiency to characterize task improvement relative to behavioral drift. Based on this formulation, the paper proposes Layer-Selective Tuning (LST) as a structured probe of update directions. LST freezes selected Transformer blocks and fine-tunes the remaining layers, exploring different combinations of bottom and top layers without explicitly optimizing the theoretical directional-efficiency objective. The experiments compare full fine-tuning, LoRA, KL-regularized fine-tuning (ASFT), and LST on Qwen3-8B/14B under QA-only supervision, covering scientific-domain tasks and multilingual translation. The authors report that certain layer configurations improve target-task performance while better preserving general capabilities, even at comparable levels of KL drift. Additional experiments examine the effects of layer configurations, embedding freezing, cross-model transfer, and the use of LST models as initialization for subsequent reinforcement learning.

Critical strengths

Comprehensive Empirical Evaluation: The paper presents an extensive empirical evaluation across multiple model scales, fine-tuning strategies, and downstream tasks, including scientific-domain adaptation and multilingual translation. Additional experiments investigate different layer configurations, matched-KL comparisons, cross-model transfer, and reinforcement-learning initialization, providing a broad empirical characterization of layer-selective fine-tuning behavior.

Critical weaknesses

1. Limited Theoretical Novelty: Re-derivation of Classical Natural Gradient: The theoretical formulation in Sections 3 and 4.1 largely reproduces the classical natural gradient derivation. Specifically, the authors approximate KL divergence using the Fisher information matrix and formulate a linearized task objective under a quadratic drift constraint, obtaining the optimal update direction −F−1g-F^{-1}g. This is a well-established result in information geometry and trust-region optimization. The proposed "directional efficiency" essentially reformulates the same optimization problem without introducing a new theoretical result.

More importantly, the analysis does not provide new insights into why particular Transformer layers should be selected or why layer-selective fine-tuning should outperform conventional methods in practice.

2. Fundamental Disconnect Between Theoretical Formulation and Practical Method: The theoretical framework defines directional efficiency as a function of the actual parameter update direction, the task gradient, and the Fisher information matrix. However, none of these quantities are used to guide the proposed Layer-Selective Tuning (LST) method. Instead, the authors simply train different subsets of Transformer layers and compare their downstream performance. There is no procedure for estimating the theoretical directional efficiency of candidate layer configurations, selecting layers based on the Fisher geometry, or optimizing toward the derived natural gradient direction.

Furthermore, since the feasible update space of full fine-tuning contains that of LST, the theoretical formulation cannot, by itself, explain why LST should outperform full fine-tuning in practice. Such an explanation would require additional analysis of optimization dynamics or implicit regularization, which is largely absent.

Consequently, the theoretical framework appears disconnected from the proposed method and provides neither predictive guidance nor an actionable optimization strategy. More fundamentally, the proposed theoretical framework does not explain the central empirical advantage of LST over full fine-tuning. According to Eq. (11), the feasible direction set of LST is a strict subset of that of FFT. Consequently, the maximum attainable theoretical directional efficiency of FFT must be at least as high as that of LST. While this does not imply that practical FFT training must outperform LST, it means that the observed superiority of LST cannot be explained by the proposed feasible-direction formulation alone. Such an advantage would require an analysis of optimization dynamics, implicit regularization, or why restricting trainable parameters leads the optimizer toward more effective updates. The paper provides no such connection.

Therefore, the theoretical framework neither predicts nor adequately explains the main empirical finding used to motivate it.

3. No Explicit Enforcement of the Claimed Drift Budget:

The paper explicitly claims to "impose a drift budget and ask how to maximize target performance within it." However, the actual training procedure does not appear to enforce any predefined drift budget.

The authors do not specify a target constraint δ\delta, introduce an optimization procedure that enforces Dref(θ)≤δD_{\mathrm{ref}}(\theta)\leq\delta, or adapt training based on the remaining drift budget. Instead, they perform conventional fine-tuning with different layer configurations and measure the resulting KL divergence afterward.

Comparing independently trained models with similar post-hoc KL values is fundamentally different from solving a drift-constrained optimization problem. Therefore, the experiments do not directly validate the central optimization formulation claimed in the introduction.

4. Lack of Methodological Novelty in Layer-Selective Tuning (LST):

The proposed LST method is essentially conventional parameter freezing. The authors select certain Transformer blocks for training while freezing the remaining blocks, using simple contiguous or split-layer configurations.

Neither the parameter-freezing mechanism nor the idea of selecting trainable Transformer layers is novel. The proposed method does not introduce a new optimization algorithm, a theoretically motivated layer-selection criterion, or an efficient procedure for identifying optimal layer subsets.

The main empirical observation—that different trainable layer configurations produce different fine-tuning outcomes—is already well understood. The paper does not establish a sufficiently general or mechanistic principle that would elevate these observations beyond a conventional layer-freezing ablation study.

5. Inconsistency Between Theoretical and Empirical Definitions of Directional Efficiency:

The theoretical directional efficiency in Eq. (9) is defined as η(u)=−g⊤uu⊤Fu,\eta(u)=\frac{-g^\top u}{\sqrt{u^\top Fu}}, which characterizes the local decrease in the training objective per unit Fisher radius.

However, the experiments replace this quantity with ηempirical=ΔTaskKL,\eta_{\mathrm{empirical}}=\frac{\Delta\mathrm{Task}}{\sqrt{\mathrm{KL}}}, where ΔTask\Delta\mathrm{Task} represents the change in downstream benchmark performance rather than the decrease in the training objective. Although these expressions are related under specific local approximations when the numerator measures the same objective, they are not generally equivalent.

Furthermore, Figure 3(b) introduces yet another efficiency measure, Accuracy/KL\mathrm{Accuracy}/\mathrm{KL}, which differs from both previous definitions.

The paper provides no direct validation that these empirical metrics correlate with the theoretically defined directional efficiency. At matched KL, differences in the empirical efficiency metric largely reflect differences in benchmark scores by construction. Therefore, the empirical comparisons do not constitute direct evidence for the proposed Fisher-geometric interpretation.

6. Questionable Full Fine-Tuning (FFT) Baselines and Insufficient Reproducibility: The full fine-tuning baselines exhibit extremely severe capability degradation. For example, in the multilingual translation experiments, Qwen3-14B's General Avg. drops from 48.95 to 1.36 after FFT, while its translation performance also deteriorates. This raises serious concerns about whether the FFT baseline was properly configured and optimized.

Surprisingly, despite the reproducibility statement claiming that Appendix E provides the training hyperparameters, the appendix does not report essential details such as learning rates, batch sizes, training epochs or steps, learning rate schedules, and checkpoint selection strategies. The paper also does not establish whether FFT received an adequate independent hyperparameter search or whether early stopping and intermediate checkpoints were considered.

These omissions make it impossible to determine whether the dramatic performance differences arise from the proposed layer-selection strategy or from inadequately configured baseline training.

Since comparisons against FFT constitute a central part of the experimental evidence, the absence of a properly documented and sufficiently tuned FFT baseline substantially undermines the credibility of the reported advantages.

Questions for authors

1. What is the actual drift-constrained optimization procedure?: The paper claims to impose a drift budget and maximize task performance within it. However, the experiments appear to perform conventional fine-tuning followed by post-hoc KL evaluation. Was a predefined drift budget δ\delta actually enforced during training? If so, please specify the constraint enforcement mechanism and the values of δ\delta used in the experiments. If not, how does the proposed approach differ operationally from unconstrained SFT followed by post-hoc drift analysis?

2. How does the theoretical framework explain the empirical superiority of LST over FFT?: According to Eq. (11), the feasible direction set of LST is a subset of that of FFT. Therefore, the theoretically optimal directional efficiency achievable by FFT must be at least as high as that of LST. What additional mechanism explains why LST consistently outperforms FFT in the reported experiments? Can the authors establish a connection between the Fisher-geometric formulation and the actual optimization dynamics that lead to this result? More generally, what new theoretical predictions does the proposed framework provide beyond the classical natural-gradient formulation?

3. How does the proposed theory guide the selection of trainable layers?: The practical LST method appears to rely on conventional parameter freezing and empirical exploration of layer configurations, without explicitly computing or optimizing the theoretical directional efficiency. Can the authors explain how the Fisher information matrix, task gradient, or theoretical directional-efficiency objective informs the choice of trainable layers? Is there a principled criterion for selecting configurations such as b4t8, b4t12, and b4t16 without first training and evaluating them? If the layer configurations are selected entirely through empirical experimentation, what methodological contribution distinguishes LST from conventional layer-freezing and layer-subset search?

4. What is the theoretical justification for the empirical directional-efficiency metrics?: Eq. (9) defines directional efficiency using the local training-objective gradient and Fisher metric, whereas the experiments use ΔTask/KL\Delta\mathrm{Task}/\sqrt{\mathrm{KL}}. Figure 3(b) further introduces Accuracy/KL\mathrm{Accuracy}/\mathrm{KL}. Can the authors provide a formal justification for these different empirical definitions and clarify under what assumptions they approximate the theoretical quantity? Have the authors evaluated whether the theoretical directional efficiency correlates with these empirical metrics or predicts the relative performance of different layer configurations?

5. Were the full fine-tuning baselines adequately optimized?: The FFT baselines exhibit extremely severe capability degradation. In particular, Qwen3-14B's General Avg. decreases from 48.95 to 1.36 in Table 1, accompanied by a decline in translation performance. Could the authors provide the complete training configurations for FFT, LoRA, ASFT, and LST, including learning rates, batch sizes, training schedules, checkpoint selection criteria, and the handling of thinking mode and reasoning formats? Were the FFT baselines independently tuned with learning-rate sweeps, early stopping, and intermediate checkpoint evaluation? Can the authors provide matched-KL comparisons against properly tuned FFT checkpoints to rule out the possibility that LST's reported advantages primarily arise from inadequately configured FFT training?

1 comment

Overall response. Our contribution is a drift-first geometric view that places heterogeneous fine-tuning updates in a shared behavioral space. QA-only tuning is merely a stringent stress test: its failure is reversed by changing the accessible update directions while keeping the model, data, objective, and training procedure unchanged.

Weakness 1. The natural-gradient solution is classical; our contribution is the shared reference-anchored geometry that separates behavioral drift into radial magnitude and direction across heterogeneous fine-tuning methods.

Weakness 2. LST is a controlled probe of the theory’s prediction that different realized directions yield different outcomes at matched drift, not an algorithm for explicitly computing the natural gradient.

Weakness 3. The fixed drift budget defines the common geometric scale for comparing updates, instantiated empirically through matched anchored-KL comparisons rather than an online constraint-enforcement algorithm.

Weakness 4. We do not claim parameter freezing as novel; layer selection provides a simple architecture-aligned intervention for changing accessible direction families while controlling other factors.

Weakness 5. The empirical metric is an operational downstream analogue of local directional efficiency, while the central evidence comes directly from performance differences at matched drift and does not depend on this ratio.

Weakness 6. FFT’s degradation under QA-only supervision is the phenomenon being studied, and our conclusion rests on matched-drift comparisons across direction families rather than on a single final FFT checkpoint.

Question 1. We study optimization under a fixed behavioral budget as a geometric comparison framework; we do not propose a new online algorithm that enforces a hard KL threshold during training.

Question 2. FFT contains more attainable directions, but gradient training realizes only one trajectory; restricting its feasible family can therefore steer optimization toward a more drift-efficient realized direction.

Question 3. The theory motivates selecting among direction families rather than prescribing exact layers, and layer subsets serve as a tractable structured probe of these families.

Question 4. ΔTask/KL\Delta\mathrm{Task}/\sqrt{\mathrm{KL}} is used as an empirical counterpart—not an exact estimator—of the local theoretical quantity, with matched-KL performance providing the primary validation.

Question 5. All methods use the same QA-only data and evaluation protocol; we will provide the complete training configurations to make this comparison fully reproducible.

Reviewer_wetFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that fine-tuning should be studied under a fixed budget of drift, measured as the KL divergence from the original model. Under this budget, the best update follows the natural gradient, so the direction of the update matters, not its size. Full fine-tuning, LoRA and training only some layers are seen as different sets of allowed directions. The authors fine-tune Qwen3-8B/14B on question-answer pairs only, with no reasoning steps. Training only the bottom 4 layers and a block of top layers improves chemistry and translation while keeping most general capabiliteis. Full fine-tuning and LoRA often fail in the same setting.

Critical strengths
  1. A simple method with a useful result. With the middle layers frozen, question-answer data alone improves translation, while full fine-tuning and LoRA lose quality (Table 1). The model is also a better starting point for RL (Figure 5b).

  2. A clean comparison. Only the trained layers change; data and training stay the same. Training the bottom 16 layers and training the bottom 4 plus top 12 layers use the same number of layers but give very different results (Table 1).

Critical weaknesses
  1. The drift budget is not enforced, and the theory is not used. No run trains under a KL limit; the method only freezes layers. "Matched drift" means pairing runs whose KL happens to be close after training. The natural-gradient result is standard (Amari, 1998). The paper does not check whether its updates are closer to it, or explain why the bottom and top layers work.

  2. KL does not track forgetting. On SmolInstruct, SFT with a KL penalty (ASFT) has lower KL than full fine-tuning but loses much more general capability (Table 2). The paper also finds that consecutive layers can be better at the same drift (Figure 3a), so the title "only direction matters" is too strong.

  3. The translation comparison is not fair, and the setup is unclear. Two of the four main languages are Nepali and Cebuano, but the translation systems support only 22 to 33 languages (Appendix E). For translation from English or Chinese, Hunyuan-MT1.5-7B is still better (Table 3). The paper also does not say how the layers were chosen, gives no training settings, and reports one run.

Questions for authors
  1. Does the theory tell us which layers to train? If not, what does it add beyond trying layer choices? A check of how close each update is to the natural gradient, even on a small model, would help.
  2. Why does SFT with a KL penalty lose more general capability than full fine-tuning at lower KL? If KL does not predict forgetting, why is it the right thing to budget?
  3. How were the layer choices made: from how many options, and on which data? Was the data separate from the test sets?
  4. How do the models compare with the translation systems on only the languages each system supports?
  5. Results over several seeds on one setting (e.g., Qwen3-8B on translation), plus the learning rate, epochs and batch size, would show whether the gains are stable.
1 comment

We thank the reviewer for the thoughtful comments and for recognizing the value of our controlled QA-only experiments. Main contribution Our contribution is a drift-first geometric view that places heterogeneous fine-tuning updates in a shared behavioral space. QA-only tuning is merely a stringent stress test: its failure is reversed by changing the accessible update directions while keeping the model, data, objective, and training procedure unchanged.

QA-only tuning itself generally fails on strong instruct models: although it may improve the target task, it often damages reasoning and general capabilities because no reasoning trajectories are provided. Our main finding is that this failure can be reliably reversed by a minimal change—changing only which layers are trainable, while keeping the model, QA-only data, objective, and training procedure unchanged. This pattern holds across scientific reasoning and translation, Qwen3-8B and Qwen3-14B, multiple neighboring layer configurations, and transfers to InternS1-mini-8B. Our contribution is therefore not translation SOTA, but showing that QA-only tuning can succeed without sacrificing existing capabilities when the update structure is properly restricted.

1. What does the theory add? The theory provides direct guidance: under a fixed behavioral-drift budget, different fine-tuning methods are distinguished by their update directions, and an optimal direction exists. This turns fine-tuning into a direction-selection problem. Layer selection is not part of the theory; it is simply the experimental tool we use to change accessible directions and test this prediction.

2. Why can lower KL cause more forgetting? The KL term in the loss penalizes only the amount of behavioral change, without specifying which direction the model should move. For example, two models may remain equally close to the reference model, while one changes task-relevant behavior and the other changes general reasoning capabilities. The latter can exhibit greater forgetting even with lower KL. Thus, ASFT can reduce overall drift without ensuring that the limited drift is used in a beneficial direction.

3. How were the layers selected? As shown in the analysis section, we evaluated all layer combinations in the low-drift region on MegaScience and identified the Pareto-efficient configurations there. We then transferred the resulting configurations directly to multilingual translation and SmolInstruct chemistry, where they consistently reversed QA-only tuning failure. Thus, the reported configurations were not selected on the translation or chemistry test sets; their success across datasets demonstrates that the direction pattern is stable and transferable.

4. Is the translation comparison fair? Our goal is not to claim translation SOTA, and Hunyuan is not our baseline. We include expert translators only to show that our method reverses QA-only tuning failure, reaches competitive translation performance, and preserves the model’s original general capabilities. Hunyuan performing better in some directions does not affect this conclusion.

5. Is the finding stable? Yes. The central finding—reversing QA-only tuning failure through a minimal change in update structure—is observed across two tasks, two Qwen3 scales, multiple neighboring split-layer configurations, and another model family without retuning the configuration. We will make this broader stability clearer and additionally report multi-seed Qwen3-8B translation results with full training details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.