- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to improve the target-task performance of instruction-tuned language models while preserving their existing capabilities. The authors propose a drift-constrained perspective on fine-tuning, where behavioral changes from the reference model are measured using KL divergence. Under a local Fisher approximation, they formulate fine-tuning as a direction-selection problem, arguing that the effectiveness of an update depends not only on its magnitude but also on its direction. To investigate this, the authors introduce Layer-Selective Tuning (LST), which restricts fine-tuning to selected transformer layers. Experiments on scientific reasoning and multilingual translation show that appropriate layer configurations can substantially improve target-task performance under QA-only supervision while preserving general capabilities.
Critical strengths
- The paper provides a useful geometric interpretation of fine-tuning, emphasizing the importance of update directions under constrained behavioral drift.
- The experiments demonstrate substantial improvements over conventional fine-tuning methods while largely preserving existing model capabilities. And the matched-drift comparisons, layer-configuration studies, and Pareto frontier analysis provide convincing evidence that update structure matters. The results on multilingual translation and subsequent RL training demonstrate the potential usefulness of the proposed approach beyond the initial fine-tuning setting.
- The paper evaluates fine-tuning under QA-only supervision, where models receive no reasoning trajectories during training but are still expected to perform multi-step reasoning at inference time. This provides a convincing stress test of the proposed approach and makes the empirical findings more compelling.
Critical weaknesses
-
The paper formulates fine-tuning as a constrained optimization problem in Equation 3. But the actual experiments seem to train different layer configurations and measure their resulting KL divergence, without explicitly enforcing the same drift constraint during optimization. Although the authors provide comparisons between configurations with similar KL values, these are primarily post-hoc comparisons and do not necessarily show how the proposed approach behaves under a fixed drift budget. I think it would be useful to evaluate different fine-tuning methods under explicitly controlled drift budgets, for example through KL-based early stopping or adaptive constraint enforcement. This would provide a more direct validation of the proposed drift-constrained formulation and help distinguish the effects of update direction from differences in the amount of behavioral drift.
-
The proposed approach relies on probing different layer subsets to identify effective update directions. In Section 4.3, the authors explicitly use a coarse search over structured layer configurations rather than directly optimizing the proposed directional-efficiency objective. I think in this way, searching for effective configuration for a new model or downstream task without performing multiple fine-tuning runs would be very challenging, makeing this method hard to adapt. The transfer results on InternS1-mini-8B and the consistent performance of neighboring configurations are good, but they do not fully establish whether the same layer-selection strategy can be applied reliably across different tasks and architectures. It would be useful to provide a more systematic selection criterion, or at least quantify the computational cost of searching for effective configurations.
Questions for authors
See the weaknesses.
We thank the reviewer for the careful reading and for recognizing the value of the geometric perspective and the QA-only experiments. The two concerns highlight an important distinction between our drift-constrained analysis and its current layer-selective instantiation.
1. Explicit control of behavioral drift
Thank you for this suggestion. Explicit control of behavioral drift in this geometry remains an open question. Adding a KL penalty to the loss does not directly provide such geometric control, and our experiments in Section 5.2 illustrate the limitations of relying on this penalty alone.
We therefore use layer selection as a controlled test-bed to study the properties of update directions. It allows us to keep the data, objective, and optimization setup fixed while varying the accessible direction families. This includes the matched-drift comparisons in Figure 3(a), which examine the effect of direction at closely matched KL values. Figure 3(b) additionally controls the number of trainable layers.
These experiments test the predictions of our geometric perspective through controlled comparisons. We will clarify the distinction between these comparisons and explicit drift control in the geometry, and describe the drift-matching procedure more clearly.
2. Layer selection and the computational cost of exploration
Thank you for raising this concern. Our exploration aims to identify structural patterns that transfer across tasks. The analysis on MegaScience covers all layer-selection configurations in the low-drift region of our experimental grid. It reveals a stable split structure, with several configurations, including b4t8, b4t16, and b4t20, working well. This pattern also holds in the translation and scientific-task experiments.
The transferable finding is the split structure itself, so applying it does not require a new search for the optimal configuration on each downstream task. The success of multiple configurations also shows that the finding does not depend on one carefully selected layer subset. We will clarify this point and report the computational cost of the exploration.