acceptodds
Under review as a conference paper at ICLR 2027

Fine-Tuning Beyond Weight-Space Geometry: One Frontier, Two Origins

Abstract

Post-training methods are commonly evaluated through the tradeoff between new-task performance and retention, yet the shape of this frontier does not reveal its cause. A similar loss of adaptation can be either necessary, because the old and new tasks demand incompatible predictions, or avoidable, because an old-data-free objective constrains the wrong computations. We expose this ambiguity using controlled causal sequence tasks with known target distributions and replay solutions. We study WindowLoRA, a family that moves from ordinary LoRA to dense teacher distillation by progressively increasing the causal and positional coverage of hidden-state matching. Across both conflicting and compatible tasks, increasing constraint density produces the same qualitative continuum: old-task error decreases while new-task error increases. The meaning of this continuum, however, changes relative to replay. Under shared-input target conflict, dense output distillation reaches the replay compromise, showing that the remaining tradeoff is intrinsic. Under compatible disjoint support, replay solves both tasks while new-input matching remains on a nonzero frontier, showing that the tradeoff is introduced by incomplete preservation supervision. We then test this interpretation in pretrained decoder-only models by comparing the representation drift and downstream retention–accuracy frontiers induced by different adaptation methods. Although stronger constraints reduce drift, none eliminates the frontier, and WindowLoRA provides controlled intermediate operating points. Finally we position the family to K-prior which explains how a new-input matching can replace the unavailable old objective only when its probes cover the old-sensitive directions of the adaptation carrier.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLMs, Continual learning, Finetuning, Low Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZbkKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work investigates WindowLoRA: LoRA that can only observe a subset of the past context. Another way to view this method is attention dropout, where at each step, a random collection of positions in the context is forced to have zero attention. This method was inspired by the failure of models to retain their pretraining capabilities. By changing the available window for fine-tuning, WindowLoRA provides a controlled mechanism to compare behavior using the full context or only a subset. This work describes a small synthetic task, acting as a clean testbed for comparing different possibilities of forgetting. Experiments on this synthetic task reveal a trade-off between retention and adaptation, where greater amounts of constraints lead to better retention but worse adaptation. WindowLoRA is then tested on real models on natural datasets, again showing a trade-off between this method and others.

Critical strengths

This work provides both a view of why retention fails and a method that can interpolate between failing to retain and retaining. WindowLoRA is thus a strong avenue for experimenting with the failures of retention and a fine-tuning module that provides better control of retention. Additionally, this method could be used similarly to dropout in MLPs, where increasing the dropout fraction leads to more stable training and possibly less overwriting of pretrained behaviors.

Critical weaknesses

The paper's structure has been a bit challenging to follow. The related work is in the middle of the paper, breaking up the method definition and the experiments. The experiment definitions and results are in different sections, without this structure being described at the beginning of Section 5. It took a handful of rereads to realize Section 5.3's results are in Section 6.3.

A core element of WindowLoRA is that it provides a window into what is causing poor post-training performance: conflicting prompts or disjoint support. Figure 2 shows an example of each. However, the only main difference between the methods is the replay result on the last one (this was also hard to identify on the first few reads). This separation can only be seen if there is pre-training data that can be replayed, not in the common setting with frontier models where the pre-training data corpus and data weighting is unknown.

Questions for authors
  1. What precisely is the task used in Table 1? How are the 4 labels selected? The main body is light on details about how this task is created, and references to the appendix were not enlightening, nor was an anonymized codebase linked. The structure of this task could be quite enlightening regarding the results. Label generation being random or structured creates considerably different tasks.
  2. Figure 1 shows that the sampled locations are evenly spaced. Just to confirm, is this a stylistic choice for the figure? My understanding is that these positions are sampled randomly.
  3. Are there experiments testing whether real data shows disjoint support to the target task or not? This seems to be a key point in the development of WindowLoRA, but it is only shown to hold in synthetic settings. We don't have the exact training data for these models, but replaying some of WikiText along with the task should work similarly to the synthetic setting.

Minor Issues: Line 80: Quote formats look better with `` and ‘‘. Line 98: method -> methods. Line 240: figure -> Figure.

9 comments

We thank for the to highlight the motivation and states that WindowLoRA is controllable framework to study retention-adaptation tradeoff and also raising interesting connection between this and DropOut method in MLPs. We address the weaknesses and questions in the following sections.

AuthorsOpenReview ↗

W1: The paper's structure has been a bit challenging to follow concern

We will reorganize in the camera-ready version. Few things we will do:

  • The related works section after section 1 as this is more standard. Alternatively we might move this section into a section before the conclusion
  • The experiments definition and setting can be merged into one section (ie. merge section 5 and section 6)
AuthorsOpenReview ↗

W2: A core element of WindowLoRA is that it provides a window into what is causing poor post-training performance

First of all, indeed that one fundamental difference is that replay assumes that pretraining data accessible either fully or partially which in many of cases is not feasible in LLMs training and we don't assume such assumption for the WindowLoRA. Another difference is that our family framework also enables us to control the retention strength, unlike replay where thing could be harder to implement.

Reviewer_zqGaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies two sources of the retention–adaptation tradeoff in fine-tuning: intrinsic conflicts between old and new task targets, and avoidable limitations of preservation objectives. Using controlled sequence tasks and replay baselines, it shows that similar tradeoff curves can arise from either source. The authors introduce WindowLoRA, which adds sparse, local teacher–student hidden-state matching to LoRA, and study how increasing matching coverage connects it to dense distillation. Experiments on Qwen2.5-1.5B and Llama-3.1-8B show improved retention at a modest adaptation cost.

Critical strengths
  1. This paper studies via controlled experiments with replay baselines clearly distinguish unavoidable target conflicts from tradeoffs of adaptation methods.

  2. This paper introduced WindowLoRA, providing an interpretable framework for understanding how the position and context coverage affects adaptation and retention.

  3. Experiments on pretrained model families show consistent retention improvements over LoRA while explicitly quantifying the adaptation cost.

Critical weaknesses
  1. The paper is difficult to follow. The notation is poorly organized and insufficiently explained, which obscure the main ideas and the connections between theory and experiments.

  2. The paper studies the proposed two origins as main contributors of the adaptation-retention tradeoff. However, other factors such as limited model capacity, optimization difficulties receive insufficient analysis.

  3. WindowLoRA requires additional teacher and student forward passes, but the main experiments lack compute-matched comparisons with LoRA, leaving its practical efficiency unclear.

Questions for authors

See Weaknesses

4 comments

We thank the reviewer for the feedbacks, concerns, and highlighting about the soundness of our controlled study, how windowLoRA along with the task as a nice way to explore more about the retention-adaptation tradeoff. We will address the concerns and weaknesses raised in the following section.

AuthorsOpenReview ↗

W1: Hard to comprehend paper especially on the formalization part.

We will reorganize the formalization of the theory and experiments and make it much clearer. Can you clarify specifically in which parts that are hard to follow?

AuthorsOpenReview ↗

W2: Insufficient analysis on factors other than two origins

We agree that the two origins are not the only contributors to the adaptation–retention trade-off, and we will make this explicit later in the paper. Model capacity and optimization can also shape it. The paper's claim is narrower than a full decomposition. In a setting where the true targets are known, replay lets us tell an intrinsic trade-off (target conflict) from a method-induced one (incomplete preservation). Capacity and optimization are held fixed in that comparison, not studied. A full analysis of them needs its own controlled designs, which we consider beyond this paper's scope.

AuthorsOpenReview ↗

W3: WindowLoRA overhead cost.

Per micro-batch, WindowLoRA adds one teacher forward and one student forward–backward. Both run over k windows of w tokens per row; at the defaults (k = 4, w = ⌊L/4⌋) that is as many tokens as the batch itself. With: B as rows (sequences) per micro-batch, k as windows per row, w as window width, P as weights used in matrix multiplies, d as hidden dimension, l as transformer layers

  • Time. The extra cost is Θ(BkwP + Bkw²d). At the defaults kw ≈ L, so WindowLoRA stays in LoRA's class Θ(BLP + lBL²d). Memory and storage. WindowLoRA adds no parameters, optimizer state or checkpoint storage, and the teacher reuses the frozen base weights.
  • Activation memory during training grows by O(lBkwd), roughly doubling it at the defaults.

We therefore present the overhead as the price of an explicit, interpretable control (λ) over the operating point, not as a cheaper route to retention. Note that the efficiency is not our focus as our goal is to provide controllable family to study about the retention-adaptation tradeoff.

Reviewer_b3vfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies adaptation-retention trade-offs in model fine-tuning and trying to distinguish two possible sources of the trade-offs: the conflicts between old and new targets, and limitations of the preservation objective. In the experiements, the authors construct synthetic sequence prediction tasks and uses replay as a reference method. The results demonstrates that, both settings could have similar trade-off curves, although replay shows that the trade-off is possible to avoid.

Then the paper introduces WindowLoRA, which uses hidden-state matching from the frozen base model (as a teacher model) as a regularization in LoRA-based fine-tuning. The teacher-student regularization is performed on separated local windows, and the new task is learned with the full context. Experiments on Qwen2.5-1.5B and Llama-3.1-8B compare adaptation performance, retention perplexity. Additionally, activation drift is compared on the Qwen experiments. WindowLoRA improves retention over the selected LoRA baseline with some degradation to task performance. The authors interpret the method as a partial approximation to the unavailable old-task objective.

Critical strengths
  1. The synthetic experiments provide a useful controlled comparison and ablation. The target task distributions are known, and old and new tasks are constructed with explicit relationships. The adapter replay and full replay variants provides useful references for interpreting the results. Experiments are repeated across different sequence lengths, model sizes, and multiple seeds.

  2. The motivation of distinguishing the two sources of adaptation cost is good. In the synthetic experiments, even replay must compromise with a cost in case of conflicting setting, while in the compatible setting, replay can solve both task at the same time. And teacher matching still introduces an adaptation cost. This address the issue that a adaptation-retention trade-off curve alone is insufficient to explain forgetting, and using source model as a regularization on new task distribution does not necessarily preserve old task objectives.

Critical weaknesses
  1. The technical novelty is limited, and the window design is not well justified. The method mainly adds a local hidden-state matching term to LoRA-based fine-tuning. The choice of W=L/4 is empirical, with window-size ablations limited to PIQA and HellaSwag datasets. There is no enough theoretical justification or broader experimental evidence to explain why this choice could transfer across tasks. It appears to be a regularizer that may require careful task-specific tuning, rather than a generally justified preservation mechanism. Also, the regularizer does not rely on any LoRA-specific property, making the name “WindowLoRA” somewhat misleading.

  2. Concerns about the real-LLM experiment baselines. WindowLoRA is the only SFT method in the main comparison that explicitly receives additional supervision from the frozen source model. Closely matched baselines, such as LoRA with output distillation or full-context hidden-state matching are not compared in this real-LLM experiment. The retention gains may therefore come from teacher regularization rather than the local-window design. Also it's important to have comparisons at matched task performance, with tuned learning rates, training budgets, and regularization strengths.

Questions for authors

Q1. How sensitive is the preferred window width to the task and sequence length? Could the authors extend the width ablation beyond PIQA and HellaSwag, particularly to tasks requiring longer-range context (e.g., complex reasoning tasks)?

Q2. Could the authors compare WindowLoRA with output distillation and full-context hidden-state matching, tuning their regularization strengths and comparing retention at similar task performance?

Q3. Would tuning LoRA’s learning rate or training steps, or the reference-KL coefficient in PPO/GRPO, achieve similar retention gains? Could PiSSA/MiLoRA also receive method-specific learning-rate tuning before drawing conclusions from their activation drift?

Q4. Since the regularizer does not depend on LoRA, have the authors tested it with other parameterizations, such as full fine-tuning or another PEFT method?

12 comments

First and foremost, we thank the reviewer to highlight the main motivation to distinguish the sources of adaptation cost is sound and our synthetic tasks formulation is useful and sound to understand such problem. We will address the concerns and weaknesses raised in the following section.

AuthorsOpenReview ↗

W1: Hyperparameters concern and WindowLoRA naming

  • Regarding about the hyperparameters choice and its transferabiity to other tasks, we emphasize that our purpose for the WindowLoRA is to show that the local hidden-state matching would balance tradeoff between retention and adaptation as what the toy task has shown in the real task, rather than finding the most ideal hyperparameters. While we do agree that the limitation is that this choice of hyperparameter is needed for dataset-specific hyperparameters search as we already pointed out in D4 appendix These two-task results support fixed L/4 windows as a practical compromise, not a universal optimum.
  • The naming of WindowLoRA is because we mainly apply the regularization term only on LoRA as of now, despite we also added additional results on few other adaptation carriers described in the other section (c.f. Our response for Q4).
AuthorsOpenReview ↗

W2: Comparison matter on WindowLoRA as the only method with frozen model supervision.

It’s not true that only WindoLoRA uses additional supervision from the frozen source model. Policy methods use KL-divergence term between the policy and its initial frozen initial copy as an additional supervision. We also did experiments on output distillation and full-context hidden-state matching as additional baselines (c.f. Our response for Q2)

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Rebuttal Response

Dear area chair and reviewers, We thank all reviewers for their time providing constructive feedbacks, questions, and suggestions for our work. We are delighted to see that they express the strengths of our work as follow:

  • Sound controlled task to distinguish the sources of retention-adaptation tradeoff (b3vf, zqGA)
  • Good motivation to distinguish the sources of retention-adaptation tradeoff (b3vf, ZbkK)
  • WindowLoRA as a family for interpretable method to understand the tradeoff while also can be used for better regularizer to balance the tradeoffs(zqGA, ** ZbkK**)

However, many of the reviewers put concern about the experiments on the real task especially concerning about lack of baselines (b3vf, ZbkK) and hard-to-follow paper writings (zqGA, ZbkK). During the discussion, we primarily address these concerns:

  • Add dense distillation, replay, and output distillation for the real task (b3vf, ZbkK). Based on the results we see the findings on the toy task is also consistent with the real task, and moreover the continuum of retention-adaptation tradeoff observed in toy task is also observed in real tasks, reinforcing WindowLoRA as interpretability framework to understand the tradeoff. Moreover, we also add such regularization to other approach than LoRA (PiSSA and PSSOFT) and it helps to reduce the retention tax with insignificant adaptation degradation
  • Clarify some vague points and figures addressed by reviewer ZbkK
  • Add hyperparameter tuned baselines (b3vf). We observe that tuning the learning rates on the baseline SFT approaches could help but they aren't as good as WindowLoRA for balancing the tradeoff, meanwhile adjusting the KL coefficient term for policy-based approaches yields insignificant effect.

As the discussion phase is about end, we would be happy to address any further questions or concerns. We would appreciate for the reviewers to inform us whether our responses have addressed their concerns.

Best,

Authors of submission 10531

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.