acceptodds
Under review as a conference paper at ICLR 2027

DualKL: Distributional Trust Region for Continual LoRA Adaptation

Abstract

Continual adaptation requires large language models to learn new tasks without losing capabilities acquired from earlier ones. Existing LoRA-based methods mainly reduce interference in parameter or gradient space, while replay and distillation often apply the same preservation strength to all previous tasks. These strategies have important limitations. Parameter-space separation does not directly guaranty stable model outputs, average drift can hide severe forgetting on an individual task, and a fixed regularization strength cannot provide different levels of protection for tasks with different degrees of vulnerability. We propose **DualKL**, a task-wise primal–dual approach that preserves previous behavior through predictive-distribution trust regions. After each task, DualKL stores a small set of representative examples together with the model’s output distributions. When learning later tasks, it measures the Kullback–Leibler (KL) divergence between the stored and current distributions for each previous task. Every task is assigned an independent constraint and an adaptive dual variable. The protection strength increases when a task exceeds its allowed drift and remains weak when the task is stable, allowing the model to retain earlier knowledge without unnecessarily restricting new-task learning. We evaluate DualKL on Standard CL, the complete Long Sequence benchmark, and TRACE across multiple language-model architectures. The results show that DualKL consistently improves overall performance and reduces both average and worst-task forgetting compared with replay, parameter-space methods, and fixed functional regularization. These findings demonstrate that selective control in function space is an effective approach for continual language-model adaptation. Our code are available at https://anonymous.4open.science/r/DualKL.

Primary area:
learning theory
Keywords:
LoRA, Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_22kQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DualKL, a method for LoRA training that preserves past tasks' predictions when learning a new task. The paper argues that parameter space constraints for preventing forgetting, like orthogonal updates to weights, do not necessarily preserve model predictions. DualKL constraints the KL divergence between the task's prediction right after learning with predictions while learning a new task to prevent the drift in that task.

Critical strengths
  • The results on the 15-task long-sequence benchmark show DualKL as having better final accuracy and less forgetting as more tasks are learned
  • Their gains hold up on different model families (llama, qwen and t5-large). The paper also reports metrics on training runtime and memory
  • Appendix includes ablations and sensitivity analyses (warmup steps, anchor set size, KL budget, etc.)
Critical weaknesses
  • The method requires access to past task examples, making it limited for practical use cases
  • For the anchor set for the previous task that is required to be maintained when learning a new task, I am concerned that it adequately represents the task
  • The method controls average KL divergence over the anchor set, which does not guarantee that every anchor satisfies the individual KL condition required by the decision-preservation guarantee.
  • Table 23 shows gains from tuning the warm-up and ramp schedule. It is unclear whether the gains transfer to other baselines like fixed-KL, or whether DualKL's adaptive weighting is driving those improvements.
Questions for authors
  • According to Table 1, the initial learning accuracy for DualKL is much higher than that of other methods. Final performance, including retention, could benefit from this. Table 23 shows that warm-up and ramping contribute to the gain, but I am not sure if it explains it entirely. Were other baseline (like the Fixed-KL) experiments also evaluated on different warm-up steps?
  • Following on the previous, can the authors provide the breakdown for per-task accuracy? What drives the increase of ~86 compared to ~80 for other tasks?
  • Can the authors comment on the anchor coverage? The anchor size ablation table compares various anchor set sizes, but have the authors tried including different subsets? Like, say, some under-represented examples
Reviewer_qEbaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses catastrophic forgetting in continual LoRA adaptation for large language models. It highlights that parameter-space constraints cannot directly ensure predictive stability, while average regularization may overlook severe forgetting on individual tasks and fixed weights cannot adapt to varying forgetting risks. To address these issues, the authors propose DualKL, which stores historical samples and their predictive distributions, uses task-wise KL constraints to limit changes in historical predictions, and introduces independent dual variables to adaptively adjust the protection strength for each task. Experiments on Standard CL, Long Sequence, and TRACE report improvements in overall performance and reductions in forgetting.

Critical strengths
  1. The paper identifies an important limitation of aggregate regularization, where average predictive drift may hide severe forgetting on individual tasks. The task-wise analysis provides a useful perspective on preserving historical knowledge beyond average performance.
Critical weaknesses
  1. The methodological novelty of DualKL appears incremental. DiSC (ICML 2026) preserves previously learned capabilities by minimizing KL divergence between teacher and student predictive distributions, while Primal Dual Continual Learning (Elenter et al., 2023) already employs task-wise constraints and corresponding dual variables to mitigate forgetting. The paper does not clearly explain how DualKL differs from these studies.

  2. The experimental comparisons are insufficient. First, the KL-based baselines are constructed by the authors, without comparisons to published methods such as [3], making it difficult to assess DualKL's effectiveness. Second, DualKL constrains predictive distributions without directly restricting parameter updates, potentially causing inter-task interference. Comparisons with recent methods addressing parameter importance and task conflicts, such as [4] and [5], are missing.

[1] Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilities. ICML, 2026. [2] Primal Dual Continual Learning: Balancing Stability and Plasticity through Adaptive Memory Allocation. arXiv, 2023. [3] Functional Regularisation for Continual Learning with Gaussian Processes. ICLR, 2020. [4] Turning the Tables: Enabling Backward Transfer via Causal-Aware LoRA in Continual Learning. NeurIPS, 2025. [5] Revisiting Weight Regularization for Low-Rank Continual Learning. ICLR, 2026.

Questions for authors
  1. The definition of functional regularization is not sufficiently clear. Could the authors further clarify its fundamental differences from parameter-space regularization, particularly in how the two approaches preserve historical knowledge and mitigate forgetting?

  2. The necessity of primal–dual optimization is not sufficiently justified. The paper does not explain why task-wise adaptive protection requires a primal–dual formulation rather than simpler adaptive weighting strategies. Could the authors justify this design choice and provide comparisons with alternative adaptive regularization methods?

  3. Analyzing convergence is important to determine whether the adaptive protection mechanism can consistently control forgetting during continual learning. Could the authors further examine the convergence and stability of dual updates as the number of tasks increases?

  4. The effectiveness of task-wise KL constraints may depend on anchor selection, as the dual variables are updated based on KL divergence measured on these samples. Although Table 20 examines different anchor-bank sizes, it does not compare selection strategies. Could the authors evaluate how different sampling strategies affect the reliability of task-wise protection?

  5. The substantial performance improvement from combining the two components requires further explanation. As shown in Table 4, each component provides only modest improvements individually, while their combination leads to a much larger gain. The authors mainly attribute this improvement to the complementarity of the two components, which is not sufficiently convincing. Could the authors provide further analysis or experimental evidence to explain this result?

11 comments
Area_Chair_Ha7ROpenReview ↗

Dear Reviewer qEba

Dear Reviewer qEba,

Please carefully review the authors' rebuttal and respond to each of your concerns. If any concerns remain unresolved, clearly specify what is missing and request concrete clarifications or additional evidence.

Bests,

Your AC.

AuthorsOpenReview ↗

Response to Reviewer qEba

Thank you again for your review. As the discussion period ends soon, we would like to kindly ask whether our response and our revised paper have addressed your concerns. If anything is still unclear, we would be glad to answer in the remaining time.

Reviewer_qEbaOpenReview ↗

Official Comment

Thank you for the clarifications and additional experiments. The revision clarifies how DualKL differs from DiSC and PDCL, and the comparisons with CaLoRA and EWC-LoRA broaden the empirical evaluation. The task-wise EMA baseline addresses my question about simpler adaptive weighting, and the comparison between class-balanced and random sampling answers my question about anchor selection. However, I have two remaining concerns.

First, results shows that DualKL reduces forgetting on DBpedia from 1.8 to 0.8 percentage points, while accuracy on Amazon falls from 90.5% to 90.2%. The gain over the PDCL-style baseline is therefore modest in terms of task performance. Moreover, DBpedia is the only previous task to protect during training on Amazon. This experiment does not test whether DualKL can use predictive drift to adjust the level of protection for each task as forgetting risks change across multiple previous tasks.

Second, FRCL is a directly relevant functional regularization baseline. The comparisons with CaLoRA and EWC-LoRA evaluate parameter-space methods, while the KL ablations assess design choices within DualKL. Without a direct comparison with FRCL, it remains unclear what advantage DualKL offers over existing functional regularization methods.

For these reasons, I remain unconvinced of the proposed method’s effectiveness.

Reviewer_bhhqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DualKL, a task-wise distributional trust-region framework for continual LoRA adaptation of large language models. The work addresses three limitations of existing continual learning methods: parameter-space constraints do not directly guarantee behavioral stability, aggregate functional regularization can conceal severe forgetting on individual tasks, and fixed regularization coefficients cannot adapt to task-specific vulnerability. The paper further connects KL drift to anchor-level decision preservation through a margin-based theoretical proposition and provides held-out probe analyses, sensitivity studies, and cross-backbone evaluations.

Critical strengths
  1. The paper identifies a meaningful limitation of parameter-space interference control: parameter distance or gradient orthogonality does not directly guarantee preservation of model behavior.
  2. Unlike a single mean KL penalty, DualKL assigns an independent constraint to each previous task. This prevents stable tasks from masking severe drift on a vulnerable task and is particularly relevant to long task sequences.
  3. The experiments cover short and long discriminative task sequences as well as generative continual learning. The reported improvements extend to T5-Large and Qwen2.5-7B-Instruct, suggesting that the approach is not restricted to the primary LLaMA architecture.
Critical weaknesses
  1. DualKL constrains the predictive distributions only on a compact set of stored examples. Its theoretical decision-preservation guarantee applies to individual anchors and does not establish preservation over the full previous-task distribution.
  2. DualKL depends on the drift budget, dual learning rate, dual-variable cap, warm-up period, and protection ramp. The reported drift-budget ablation is non-monotonic: overly restrictive constraints reduce new-task learning, while loose constraints permit greater forgetting.
Questions for authors
  1. How reliably do the task-wise trust regions protect previous-task performance on rare, boundary, or distribution-shifted examples that are not represented in the stored memory?
  2. How sensitive is DualKL to the choice of drift budget, dual learning rate, and protection schedule when transferring across different task sequences and model architectures?

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Area Chairs

Dear AC, thank you for coordinating the discussion. We have now responded to all reviewer concerns with additional experiments. As the discussion period remains open, we would be grateful if you could check with the reviewers whether our responses adequately address their concerns, or whether any further questions or clarifications remain. We are happy to provide additional results or analyses promptly within the discussion window. Thank you for your time.

AuthorsOpenReview ↗

Response to Reviewer 22kQ

Thank you again for your review and your follow-up on our additional experiments. As the discussion period ends soon, we would like to kindly ask whether our response and the revised paper have addressed your concerns. If anything is still unclear, we would be glad to answer in the remaining time.

The clarification that the original classification results used consistent, dropout-disabled evaluation addresses my main evaluation concern. Given the reported improvements across benchmarks and model families, I am raising my rating. However, the benefit of adaptive weighting over Fixed KL under matched protection schedules remains unresolved, and I have not treated the planned experiments as evidence.

AuthorsOpenReview ↗

Response to Reviewer 22kQ

We sincerely thank Reviewer 22kQ for the positive assessment and consistent response. We also appreciate the remaining concern that we have conducted this controlled comparison and will include the complete empirical results and the corresponding analysis in the revised PDF, rather than presenting it as a planned experiment.

We hope the added results will fully resolve this question, and we remain available for any further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.