acceptodds
Withdrawn from ICLR 2027

Layerwise Function Anchoring: Preserving Sub-Module Functions on Sampled Hidden States for Continual Domain Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-tuning a language model on a narrow domain erodes the broad capabilities it arrived with, and can lead to catastrophic forgetting. The classical remedy, rehearsal, mitigates forgetting by replaying pretraining text at every stage. We present Layerwise Function Anchoring (LFA), which preserves capability with a statistic instead. LFA estimates, once, the distribution of hidden states entering each sub-module of the frozen original when text is passed through: a compact, sequence-free, task-agnostic statistic (under a tenth of the model's size). During adaptation LFA penalizes each sub-module's output drift independently on point samples , bypassing resource-intensive self-attention. No exemplars are stored, no text is generated during adaptation, and the statistic can be updated exactly as domains accumulate. Two knobs do the work: prevents forgetting by penalizing layerwise functional drift, and , an L2-SP penalty on all weights, covers the directions does not see. Anchoring sub-modules independently risks compounding error through depth; every whole-network measurement below (perplexity, judged answers, rule-scored generation) bounds that risk, though none isolates it, and the cleanest bound is tight: on Qwen3-0.6B adapted to a philosophy corpus under LoRA, unanchored adaptation inflates general-text perplexity by over 400% and LFA holds it to +1.1%. LFA's time overhead falls with model size, +14.8% per pass at 0.6B, +6.3% at 1.7B and +5.5% at 3B, at lower peak memory than Learning without Forgetting (LwF) at 0.6B and 1.7B. Judged by a frontier LLM, LFA matches tuned LwF at each method's best dose in paired per-question tests and is not separable from tuned replay, both strengthened with the same L2-SP penalty. Merging a learned domain's statistics into supports continual learning with no retained text: after a second domain LFA leads replay and LwF on the new domain's judged quality at all three ranks, and trails replay on old-domain perplexity. After a third domain its old-domain judged quality ends tied with replay, and LFA is ahead of both rehearsal and distillation on rule-scored generation (GSM8K, IFEval), where every perplexity on given text puts replay first. LFA's anchoring statistic need be produced only once, allowing model providers to support continual learning without divulging proprietary training data. When not available, the statistic can be formed using self-generated sequence samples, so LFA can run data-free, with nothing beyond the new domain's own text (one model, one domain). At Qwen3-1.7B (one seed) the continual-learning ordering is the same and every method loses less against the base model. All experiments use LoRA; seed coverage is partial.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual-learning, forgetting, catastrophic-forgetting, data-free, LLM, domain-adaptation, peft, lora

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YdUvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Layerwise Function Anchoring (LFA) for continual domain adaptation. It stores estimates of submodule input distributions and penalizes changes in submodule outputs on samples from those distributions, together with L2-SP. Statistics from newly learned domains can be merged into the stored artifact, allowing subsequent adaptation without retaining previous-domain text.

LoRA experiments on Qwen3-0.6B and a reduced-tuning Qwen3-1.7B replication show relatively low training overhead and advantages over replay and LwF on sampled generative evaluations. Replay performs better on prior-domain and general-text perplexity.

Critical strengths
  1. Replacing repeated processing of old sequences with stored statistics has practical value. Equation (2) clearly explains why input second moments suffice for the anchoring objective at linear submodules.

  2. The paper strengthens several baselines with L2-SP and includes two judges, partial replications, and step-matched controls. Its disclosure of implementation defects and unfavorable results is helpful.

Critical weaknesses
  1. The contribution remains too incremental. This is my main concern. At linear submodules, Equation (2) is an input-second-moment-weighted quadratic penalty on parameter changes. The paper acknowledges its connections to function regularization, activation statistics, and subspace protection (Section 3.1 and Appendix O).

    LFA's particular contribution is to sample inputs directly for independent submodule matching, model nonlinear MLP inputs, and extend the statistics across domains. These choices form a useful adaptation procedure, but I do not find the advance over the established principles sufficiently substantial.

    The discussion of projection methods also needs qualification. SGP already assigns importance using activation singular values and permits controlled updates within previously important subspaces (Saha and Roy, AAAI 2023, Method, Equations 2–3). It differs from LFA's function penalty, but shows that allowing graded changes in these directions is not itself new. The missing discussion does not prove that LFA duplicates prior work; it leaves part of the novelty argument incomplete.

  2. The recommended training configuration depends on a defective loader schedule. Appendix N reports that random chunking offsets dropped short QA documents, leaving the supplement present in only eight of twenty epochs. The main LFA, LwF, and replay runs share this schedule, so the defect alone does not establish an unfair comparison between them. Some upstream-trainer baselines, however, receive different supplement exposure.

    The corrected run reaches a comparable perplexity point five epochs earlier. Its epoch-15 judged score remains close to the original result, but its epoch-20 score is lower. This shows that training duration and checkpoint selection depend on the realized exposure schedule. Without a corrected and retuned comparison, the robustness of the recommended configuration remains unclear. The existing control does not establish that the relative rankings would change.

  3. Single-domain judged comparisons remain exposed to selection bias. Appendix E selects configurations and training durations using judged scores on the same fixed set of 154 questions used for reporting comparisons. The single-domain search also gives LFA 55 configurations versus nine for replay (Section 4).

    Paired tests on selected checkpoints do not account for this adaptive selection. The post hoc equivalence margin in Appendix D, which the authors appropriately describe as descriptive, also does not resolve the selection issue.

    This concern should be limited to the judge-guided comparisons: GSM8K and IFEval were not tuning objectives, and the 1.7B settings were chosen on perplexity. Those evaluations provide supporting evidence, but do not independently validate the single-domain judged comparisons.

  4. The generative evaluation leaves an important robustness question unresolved. Appendix H generates 300 outputs to obtain 200 scored examples, discarding and replacing truncated outputs. Paired tests correctly use shared item IDs, but the overall scores still depend on which outputs complete rather than measuring success on one fixed set with truncations counted as failures.

    Repeated decoding supports several reported orderings, including the after-C comparisons at 1.7B. Nevertheless, this does not address the treatment of truncations or the limited training-seed coverage. A fixed-item evaluation would make the reported generative advantages easier to assess.

Questions for authors
  1. What is the main methodological advance beyond the established regularization, activation-statistics, and feature-replay principles? How does the distinction from projection methods change when SGP is considered?

  2. Under a corrected loader, with training duration retuned and supplement exposure matched, do the key LFA–baseline comparisons hold?

  3. Is there a judged evaluation set that was not used for configuration or checkpoint selection?

  4. Do the main Table 2 conclusions hold on identical fixed items when truncated outputs count as failures?

Reviewer_qCMHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces regularization terms in the loss with the goal of mitigating forgetting on previous tasks when finetuning new tasks. There is a term that aims to match the representation at each layer to the representation of the original model. Here, the hidden states are sampled from a distribution close to the distribution on natural previous tasks, computed once at the beginning of continued training. There is also a term that minimizes the L2 distance of each weight matrix before and after training. Finetuning is done with LoRA on Qwen3 models at the 0.6-3B scales on 3 unrelated corpuses.

Critical strengths
  • The motivation that experience replay is not tenable for language models is sound and correct.
  • The approach proposed by the authors is reasonable (though there are some potential weaknesses) and deserves to be cleanly evaluated.
Critical weaknesses
  • The paper is very poorly written and not easily digestible by humans. For example the abstract is far too long and contains too many details. Eg. this is a sentence in the introduction "What we find is a method that works, by a different mechanism; its standing against tuned alternatives is our evidence, not a claim that it is the best option"
  • Terminology for common ideas is atypical. I don't believe the writing of the paper to be suitable for a venue such as ICLR without significant revision.
  • The introduction contains far too many experimental details.
  • Experiments are only in the 0.6B-3B scale. Further evaluation would be needed to have a sense of whether the method scales to larger models. Also, the method was only trained with LoRA, which naturally restricts how much the base weights can change. To know whether the method works as motivated, it is important to test full finetuning.
  • The paper has no figures, only a single diagram. The results tables are difficult to read.
  • The paper contains an unreasonably large appendix which is very terse and hard to read.
  • As far as I understand, many of the evaluations are on pretrained models. A more meaningful result given the motivation would be to test the method on post-trained models and report benchmark performance rather than perplexity.
  • For the method, it is a reasonable idea but should compete against the new learning objective. Given this, I'd expect a learning-forgetting tradeoff which I did not see in the results. The experiments are not rigorous or comprehensive enough to know whether the method works and how it compares to baselines. For example, simple unconstrained training does not appear as a baseline at all anywhere in the main paper and the main baseline is LwF.
Questions for authors

See above.

Reviewer_NvJ6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces Layerwise Function Anchoring (LFA), a method designed to reduce catastrophic forgetting when large language models are fine-tuned sequentially on new domains. The method tries to preserve behavior at the level of individual sub-modules instead of through whole-sequence replay or distillation. The authors argue that instead of retaining and replaying old training text, storing compact statistical estimates of the hidden-state distributions entering each sub-module (Q/K/V/O heads in attention layers and the MLP projection layers) and then penalizing the deviations between the original and adapted sub-module’s outputs on the sampled hidden states is sufficient to reduce the effect of catastrophic forgetting. Method: The frozen base model’s hidden-state distributions, P(h) is fitted at each sub-module on a proxy corpus. This is a correlated Gaussian at linear sites and a 32-component mixture at MLP inputs. These distribution parameters are stored per task wise as an artifact. During adaptation, vectors are sampled from P(h) and each sub-module’s drift is penalized separately together with an L2-SP weight penalty. For continual learning, adapters are fused and the new domain’s states are merged into p(h), so earlier texts are not needed. A variant uses self- generated proxy text. The authors validate their method on Qwen3-0.6B (with 1.7B and 3B in limited form) on three small domains (philosophy, cybersecurity, archaeology) at LoRA ranks 4, 16 and 32. Evidence comes from perplexity, LLM-judged domain answering with paired tests and an equivalence margin, and rule-scored GSM8K and IFEval generation. The paper reports: ● It limits WikiText-2 perplexity increase to +1.1%, against +444% unanchored. ● It ties tuned LwF+µ and replay+µ on single-domain judged quality at each method’s best dose. ● It beats them on the new domain after a second domain and on GSM8K and IFEval after a third, while trailing replay on old-domain and general-text perplexity. ● Its time overhead is lower than LwF and replay’s, falling with model size. ● A self-supplied artifact ties the real-text one and ties SSR. The authors state that results are matches rather than wins, and that judged old-domain advantages are largely answer style and length.

Critical strengths
  1. The method provides a practical, cheap and replay-text free preservation recipe with considerable effectiveness over unanchored adaptation.
  2. The method shows some advantages beyond ties. After a second domain it is ahead of the tuned LwF+µ and replay+µ arms on the new domain at all three ranks, so anchoring does not appear to cost plasticity.
  3. The method’s compute overhead is lower compared to replay based methods.
  4. The authors evaluations are carefully calibrated and reports negative and null reports honestly.
Critical weaknesses
  1. The evidence for the central claims rest on a weak measurement and On the judged single-domain measure, the evidence does not show that LFA is better than the alternatives: a. Judged scores sit near the floor (about 1 on a 5 point scale). The trained model(0.968) ties the frozen base model (0.931), so the judge can barely distinguish between the adapted and unadapted model. b. The headline results are mostly ties. LwF+µ is ahead at matched dose 15. Whereas replay is not separable, and LFA is behind SSR one lambda step from the shipped value. c. The authors themselves say old-domain judged leads are largely because of answer style and length. d. 17 to 26% of the verdicts depend on which judge LLM is used .

  2. The proposed mechanism is not connected to the outcomes, and the theory does not cover what is trained. a. Prop. 1 covers an exact limit that the paper says training does not reach. And the proposition proves nothing for a finite lambda. It also does not cover compounding over depth or over merges. b. LFA’s functions drift 37-313x less than the baselines. Yet it’s old domain perplexity increase is largest at every rank (Table 1). The authors state that none of the drift numbers predict judged quality. c. The key ablation (fitted P(h) vs isotropic gaussian) is confounded by n (512 vs 16) and epochs (30, 20).

  3. The generative lead runs against every likelihood measure, and the paper does not explain it. After the third domain, replay is first on WikiText-2 (17.71 vs LFA’s 18.76), on instruction text, and on GSM8K gold-solution perplexity (3.11 vs 3.61), yet LFA scores higher on sampled GSM8K and IFEval. LFA also ends below the base model on both tasks (65 vs 101; 104 vs 116). The authors attribute this to LFA preserving “format and multi-step reasoning behaviour”. That is a hypothesis, the authors did not perform any tests to confirm that. Combined with the drift result, it leaves open what the mechanism actually preserves.

Questions for authors
  1. Were lambda, dose and rank chosen using the same 154 judged questions that produce the reported paired tests? If so, please report results on questions held out from tuning.
  2. What are the source datasets for the corpus P(h) is fitted on?
  3. LFA’s functions drift 37-313x less than LwF+ µ and replay+µ , yet LFA has the largest old-domain perplexity increase at every rank. The authors say none of the drift numbers predict judged quality. What does the drift evidence establish about the mechanism?
  4. Do the conclusions hold across several domain orderings and chains of at least 5 domains? Domain A has only 77 documents, so larger domains would also help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.