- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Layerwise Function Anchoring (LFA) for continual domain adaptation. It stores estimates of submodule input distributions and penalizes changes in submodule outputs on samples from those distributions, together with L2-SP. Statistics from newly learned domains can be merged into the stored artifact, allowing subsequent adaptation without retaining previous-domain text.
LoRA experiments on Qwen3-0.6B and a reduced-tuning Qwen3-1.7B replication show relatively low training overhead and advantages over replay and LwF on sampled generative evaluations. Replay performs better on prior-domain and general-text perplexity.
Critical strengths
-
Replacing repeated processing of old sequences with stored statistics has practical value. Equation (2) clearly explains why input second moments suffice for the anchoring objective at linear submodules.
-
The paper strengthens several baselines with L2-SP and includes two judges, partial replications, and step-matched controls. Its disclosure of implementation defects and unfavorable results is helpful.
Critical weaknesses
-
The contribution remains too incremental. This is my main concern. At linear submodules, Equation (2) is an input-second-moment-weighted quadratic penalty on parameter changes. The paper acknowledges its connections to function regularization, activation statistics, and subspace protection (Section 3.1 and Appendix O).
LFA's particular contribution is to sample inputs directly for independent submodule matching, model nonlinear MLP inputs, and extend the statistics across domains. These choices form a useful adaptation procedure, but I do not find the advance over the established principles sufficiently substantial.
The discussion of projection methods also needs qualification. SGP already assigns importance using activation singular values and permits controlled updates within previously important subspaces (Saha and Roy, AAAI 2023, Method, Equations 2–3). It differs from LFA's function penalty, but shows that allowing graded changes in these directions is not itself new. The missing discussion does not prove that LFA duplicates prior work; it leaves part of the novelty argument incomplete.
-
The recommended training configuration depends on a defective loader schedule. Appendix N reports that random chunking offsets dropped short QA documents, leaving the supplement present in only eight of twenty epochs. The main LFA, LwF, and replay runs share this schedule, so the defect alone does not establish an unfair comparison between them. Some upstream-trainer baselines, however, receive different supplement exposure.
The corrected run reaches a comparable perplexity point five epochs earlier. Its epoch-15 judged score remains close to the original result, but its epoch-20 score is lower. This shows that training duration and checkpoint selection depend on the realized exposure schedule. Without a corrected and retuned comparison, the robustness of the recommended configuration remains unclear. The existing control does not establish that the relative rankings would change.
-
Single-domain judged comparisons remain exposed to selection bias. Appendix E selects configurations and training durations using judged scores on the same fixed set of 154 questions used for reporting comparisons. The single-domain search also gives LFA 55 configurations versus nine for replay (Section 4).
Paired tests on selected checkpoints do not account for this adaptive selection. The post hoc equivalence margin in Appendix D, which the authors appropriately describe as descriptive, also does not resolve the selection issue.
This concern should be limited to the judge-guided comparisons: GSM8K and IFEval were not tuning objectives, and the 1.7B settings were chosen on perplexity. Those evaluations provide supporting evidence, but do not independently validate the single-domain judged comparisons.
-
The generative evaluation leaves an important robustness question unresolved. Appendix H generates 300 outputs to obtain 200 scored examples, discarding and replacing truncated outputs. Paired tests correctly use shared item IDs, but the overall scores still depend on which outputs complete rather than measuring success on one fixed set with truncations counted as failures.
Repeated decoding supports several reported orderings, including the after-C comparisons at 1.7B. Nevertheless, this does not address the treatment of truncations or the limited training-seed coverage. A fixed-item evaluation would make the reported generative advantages easier to assess.
Questions for authors
-
What is the main methodological advance beyond the established regularization, activation-statistics, and feature-replay principles? How does the distinction from projection methods change when SGP is considered?
-
Under a corrected loader, with training duration retuned and supplement exposure matched, do the key LFA–baseline comparisons hold?
-
Is there a judged evaluation set that was not used for configuration or checkpoint selection?
-
Do the main Table 2 conclusions hold on identical fixed items when truncated outputs count as failures?