- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Meta-HA studies replay-free continual adaptation of frozen LLMs. It meta-trains LoRA priors over simulated task sequences, trains a controller on each downstream task, and uses its outputs to write updates into persistent adapter memory. Experiments on three NLP continual-learning benchmarks report improved average performance and backward transfer, with additional backbone, task-order, and component studies.
Critical strengths
-
Table 1 shows consistent gains across three benchmarks, including near-zero BWT. The additional models and task orders support the usefulness of the approach beyond one configuration.
-
ANML receives the same meta-training stage, and Tables 2-3 test controller removal, random initialization, and multi-task pretraining. These are useful controls for attributing the gains.
Critical weaknesses
-
Important 2026 comparisons are missing (Sections 2 and 5.1). Meta-UCF (Xiao et al., ICLR 2026) also generates LoRA updates with a shared hypernetwork for LLM continual learning. It uses replay, so that difference should be made explicit, but it is a close methodological comparison. SLAO, from Merge before Forget (Qiao and Mahdavi, ICLR 2026), continually merges updates into one LoRA and evaluates on closely related benchmarks. Neither appears in the references or experiments. Including FOREVER is welcome, but these omissions leave the claimed advantage over generated adapters and single-adapter consolidation uncertain.
-
The novelty claim needs a narrower explanation (Section 4; Algorithm 1). The outer loop learns LoRA priors, while the controller is reset and trained conventionally for each task. This looks like meta-learned adapter state combined with a task-trained writer; the paper should explain what makes it a transferable update strategy beyond that combination. K-DeCore (Chen et al., NeurIPS 2025) is relevant background on separating reusable and task-dependent functions with fixed trainable parameter counts. It uses example replay and does not generate adapter updates, so it is a positioning reference rather than a required baseline.
-
The experimental evidence does not fully establish resource-efficient transfer (Section 5.1; Appendices A and D). Meta-training uses 1,187 auxiliary tasks. Matched ANML training and Table 3 help, but total training cost and parameter budgets are not compared. Evaluation-task exclusion is stated without source-dataset or example-level exclusion details, leaving the strength of cross-dataset transfer unclear. Category-based pool ablations are useful but do not resolve this. Seed counts and uncertainty estimates for the main tables are also missing.
-
A central implementation step remains underspecified (Section 4.2; Appendix C; Algorithm 1, line 13). The trained controller writes using the full task training set, but the paper does not explain how example-level writes are aggregated or whether each write changes the state used to compute subsequent writes. These choices affect update magnitude and order dependence. Explicit pseudocode separating controller fitting, memory consolidation, and prediction would make the method reproducible.
Questions for authors
- How does Meta-HA differ from Meta-UCF's parameter generator, and why was SLAO excluded from the comparisons?
- Were alternative formulations of the same source datasets excluded from meta-training? Please give the exclusion rule and clarify validation/test separation.
- How are full-dataset writes aggregated? Please also report the existing runs' seed counts, parameter counts, and meta-training versus downstream training cost.
5 comments
For Weakness 4 and Q3: about "memory writing details and other key steps".
During controller optimization, each training batch generates a temporary update using Eqs. (3)–(4). This update is temporarily applied to compute the task loss and optimize the controller, but is discarded afterward, leaving the persistent task memory unchanged. Thus, updates generated by earlier training batches do not alter the memory state used by subsequent batches. After controller optimization, the trained controller is fixed and used to generate updates for all examples in the full task training set, starting from the same persistent memory state. Each example-level update is obtained by averaging over its target tokens (Eq. 4), and the resulting updates are then averaged equally across all training examples. The brief pseudocode of controller training, memory writing, consolidation and prediction are as follows:
Algorithm: Inner Optimization,Task-Level Memory Consolidation and Prediction
Input: Task training set , sampled subset , meta-learner , task memory , and controller .
Stage 1: Controller Optimization
- Initialize the task-specific controller and freeze the meta-learner and persistent task memory.
- For each training batch from :
- Generate masked example-level updates using Eqs. (3)–(4).
- Compute the temporary batch update: = mean( for in ).
- Reshape into LoRA-factor increments and temporarily apply them to task memory for prediction and controller optimization.
- Discard the temporary update after the optimization step, leaving persistent task memory unchanged.
Stage 2: Task-Level Memory Consolidation
- Fix the optimized controller.
- Traverse the full training set and generate masked example-level updates using Eqs. (3)–(4).
- Compute the task-level update by averaging over all training examples: = mean( for in ).
- Reshape into LoRA-factor increments and .
- Write to persistent task memory exactly once: , .
Stage 3: Prediction
Disable the controller and perform prediction using the frozen base model, meta-learner, and consolidated task memory (Eq. 6), without generating additional updates.
We have added the aggregation formula and corresponding pseudocode to Subsection 4.3 and Algorithm 1 of the revised manuscript.
For Q3: about "more training details include seed counts, parameter counts and training cost".
- Seed Count. To evaluate the robustness of Meta-HA, we conducted additional experiments using three random seeds (40, 41, and 42) on Qwen3-0.6B across all three CL benchmarks. The results are reported as mean ± standard deviation:
| Seed | Std. CL AP | Std. CL BWT | Long Seq. AP | Long Seq. BWT | SuperNI AP | SuperNI BWT |
|---|---|---|---|---|---|---|
| 40 | 76.4 | 1.1 | 73.2 | 2.8 | 46.2 | 2.4 |
| 41 | 74.6 | -0.3 | 70.8 | -0.8 | 46.4 | -1.8 |
| 42 | 75.3 | -0.2 | 71.7 | -0.3 | 47.4 | -0.8 |
| Mean ± Std. | 75.4 ± 0.91 | 0.2 ± 0.78 | 71.9 ± 1.21 | 0.6 ± 1.95 | 46.7 ± 0.64 | -0.1 ± 2.19 |
The results confirm that Meta-HA's performance gains are robust across random seeds. Compared with the strongest baseline reported in Table 1, Meta-HA achieves best performance on Standard CL, Long Sequence, and SuperNI, respectively. These additional results strengthen the empirical evidence that Meta-HA's continual learning improvements are not dependent on a particular random seed. We have included the seed counts and corresponding uncertainty statistics in Appendix D.6 of the revised manuscript.
- Parameter Budget. We provide a detailed parameter comparison on Qwen3-0.6B:
| Method | LoRA Memory | Controller / Modulation | Total Trainable |
|---|---|---|---|
| LoRA (r=8) | 1.15M | – | 1.15M |
| ANML | 1.15M | 0.041M | 1.19M |
| ANML (All-layer) | 1.15M | 1.15M | 2.29M |
| Meta-HA | 1.15M | 10.32M | 10.47M |
ANML configuration. Following the original ANML design, we apply the modulation network to the final layer, resulting in 1.19M trainable parameters. We also investigated extending modulation to all LoRA layers to increase its parameter budget. However, despite increasing the trainable parameters to 2.29M, this variant performed worse than the standard implementation. Therefore, we retain the original configuration for a faithful and competitive comparison.
Meta-HA's advantage. Although Meta-HA introduces additional training-time parameters through the controller, these parameters are dedicated to learning task-adaptive update behaviors rather than expanding persistent knowledge capacity. The meta-learner and task memory operate within the same rank-8 LoRA space (1.15M), maintaining the same effective adapter rank as conventional LoRA. Moreover, the controller is disabled after memory consolidation and does not participate in inference. Thus, Meta-HA achieves stronger continual learning performance without increasing the effective persistent adapter rank or introducing additional controller inference overhead, despite its higher training-time parameter budget.
We have included this analysis in the Appendix D.7 of the revised manuscript.
- Training Cost. We compare Meta-HA and ANML under the same meta-training protocol using two NVIDIA RTX 4090 (48GB) GPUs.
| Method | Meta-training (GPU-hours / 100 episodes) | Long Seq. Training(GPU-hours) |
|---|---|---|
| ANML | 49.7 | 1.0 |
| Meta-HA | 65.2 | 0.8 |
Meta-HA incurs 31.2% higher meta-training cost but achieves 20% lower downstream training cost than ANML. The additional meta-training overhead arises from repeatedly optimizing the controller and meta-learner through bi-level learning. However, this upfront investment enables more efficient downstream knowledge acquisition. Meta-HA trains its controller on only a subset of task samples, then uses the learned update mechanism to consolidate knowledge from the full dataset without further gradient-based optimization. In contrast, ANML relies on conventional gradient-based adaptation over the task training data. Consequently, Meta-HA reduces downstream training cost despite its substantially larger controller, demonstrating the efficiency benefit of learning transferable update behaviors.
We have included this analysis in the Appendix D.8 of the revised manuscript.
Response to Reviewer 6EtS
Response to Reviewer 6EtS
We sincerely thank you for your valuable comments to help improve our paper.
For Weakness 1 and Q1: about "analysis of Meta-UCF and SLAO".
Meta-UCF. Both methods generate LoRA parameters, but differ in training strategy and knowledge consolidation. Meta-UCF continuously trains a shared hypernetwork to generate task-specific LoRA adapters conditioned on historical replay samples. In contrast, Meta-HA meta-learns transferable conditioning priors offline, which guide task-specific controllers to generate incremental updates for persistent memory consolidation without downstream replay. We attempted to reproduce Meta-UCF, but insufficient implementation details prevented a reliable comparison. We therefore clarified the methodological differences in the Related work of the revised manuscript.
SLAO. We have now reproduced SLAO following its algorithm and evaluated it on Qwen3-0.6B, averaging results across multiple task orders.
Meta-HA consistently achieves higher AP across all three benchmarks, with improvements of 1.5, 0.3, and 4.0 points on Standard CL, Long Sequence, and SuperNI, respectively. We have incorporated these comparisons into Table 1 of the revised manuscript.
For Weakness 2: about "explanation of transferable update strategy".
Meta-HA's contribution is not merely combining a meta-learned adapter state with a task-specific writer, but learning a transferable update strategy through the bi-level optimization of their interaction.
What makes the strategy transferable? The meta-learner fθ learns input-conditioned priors through outer-loop optimization over CL episodes, where its effectiveness is evaluated after sequential adaptation and memory consolidation. Crucially, the meta-learner and accumulated task memory fϕ jointly contribute to prediction (Eq. 6) and condition the controller's update generation (Eqs. 2–5). Although each task-specific controller gψ is reset and independently optimized, its adaptation and generated updates depend on these shared priors and the evolving memory state. Thus, what transfers is not the controller parameters, but a meta-learned prior optimized for the entire sequential update-and-consolidation process, rather than merely an initialization for predefined gradient-based adaptation.
Distinction from K-DeCore. K-DeCore separates task-agnostic schema filtering and task-specific query building, focusing on functional decomposition for structured knowledge reasoning. In contrast, Meta-HA separates transferable update conditioning, task-specific parameter update generation, and persistent knowledge consolidation. Unlike K-DeCore's replay-based approach, Meta-HA explicitly generates LoRA-factor updates without requiring historical sample replay during downstream adaptation.
We have clarified these distinctions and position K-DeCore appropriately in the Subsection 4.1 and Related Work of the revised manuscript.