- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MIITA addresses supervised continual learning for small language models under a fixed persistent-storage budget. It freezes the backbone and converts each supervised example into an averaged negative loss gradient in the pre-LM-head hidden space. Similar correction directions are consolidated into prototypes, with semantic anchors supporting retrieval. At inference, semantic similarity and a predictive-entropy gradient guide prototype selection; the aggregated direction is applied through a state-dependent gate during decoding.
The contributions are a correction-oriented memory representation, budget-aware memory management, and a lightweight inference-time adaptation mechanism. Validation covers four benchmarks spanning classification, multilingual QA, and instruction following, with additional model-size, storage-budget, retrieval, and gating analyses. The paper reports consistently better final performance and backward transfer than its evaluated baselines. Its theory provides local first-order interpretations and a conditional loss-reduction bound.
Critical strengths
- Computing correction vectors directly from the LM head avoids backpropagation through the backbone, while inference requires neither demonstrations in the prompt nor parameter optimization. Separating semantic retrieval keys from functional correction values is a useful design choice. This matters because it targets both storage and inference costs, rather than addressing forgetting alone.
- Table 1 reports improvements across all four benchmarks and both principal model sizes. Appendix D.1 strengthens the comparison by including byte-matched SAPT-LoRA and kNN-LM: MIITA exceeds SAPT-LoRA by 3.65 OP points on BANKING77 and 3.70 on SUPERNI. Table 2 also shows that combining semantic and uncertainty signals improves over either alone, while the gating and adversarial-retrieval experiments provide additional evidence for the proposed inference mechanism. These results make the approach promising beyond a single dataset or baseline.
Critical weaknesses
- The experimental reporting does not sufficiently establish the claimed learning–retention advantage or make the storage-matched comparisons auditable. The frozen backbone is reported as a latency reference, but its task performance is absent. Because MIITA preserves backbone weights, this reference is essential for measuring how much it actually learns beyond existing capabilities. Aggregate OP and BWT also conceal the immediate acquisition and subsequent retention of individual tasks; less-negative BWT can partly reflect lower initial acquisition.
- The central comparison depends on extremely small storage allocations, but the manuscript does not report the actual LoRA ranks and target modules, numerical storage precision, or per-method storage breakdowns. Appendix D.1 states that these quantities are controlled without providing the configurations. Task identities/orders, optimization settings, and the source of variability across the three runs are also insufficiently specified. These omissions materially limit confidence that the reported advantage reflects the method rather than particular baseline configurations.
- Insufficient positioning against function-vector methods and isolation of the proposed contribution. The paper omits discussion of function-vector-based continual learning methods [1], which directly connect compact activation-space vectors to continual learning, propose function-vector-guided training, and demonstrate that injecting original task function vectors during inference can recover performance after forgetting. It is therefore important prior work for MIITA’s claim of preserving and reusing functional adaptation signals.
- The memory-format ablation does not isolate the benefit of storing correction directions. ExampleMem and SummaryMem use prompt insertion, whereas DirectionMem uses hidden-state adaptation. Because both what is stored and how it is applied change, the results do not establish whether the gains come from correction directions, hidden-state adaptation, or their combination.
- Forward transfer to unseen tasks is not evaluated. The paper reports final performance and backward transfer (BWT), but not forward transfer (FWT). It therefore remains unclear whether accumulated correction memories improve or impair performance on future tasks before learning them, limiting the assessment of MIITA’s generalization beyond previously seen tasks.
[1] Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning. ICLR 2025.
Questions for authors
- What are the frozen backbone’s scores on all four benchmarks? Can you provide the stage-by-task performance matrices, or at least immediate post-learning and final scores for each task, to distinguish acquisition from retention?
- What exact storage precision, adapter ranks/target modules, and storage allocations were used for each baseline? Please also specify benchmark task selections/orders, validation-based hyperparameter selection, and what varies across the three runs?
- Does the reported MIITA use the union of semantic and directional candidates in Eq. (8), or the semantic-only candidate pool described in Appendix C.4? What values are used for the semantic-key layer, restricted vocabulary size, entropy-probe scale, and stability threshold, and what precisely determines the memory-repair operations?
6 comments
Response to Reviewer 95DD (Part 2/3): W2-W3 & Q2
W2 & Q2. Experimental configurations and reproducibility.
(1) Configurations and storage accounting. Appendix C.2 reports the baseline adaptations, adapter configurations, and counted storage components. For SuperNI with Qwen3-0.6B, persistent numerical arrays use FP32 and the shared storage cap is 133,240 bytes:
| Method | Configuration | Serialized storage (bytes) |
|---|---|---|
| LoRA (sequential) | Rank 1; last six layers, q/v projections | 128,880 |
| Per-task LoRA | Rank 1; last-layer v projection; 15 adapters; oracle task ID | 122,940 |
| SAPT-LoRA | Rank 1; last-layer v projection; 15 adapters; 32-D keys; 32→8→32 selector | 129,884 |
| MBPA++ | Frozen-SLM retrieval; temporary local LoRA | 129,322 |
| CIS | 256-token summary cap; k=4 | 97,824 |
| InCA | 64-D tags; k=4; ρ=0.01; 256-token summary cap | 117,762 |
| CT0 | Rank 1; last-layer q/v projections; replay | 129,578 |
| FOREVER | LoRA with model-time replay scheduling | 133,114 |
| DKVB | 128 subspaces; 16 pairs/subspace; 8-D keys/values | 133,240 |
| Srinath K et al. | Shared replay-adapter; task/domain replay | 127,896 |
| MIITA | Main-paper configuration | 127,123 |
Totals include all method-specific persistent state and serialization metadata, excluding the shared frozen backbone. For per-task LoRA and SAPT-LoRA, all retained task adapters count toward the global cap; SAPT-LoRA additionally includes its selection and reflection state. Appendix D.1 references these configurations.
(2) Task protocols, tuning, and repeated runs. Banking77 follows the official categories.json order, grouped consecutively into seven stages of 11 intents. DSC follows DKVB's ten-domain partition with positive/negative labels, ordered as Tools and Home Improvement, Pet Supplies, Office Products, Kindle Store, Movies and TV, Video Games, Musical Instruments, Sports and Outdoors, Patio Lawn and Garden, and Toys and Games. The SuperNI task order is task748, task073, task1590, task639, task1572, task1687, task591, task363, task1510, task1729, task181, task511, task002, task1290, task875. PAXQA uses EN → AR → ZH → RU. Within each benchmark, all methods share the task partition and order.
Optimized baselines use Adam, learning rate , effective batch size 16, and 10 epochs per stage. We hold out 10% of each training stage for validation, filter configurations by the storage cap, and select by mean validation score across observed tasks and checkpoints. Final runs use the full original training stream and fixed configurations; only the random seed varies. Appendices C.1 and C.2 specify these protocols, including the CIS, InCA, and DKVB search ranges.
W3. Relationship to function-vector-based continual learning.
Jiang et al. [1] provide closely related evidence that injecting source-task function vectors can restore performance after continual tuning, and introduce function-vector-guided (FVG) training with consistency and KL losses. Their function vectors summarize demonstration-induced attention-head activations. MIITA instead derives correction directions from supervised prediction errors and continually consolidates and retrieves them under a fixed storage cap, while keeping the backbone frozen. We have expanded Section 2 to explain this connection and the distinct memory construction and adaptation mechanisms.
We have also added LoRA + FVG to Appendix D.8 by augmenting our sequential LoRA baseline with the two FVG losses. The comparison uses Qwen3-0.6B, the same task streams and training settings, and each dataset's 1% persistent-storage cap, including all retained adapters, function vectors, and other method-specific state. Results are means over three independent runs.
| Method | Banking77 OP ↑ | Banking77 BWT ↑ | SuperNI OP ↑ | SuperNI BWT ↑ |
|---|---|---|---|---|
| Sequential LoRA | 65.31 | -11.44 | 31.24 | -8.91 |
| LoRA + FVG | 69.52 | -9.14 | 33.79 | -7.21 |
| MIITA | 77.67 | -6.17 | 40.58 | -4.89 |
Adding FVG improves OP over sequential LoRA by 4.21 and 2.55 points on Banking77 and SuperNI, respectively, with BWT improvements of 2.30 and 1.70 points. MIITA achieves further OP gains of 8.15 and 6.79 points over LoRA + FVG, and BWT gains of 2.97 and 2.32 points, showing higher final task performance and less average forgetting in this storage-constrained comparison.
Appendix D.8 also reports comparisons with other related vector-based methods under the same evaluation protocol and storage budget.
[1] Jiang G, Jiang C, Li Z, et al. Unlocking the power of function vectors for characterizing and mitigating catastrophic forgetting in continual instruction tuning. ICLR 2025.
Response to Reviewer 95DD (Part 3/3): W4-W5 & Q3
W4. Isolating the contribution of correction-direction memory.
The relevant controlled comparison in Figure 4(a) is SemanticMem versus DirectionMem. SemanticMem replaces correction-direction vectors with semantic prototype vectors while keeping the retrieval, aggregation, and temporary hidden-state adaptation procedures unchanged. Under the same storage budget, DirectionMem achieves higher OP and less forgetting than SemanticMem. This comparison supports the benefit of storing supervised correction information when the adaptation mechanism is held fixed.
ExampleMem and SummaryMem provide complementary comparisons with prompt-based textual memory. We have revised Section 4.5 to distinguish these comparisons and clarified SemanticMem's shared adaptation pipeline in Appendix C.3.
W5. Forward transfer to unseen tasks.
We evaluated forward transfer on SuperNI with Qwen3-0.6B under the 1% persistent-storage budget. For each task i = 2, ..., 15, MIITA is evaluated immediately before learning that task, using only memory accumulated from preceding tasks. The frozen backbone is evaluated on the same test sets with the same prompts and decoding settings. FWT is the mean difference between these pre-learning scores and the corresponding frozen-backbone scores.
| Method | Mean pre-learning ROUGE-L ↑ | FWT ↑ |
|---|---|---|
| Frozen backbone | 11.27 | 0.00 |
| MIITA | 12.78 | +1.51 |
Both means cover tasks 2–15. The positive FWT of 1.51 points shows that accumulated correction memory provides a modest average benefit on unseen tasks before receiving their supervision. We have added the protocol and results to Appendix D.10 and summarized the finding in Section 4.2.
Q3. Retrieval configuration and memory repair.
MIITA uses the union in Eq. (8): the top semantic candidates and top directional candidates are combined with duplicates removed, reranked by Eq. (9), and the top prototypes are aggregated. We have corrected Appendix C.4 to describe this procedure.
Both memory and query semantic keys use the final hidden layer before the LM head, with mean pooling over input tokens and normalization. We use , probe scale in Eq. (9), and stability threshold . The directional retrieval branch is enabled when ; otherwise, candidates come from semantic retrieval. The key-layer choice is explicit in Section 3.2.2, and Appendix B.2 now specifies these parameter values.
Memory repair follows Section 3.2.3 and Algorithm 1 and is triggered when a memory write exceeds the storage cap. It first removes an anchor from the most similar pair within a prototype when its cosine is at least , retaining at least one anchor per prototype. If no such pair exists and at least two prototypes remain, it merges the pair with the highest directional cosine using their support counts as weights. If the merged prototype alone exceeds the cap, its most redundant anchors are further pruned until it fits or only one anchor remains. Eviction selects the minimum-value prototype under Eq. (5) when neither pruning nor merging is available. The footprint is rechecked after each repair until the cap is satisfied. Appendix B.2 now details these selection and transition rules.
Thank you for the careful review and constructive suggestions. We welcome further discussion of these results and clarifications.
Response to Reviewer 95DD (Part 1/3): W1 & Q1
We thank Reviewer 95DD for recognizing the value of separating semantic retrieval keys from functional correction values. Your comments helped us strengthen the evaluation of task acquisition, retention, and forward transfer, and improve the reproducibility of our experiments. We address each concern below.
We have added the frozen-backbone scores to Table 1 for both model sizes across all four benchmarks, using the same prompts, test sets, decoding settings, and task metrics as MIITA. The table below summarizes acquisition and retention: Immediate is the mean score immediately after each task is learned (the diagonal mean), Final is the final task-averaged score (OP), and Final gain is MIITA final OP minus Frozen OP.
MIITA retains final gains of 22.69–45.53 points over the frozen backbone. To present the 426 task-stage scores compactly, we have added all eight performance matrices as heatmaps in Appendix D.9, with a reference in Section 4.2. The diagonal shows immediate acquisition, each column tracks the same task through later stages, and the final row shows its retained performance.
For comparison, SAPT-LoRA's mean immediate scores, derived from its reported OP and BWT using Aimm=OP−TT−1BWT, are 80.09 on Banking77 and 42.13 on SuperNI with Qwen3-0.6B. MIITA reaches 82.96 and 45.14, respectively, while its BWT is −6.17/−4.89 versus −7.08/−5.63 for SAPT-LoRA. Thus, these comparisons show higher initial acquisition together with smaller subsequent average declines. Appendix D.9 includes this analysis alongside the full matrices.