- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GroupRoute, which leverages shallow instruction representations from a frozen model to group and route continual learning tasks. It also introduces InK, a method that employs instruction-conditioned adapters and Fisher regularization to mitigate catastrophic forgetting.
Critical strengths
-
The study empirically confirms that input content within a full prompt can interfere with task recognition.
-
The router requires no training and can be integrated with various continual learning methods.
-
The authors conduct experiments that cover three backbone models and three benchmarks, and the single-adapter InK configuration demonstrates strong performance and storage efficiency.
Critical weaknesses
-
The distinction from closely related work [1] needs to be clearer. The comparison is limited to TF-IDF-based task identification; an end-to-end comparison or a clearer justification of the methodological differences is needed.
-
The default grouping produces nearly as many groups as tasks (e.g., 35 groups for 42 tasks), making it difficult to separate the benefits of the grouping strategy from those of parameter isolation.
-
The comparison between InK and EWC (IA³) changes instruction conditioning, parameter count, and the Fisher structure simultaneously. More controlled ablations are needed to isolate the contribution of each component.
Reference
[1] Qiao, Jingyang, et al. "Large continual instruction assistant." arXiv preprint arXiv:2410.10868 (2024).
Questions for authors
-
With the same instruction-based router, how does task grouping compare with using an independent adapter for each task in terms of performance and storage?
-
Under a matched parameter budget, what are the individual contributions of instruction conditioning and block-diagonal Fisher regularization?
-
Do the main improvements hold across multiple random seeds and additional task orders, particularly on SuperNI-42?
4 comments
Response to Reviewer 3vkx (2/2): Parameter Budget & Task Order Stability (W3, Q2, Q3)
W3 + Q2: validity of the unconditioned control, matched-budget comparison, and individual contributions.
In standard IA³, each channel has only an unconditioned intercept (); an unconditioned adapter on the same layers cannot use 2d parameters because two unconditioned vectors a₁ + a₂ collapse into a single vector of size d. Conditioning on key introduces slope b alongside intercept a (, Equation 3). Moving from d to 2d is an inherent feature of linear conditioning rather than an arbitrary capacity increase, making EWC (IA³) at d = 0.97M the natural same-layer control.
Nevertheless, to match and undercut the parameter budget of EWC (IA³), we evaluate single-adapter InK restricted to layer subsets (Table 18; Gemma-2B-it):
| Method (K = 1) | Adapted layers | Params | O3 AP | O4 AP | SuperNI AP | TRACE AP |
|---|---|---|---|---|---|---|
| EWC (IA³) | All linear layers | 0.97M | 60.27 | 67.02 | 33.12 | 45.85 |
| InK | q, k, v, o, down | 0.76M | 76.33 | 77.56 | 41.30 | 46.27 |
| InK | q, k, v, o, gate | 0.76M | 77.09 | 78.89 | 41.81 | 45.92 |
| InK | k, v, gate, down | 1.20M | 76.70 | 78.99 | 41.35 | 47.66 |
Even with 22% fewer parameters (0.76M vs. 0.97M), single-adapter InK outperforms EWC (IA³) by +16.1 to +16.8 AP on O3, +10.5 to +11.9 on O4, and +8.2 to +8.7 on SuperNI-42.
The individual contributions of conditioning and the Fisher structure are separated by the chain-rule quadratic identity underlying Equation 4:
where is the curvature with respect to the effective log-scale.
Instruction conditioning () provides plasticity. In unconditioned EWC (IA³) where b = 0, penalizing freezes channel i for subsequent tasks. With conditioning, the penalty instead opens a zero-cost compensating direction () along which task s's scale is preserved (), while a new task t with distinct key can freely adjust its effective scale by .
The 2 × 2 block-diagonal Fisher ensures stability without destroying this plasticity. Its off-diagonal coupling term is precisely what preserves this compensating direction. Dropping the off-diagonal terms to use a standard diagonal Fisher () would penalize and separately even when they cancel out, freezing both parameters and losing this plasticity.
Q3: task order sensitivity and stability on SuperNI-42.
Instruction grouping and inference routing are computed deterministically from the frozen backbone with zero random-seed variance. On SuperNI-42, 31 of the 35 groups (74% of tasks) are provably invariant to arrival order by construction, because their pairwise cosines with all other 41 tasks fall strictly below τ_join (0.7485). Because each group initializes from the pristine frozen backbone and trains independently, any of the 42! arrival permutations leaves the group assignments, trained weights, routing decisions, and test scores of all 31 groups identical. Arrival order can only affect the four multi-task groups (11 tasks; Table 9), each containing only 2 to 4 closely related tasks (vs. 42 in the full stream), making these short local sequences far less sensitive to task ordering.
This insulating effect is also observed empirically on the 15-task benchmark, where two full permutations are evaluated across three backbones (Tables 2, 15, and 16). While standalone continual learning baselines shift by 10.0 to 22.7 AP between orders (for example, EWC on LLaMA and T5), GroupRoute + InK varies by only 0.04 to 0.98 AP, and GroupRoute + InfLoRA varies by at most 0.24 AP.
Regarding run variance on SuperNI-42, re-running all 15+ baselines across multiple random seeds over the 42-task stream would repeat the largest experimental block of the paper, which is not feasible within the discussion period. However, GroupRoute's margins over existing methods are substantial (+6.7 to +16.6 AP on Gemma, +13.5 to +20.4 AP on LLaMA). These differences are an order of magnitude larger than the minor fluctuations observed across near-replicate configurations (within 0.22 AP across the layer configurations of Table 18, and within 0.57 AP between the p95 and p99 thresholds in Figure 3).
Thanks for your detailed response. They have answered my questions. I have raised my score accordingly
Thank you so much for taking the time to read our detailed responses and for raising your score. Your thoughtful questions genuinely helped us clarify the paper, and we truly appreciate your engagement and support.
Response to Reviewer 3vkx (1/2): Distinction from LCIA & Task Grouping (W1, W2, Q1)
We thank the reviewer for the positive assessment and insightful questions.
W1: distinction from LCIA (Qiao et al., 2025).
We cite [1] in its published ICML 2025 version (LCIA; Qiao et al., 2025) in Section 5 (Related Work), and we will expand Section 5 in the revised manuscript to make the methodological distinctions explicit alongside our Appendix A.3 comparison (Table 6). Specifically, LCIA is designed for multimodal vision-language models (LLaVA-1.5 and Qwen-VL on the CoIN benchmark, with text evaluated only on T5-small in an appendix), so an end-to-end port to our text streams and backbones has no official reference setup, making its TF-IDF instruction matcher the directly comparable component. Beyond the target domain, GroupRoute differs from LCIA in three fundamental methodological ways:
Semantic neural representations vs. surface bag-of-words. TF-IDF relies strictly on surface word overlap rather than task semantics. Under LCIA's published 0.90 similarity threshold, TF-IDF fails on 82.2% of the reworded instructions in Appendix A.3 (matching only 17.8%), because simple synonym changes (such as "generate" becoming "produce", Appendix A.2) sharply reduce lexical overlap. Even when we drop that threshold in Table 6 and allow it to pick the closest known task, TF-IDF still drops to 95.94% because shared boilerplate phrasing across distinct tasks (such as "In this task, you are given...") misleads word-level matching. In contrast, GroupRoute extracts early-layer neural keys (layers 1 to 3) that capture the underlying semantic operation directly, retaining 99.38% to 99.69% accuracy under rewording across all three backbones and 47.36 AP end-to-end on SuperNI-42 (vs. 47.37 on original wordings).
Preserving pretrained capabilities on unseen tasks. GroupRoute uses a centered representation space (Equation 2 in Section 3.1) to route queries dissimilar to learned tasks to the frozen backbone, preserving general pretrained capabilities after continual training (Table 3). LCIA possesses no mechanism to detect out-of-stream tasks or protect pretrained performance.
Model-agnostic framework vs. monolithic method. LCIA is tied to its own specific LoRA+EMA learner. In contrast, GroupRoute is a modular, model-agnostic routing framework that upgrades diverse continual learning paradigms, including low-rank adaptation (InfLoRA, O-LoRA), prompt tuning (DualPrompt), and activation scaling (InK).
W2 + Q1: task grouping versus independent per-task adapters (K = T), and parameter isolation.
Using one independent adapter per task (K = T) with our instruction router breaks down whenever tasks share identical or near-identical instructions, such as TRACE's two NumGLUE math tasks and SuperNI-42's Jeopardy tasks 306–308 (cosine 1.00; Table 9). Because identical instructions produce identical prototypes, the router cannot distinguish them and directs all queries to only one adapter, leaving the other adapters unused and misrouting queries to an adapter not trained for them. GroupRoute resolves this non-identifiability by merging identical-instruction tasks into a shared group adapter; even against the K = T Task-ID oracle (an upper bound on any K = T router, since it receives ground-truth task IDs at test time), GroupRoute + InK matches its accuracy (47.37 vs. 47.90 AP on SuperNI-42; 81.43/81.47 vs. 81.56 on O3/O4) while storing fewer parameters (37.74M vs. 40.74M on SuperNI-42, Table 17).
Regarding parameter isolation, Section 4.2 (Figure 3) already evaluates multi-task sharing under tighter grouping budgets on SuperNI-42: merging the 42 tasks into 13 groups (p50, a 63% reduction in groups) or 6 groups (p10, an 83% reduction) retains 45.59 and 45.14 AP, outperforming every 42-adapter whole-prompt baseline (EPI 40.70, SAPT 36.99, GainLoRA 30.81). Furthermore, at the extreme of zero parameter isolation (K = 1, single-adapter InK in Tables 2 and 15, where all 42 tasks share one 1.94M adapter), our method achieves 43.50 AP on Gemma-2B-it and 47.78 on LLaMA-2-7B, confirming that instruction-guided sharing provides substantial gains independently of group isolation.
(Continued in "Response to Reviewer 3vkx (2/2)" below.)