acceptodds
Under review as a conference paper at ICLR 2027

GroupRoute: Instruction-Guided Grouping and Routing for Continual Learning

Abstract

Continual instruction tuning requires language models to learn a sequence of tasks specified by natural-language instructions while maintaining performance on earlier tasks. When explicit task identifiers are unavailable at inference, the model must determine the requested operation from the prompt. Whole-prompt representations, however, can entangle the requested operation with input content, confusing tasks with similar inputs or shared source material. We show that instruction representations extracted from the earliest Transformer layers (layer indices 1–3) of frozen language models are sufficient for reliable task identification. On a shared-corpus benchmark, routing from the instruction alone achieves 99.4–99.7% accuracy even when instructions are reworded. Building on these insights, we introduce GroupRoute, an instruction-guided framework that groups related tasks as they arrive, guides training within each group, and routes queries to the corresponding group at inference. Task assignment and inference routing use frozen backbone representations and require no trainable routing parameters or additional training. GroupRoute routes queries that do not sufficiently match an observed task to the frozen backbone, preserving pretrained capabilities on unseen tasks, while routing related queries to learned groups enables positive transfer. We further introduce Instruction-Keyed Continual Learning (InK), which combines instruction-conditioned scaling with a block-diagonal Fisher penalty to learn successive tasks with fixed adapter and consolidation storage within each group. Across the long-sequence benchmark, SuperNI-42, and TRACE, GroupRoute consistently improves performance and reduces forgetting for a range of continual learning methods, including InfLoRA, O-LoRA, DualPrompt, and InK. On SuperNI-42, a single InK adapter already outperforms all compared methods without GroupRoute on two of three backbones, and InK combined with GroupRoute exceeds all evaluated whole-prompt routing baselines on all three.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, continual parameter-efficient fine-tuning, continual instruction tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3vkxOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GroupRoute, which leverages shallow instruction representations from a frozen model to group and route continual learning tasks. It also introduces InK, a method that employs instruction-conditioned adapters and Fisher regularization to mitigate catastrophic forgetting.

Critical strengths
  1. The study empirically confirms that input content within a full prompt can interfere with task recognition.

  2. The router requires no training and can be integrated with various continual learning methods.

  3. The authors conduct experiments that cover three backbone models and three benchmarks, and the single-adapter InK configuration demonstrates strong performance and storage efficiency.

Critical weaknesses
  1. The distinction from closely related work [1] needs to be clearer. The comparison is limited to TF-IDF-based task identification; an end-to-end comparison or a clearer justification of the methodological differences is needed.

  2. The default grouping produces nearly as many groups as tasks (e.g., 35 groups for 42 tasks), making it difficult to separate the benefits of the grouping strategy from those of parameter isolation.

  3. The comparison between InK and EWC (IA³) changes instruction conditioning, parameter count, and the Fisher structure simultaneously. More controlled ablations are needed to isolate the contribution of each component.

Reference

[1] Qiao, Jingyang, et al. "Large continual instruction assistant." arXiv preprint arXiv:2410.10868 (2024).

Questions for authors
  1. With the same instruction-based router, how does task grouping compare with using an independent adapter for each task in terms of performance and storage?

  2. Under a matched parameter budget, what are the individual contributions of instruction conditioning and block-diagonal Fisher regularization?

  3. Do the main improvements hold across multiple random seeds and additional task orders, particularly on SuperNI-42?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3vkx (1/2): Distinction from LCIA & Task Grouping (W1, W2, Q1)

We thank the reviewer for the positive assessment and insightful questions.

W1: distinction from LCIA (Qiao et al., 2025).

We cite [1] in its published ICML 2025 version (LCIA; Qiao et al., 2025) in Section 5 (Related Work), and we will expand Section 5 in the revised manuscript to make the methodological distinctions explicit alongside our Appendix A.3 comparison (Table 6). Specifically, LCIA is designed for multimodal vision-language models (LLaVA-1.5 and Qwen-VL on the CoIN benchmark, with text evaluated only on T5-small in an appendix), so an end-to-end port to our text streams and backbones has no official reference setup, making its TF-IDF instruction matcher the directly comparable component. Beyond the target domain, GroupRoute differs from LCIA in three fundamental methodological ways:

  • Semantic neural representations vs. surface bag-of-words. TF-IDF relies strictly on surface word overlap rather than task semantics. Under LCIA's published 0.90 similarity threshold, TF-IDF fails on 82.2% of the reworded instructions in Appendix A.3 (matching only 17.8%), because simple synonym changes (such as "generate" becoming "produce", Appendix A.2) sharply reduce lexical overlap. Even when we drop that threshold in Table 6 and allow it to pick the closest known task, TF-IDF still drops to 95.94% because shared boilerplate phrasing across distinct tasks (such as "In this task, you are given...") misleads word-level matching. In contrast, GroupRoute extracts early-layer neural keys (layers 1 to 3) that capture the underlying semantic operation directly, retaining 99.38% to 99.69% accuracy under rewording across all three backbones and 47.36 AP end-to-end on SuperNI-42 (vs. 47.37 on original wordings).

  • Preserving pretrained capabilities on unseen tasks. GroupRoute uses a centered representation space (Equation 2 in Section 3.1) to route queries dissimilar to learned tasks to the frozen backbone, preserving general pretrained capabilities after continual training (Table 3). LCIA possesses no mechanism to detect out-of-stream tasks or protect pretrained performance.

  • Model-agnostic framework vs. monolithic method. LCIA is tied to its own specific LoRA+EMA learner. In contrast, GroupRoute is a modular, model-agnostic routing framework that upgrades diverse continual learning paradigms, including low-rank adaptation (InfLoRA, O-LoRA), prompt tuning (DualPrompt), and activation scaling (InK).

W2 + Q1: task grouping versus independent per-task adapters (K = T), and parameter isolation.

Using one independent adapter per task (K = T) with our instruction router breaks down whenever tasks share identical or near-identical instructions, such as TRACE's two NumGLUE math tasks and SuperNI-42's Jeopardy tasks 306–308 (cosine 1.00; Table 9). Because identical instructions produce identical prototypes, the router cannot distinguish them and directs all queries to only one adapter, leaving the other adapters unused and misrouting queries to an adapter not trained for them. GroupRoute resolves this non-identifiability by merging identical-instruction tasks into a shared group adapter; even against the K = T Task-ID oracle (an upper bound on any K = T router, since it receives ground-truth task IDs at test time), GroupRoute + InK matches its accuracy (47.37 vs. 47.90 AP on SuperNI-42; 81.43/81.47 vs. 81.56 on O3/O4) while storing fewer parameters (37.74M vs. 40.74M on SuperNI-42, Table 17).

Regarding parameter isolation, Section 4.2 (Figure 3) already evaluates multi-task sharing under tighter grouping budgets on SuperNI-42: merging the 42 tasks into 13 groups (p50, a 63% reduction in groups) or 6 groups (p10, an 83% reduction) retains 45.59 and 45.14 AP, outperforming every 42-adapter whole-prompt baseline (EPI 40.70, SAPT 36.99, GainLoRA 30.81). Furthermore, at the extreme of zero parameter isolation (K = 1, single-adapter InK in Tables 2 and 15, where all 42 tasks share one 1.94M adapter), our method achieves 43.50 AP on Gemma-2B-it and 47.78 on LLaMA-2-7B, confirming that instruction-guided sharing provides substantial gains independently of group isolation.

(Continued in "Response to Reviewer 3vkx (2/2)" below.)

AuthorsOpenReview ↗

Response to Reviewer 3vkx (2/2): Parameter Budget & Task Order Stability (W3, Q2, Q3)

W3 + Q2: validity of the unconditioned control, matched-budget comparison, and individual contributions.

In standard IA³, each channel has only an unconditioned intercept (ut=a∈Rmu_{t} = a \in \mathbb{R}^{m}); an unconditioned adapter on the same layers cannot use 2d parameters because two unconditioned vectors a₁ + a₂ collapse into a single vector of size d. Conditioning on key k^t\hat k_{t} introduces slope b alongside intercept a (ut=a+b⊙k^tu_{t} = a + b \odot \hat k_{t}, Equation 3). Moving from d to 2d is an inherent feature of linear conditioning rather than an arbitrary capacity increase, making EWC (IA³) at d = 0.97M the natural same-layer control.

Nevertheless, to match and undercut the parameter budget of EWC (IA³), we evaluate single-adapter InK restricted to layer subsets (Table 18; Gemma-2B-it):

Method (K = 1)Adapted layersParamsO3 APO4 APSuperNI APTRACE AP
EWC (IA³)All linear layers0.97M60.2767.0233.1245.85
InKq, k, v, o, down0.76M76.3377.5641.3046.27
InKq, k, v, o, gate0.76M77.0978.8941.8145.92
InKk, v, gate, down1.20M76.7078.9941.3547.66

Even with 22% fewer parameters (0.76M vs. 0.97M), single-adapter InK outperforms EWC (IA³) by +16.1 to +16.8 AP on O3, +10.5 to +11.9 on O4, and +8.2 to +8.7 on SuperNI-42.

The individual contributions of conditioning and the Fisher structure are separated by the chain-rule quadratic identity underlying Equation 4:

[ΔaiΔbi][fs,ifs,ik^s,ifs,ik^s,ifs,ik^s,i2][ΔaiΔbi]=fs,i(Δai+k^s,iΔbi)2\begin{bmatrix} \Delta a_{i} & \Delta b_{i} \end{bmatrix} \begin{bmatrix} f_{s,i} & f_{s,i} \hat k_{s,i} \\\\ f_{s,i} \hat k_{s,i} & f_{s,i} \hat k_{s,i}^{2} \end{bmatrix} \begin{bmatrix} \Delta a_{i} \\\\ \Delta b_{i} \end{bmatrix} = f_{s,i} (\Delta a_{i} + \hat k_{s,i} \Delta b_{i})^{2}

where fs,i=E[(∇uiLs)2]f_{s,i} = \mathbb{E}[(\nabla_{u_{i}} L_{s})^{2}] is the curvature with respect to the effective log-scale.

Instruction conditioning (ut=a+b⊙k^tu_{t} = a + b \odot \hat k_{t}) provides plasticity. In unconditioned EWC (IA³) where b = 0, penalizing fs,i(Δai)2f_{s,i} (\Delta a_{i})^{2} freezes channel i for subsequent tasks. With conditioning, the penalty fs,i(Δai+k^s,iΔbi)2f_{s,i} (\Delta a_{i} + \hat k_{s,i} \Delta b_{i})^{2} instead opens a zero-cost compensating direction (Δai=−k^s,iΔbi\Delta a_{i} = -\hat k_{s,i} \Delta b_{i}) along which task s's scale is preserved (Δus,i=0\Delta u_{s,i} = 0), while a new task t with distinct key k^t,i≠k^s,i\hat k_{t,i} \ne \hat k_{s,i} can freely adjust its effective scale by Δut,i=(k^t,i−k^s,i)Δbi\Delta u_{t,i} = (\hat k_{t,i} - \hat k_{s,i}) \Delta b_{i}.

The 2 × 2 block-diagonal Fisher ensures stability without destroying this plasticity. Its off-diagonal coupling term fs,ik^s,if_{s,i} \hat k_{s,i} is precisely what preserves this compensating direction. Dropping the off-diagonal terms to use a standard diagonal Fisher (fs,i(Δai)2+fs,ik^s,i2(Δbi)2f_{s,i} (\Delta a_{i})^{2} + f_{s,i} \hat k_{s,i}^{2} (\Delta b_{i})^{2}) would penalize Δai\Delta a_{i} and Δbi\Delta b_{i} separately even when they cancel out, freezing both parameters and losing this plasticity.

Q3: task order sensitivity and stability on SuperNI-42.

Instruction grouping and inference routing are computed deterministically from the frozen backbone with zero random-seed variance. On SuperNI-42, 31 of the 35 groups (74% of tasks) are provably invariant to arrival order by construction, because their pairwise cosines with all other 41 tasks fall strictly below τ_join (0.7485). Because each group initializes from the pristine frozen backbone and trains independently, any of the 42! arrival permutations leaves the group assignments, trained weights, routing decisions, and test scores of all 31 groups identical. Arrival order can only affect the four multi-task groups (11 tasks; Table 9), each containing only 2 to 4 closely related tasks (vs. 42 in the full stream), making these short local sequences far less sensitive to task ordering.

This insulating effect is also observed empirically on the 15-task benchmark, where two full permutations are evaluated across three backbones (Tables 2, 15, and 16). While standalone continual learning baselines shift by 10.0 to 22.7 AP between orders (for example, EWC on LLaMA and T5), GroupRoute + InK varies by only 0.04 to 0.98 AP, and GroupRoute + InfLoRA varies by at most 0.24 AP.

Regarding run variance on SuperNI-42, re-running all 15+ baselines across multiple random seeds over the 42-task stream would repeat the largest experimental block of the paper, which is not feasible within the discussion period. However, GroupRoute's margins over existing methods are substantial (+6.7 to +16.6 AP on Gemma, +13.5 to +20.4 AP on LLaMA). These differences are an order of magnitude larger than the minor fluctuations observed across near-replicate configurations (within 0.22 AP across the layer configurations of Table 18, and within 0.57 AP between the p95 and p99 thresholds in Figure 3).

Thanks for your detailed response. They have answered my questions. I have raised my score accordingly

Thank you so much for taking the time to read our detailed responses and for raising your score. Your thoughtful questions genuinely helped us clarify the paper, and we truly appreciate your engagement and support.

Reviewer_5f15OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses task identification and parameter sharing in continual instruction tuning when task identifiers are unavailable. Observing that whole-prompt representations are often dominated by input content, the authors extract representations solely from the instruction using early layers of a frozen backbone. GroupRoute uses these representations to assign tasks to groups (each with separate adaptation parameters) and route inference queries, falling back to the frozen backbone for dissimilar queries. The paper also introduces InK, an instruction-conditioned extension of IA3 with a channelwise block-diagonal Fisher penalty.

Critical strengths
  1. Isolating the instruction from input content is a highly practical approach for routing, supported by sound comparisons in Table 1.

  2. GroupRoute successfully enhances multiple underlying methods (InfLoRA, O-LoRA, DualPrompt), proving the framework is versatile rather than tied to one specific adapter.

Critical weaknesses
  1. Table 1 alters both the representation source and the routing method simultaneously. To isolate the value of the early-layer neural representation, the paper needs direct comparisons within the GroupRoute pipeline against an instruction-only final-layer baseline, an instruction-only EPI, and TF-IDF. TF-IDF already reaches 95.94% (Appendix A.3), suggesting much of the benchmark is tractable through simple string matching. A direct comparison would clarify exactly what the neural representation adds.

  2. The default grouping heavily isolates tasks (e.g., 35 groups for SuperNI-42), meaning some forgetting reduction is simply the result of keeping task parameters separate. To prove the framework successfully discovers beneficial sharing relationships, it must be compared against a baseline that trains one independent adapter per task but routes them using the proposed instruction router. Furthermore, testing random groups matched in size and reporting per-task results for multi-task groups would better validate the similarity metric.

Questions for authors
  1. How does the pipeline perform with instruction-only final-layer representations or TF-IDF matching?

  2. What is the empirical benefit of grouping versus routing to one independent adapter per task?

  3. What are the specific rejection rates for the 0.5 fallback threshold on seen versus unseen tasks?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5f15 (1/2): Instruction Representations & Routing (W1, Q1)

We thank the reviewer for the constructive assessment and thoughtful questions.

W1 + Q1: isolating the early-layer instruction representation against final-layer instructions, instruction-only EPI, and TF-IDF.

  • Instruction-only final-layer representations. Section 2.1 (Figure 2) already sweeps instruction-only representations through all candidate sites and layers from the first to the final layer on the independent PromptSource corpus (928 instruction wordings from 165 tasks). Task-identification retrieval accuracy peaks in layers 1 to 3 and drops substantially by the final layer across all three backbones (71.7% → 66.4% on LLaMA-2-7B, 73.1% → 46.9% on Gemma-2B-it, and 79.8% → 18.8% on T5-large, along the selected projection), while reading at layers 1 to 3 saves 88% to 94% of forward-pass layers. These results already establish the advantage of early layers for task identification before any benchmark is seen, so we keep the selected sites and layers fixed across Table 1 and all downstream experiments to avoid per-benchmark tuning.
  • Table 1 isolation & instruction-only EPI. Within Table 1, Rows 1 and 2 (Instruction vs. Whole prompt at the same selected sites and layers) apply the exact same decision rule (Equation 1), isolating the representation source directly (99.4% to 99.7% vs. 57.4% to 72.5% under rewording). By contrast, EPI routes on whole prompts using Mahalanobis distance, which requires a within-task covariance matrix Σ across training examples; if EPI is given the instruction only, every training example of a task has the exact same instruction text, so Σ = 0 and Mahalanobis distance degenerates.
  • TF-IDF matching. Under LCIA's published rule (Qiao et al., 2025), which requires a 0.90 similarity threshold to match an existing task, TF-IDF matches only 17.8% of the reworded instructions in Appendix A.3 (failing on 82.2%); it reaches 95.94% in Table 6 (still below the 99.38% to 99.69% of early-layer neural keys) only when that threshold is removed so every query picks the closest of the 40 known tasks. Within the GroupRoute pipeline, however, an unthresholded closest-task rule cannot be used: GroupRoute requires thresholded similarity in a fixed, centered space for online grouping (τ_join) and unseen-task rejection (the 0.5 boundary in Equation 2 and Table 3), while InK (ut=a+b⊙k^tu_{t} = a + b \odot \hat k_{t}, Equations 3 and 4) scales hidden activations at each adapted layer and requires keys k^t∈Rm\hat k_{t} \in \mathbb{R}^{m} matching that layer's channel dimension m.

(Continued in "Response to Reviewer 5f15 (2/2)" below.)

AuthorsOpenReview ↗

Response to Reviewer 5f15 (2/2): Task Grouping & Fallback Boundary (W2, Q2, Q3)

W2 + Q2: grouping versus one independent adapter per task (K = T), multi-task sharing, and random grouping.

  • Grouping vs. K = T independent adapters: On SuperNI-42 at default p99, 31 distinct tasks already form single-task groups identical to independent adapters, while the remaining 11 tasks merge into four multi-task groups of identical or closely related instructions (Table 9, e.g., Jeopardy 306–308 at cosine 1.00; similarly TRACE's two NumGLUE math tasks in Appendix B). Training K = T independent adapters with our instruction router fails on these identical-instruction tasks: because their prototypes are identical, the router cannot tell their queries apart and sends them all to one task's adapter (never trained on the other tasks' data). Merging tasks with cos ≥ τ_join into a shared group adapter resolves this collision and absorbs the few reworded misses in Table 1 within their group. Moreover, even compared against the K = T Task-ID oracle (an upper bound on any K = T router, since it receives ground-truth task IDs at test time), GroupRoute + InK matches its AP across all three backbones (47.37 vs. 47.90 on SuperNI-42 and 81.43/81.47 vs. 81.56 on O3/O4 in Table 2; Tables 15 and 16 for LLaMA-2-7B and T5-large) while keeping forgetting near zero (0.00 to 0.56 across O3, O4, and SuperNI-42) and storing fewer parameters (37.74M vs. 40.74M, Table 17), confirming that the merged multi-task groups (Tables 8 and 9) maintain oracle-level accuracy.

  • Broader multi-task sharing (Figure 3): The gains do not rely solely on 35-group parameter isolation. In Section 4.2 (Figure 3), lowering τ_join on SuperNI-42 compresses the 42 tasks from 35 groups down to 13 groups (p50, a 63% reduction) or 6 groups (p10, an 83% reduction, ~7 tasks per group) while retaining 45.59 and 45.14 AP. Even at the extreme of a single shared adapter for all 42 tasks with zero group isolation (K = 1, single-adapter InK in Tables 2 and 15; 1.94M parameters on Gemma), our method reaches 43.50 AP on Gemma and 47.78 on LLaMA, outperforming every 42-adapter whole-prompt routing baseline (40.70 AP and below).

  • Random grouping: In GroupRoute, grouping and routing both rely on instruction similarity: tasks with similar instructions (cos ≥ τ_join) are placed in the same group during training, so at test time a query can be routed by matching its instruction. If tasks are instead grouped at random, there is no way at test time to tell which group a query belongs to. Even if we form random groups of the same sizes during training and use our instruction router at test time, tasks that share the exact same instruction (such as 306–308 on SuperNI-42 or numglue_cm/ds on TRACE) are placed into different groups. Because their instructions are identical, at test time the router cannot tell them apart and sends all of their queries to just one task's adapter (which was never trained on the other tasks), whereas GroupRoute merges them and achieves 100% group routing accuracy across all 42 tasks and 336 rewordings.

Q3: rejection rates for the 0.5 fallback boundary on seen versus unseen tasks.

The 0.5 fallback boundary is active across all GroupRoute evaluations (not hand-tuned: derived in Equation 2 from ∣q−kt∣2<∣q−0∣2\\|q - k_{t}\\|^{2} < \\|q - \mathbf{0}\\|^{2}, where the origin 0 is the generic instruction mean after centering):

  • Seen tasks: 0.00% rejection across all benchmarks (O3, O4, SuperNI-42, and TRACE), both on original instructions (cosine 1.00) and across all 320 reworded instructions on all three backbones (median cosine 0.86 to 0.88).
  • Unseen tasks after TRACE training (Table 3): 100.0% rejection to the frozen backbone on BoolQ, PIQA, MMLU, and GSM8K (preserving pretrained scores identically); 96.16% on BBH (26 of 27 subtasks rejected; only multistep_arithmetic_two, 3.84%, exceeds 0.5 by matching TRACE's NumGLUE math instruction); and 93.04% on TyDiQA (all 10 non-English languages rejected, while English reading-comprehension queries, 6.96%, match TRACE's English ScienceQA instruction and gain +1.27 to +2.51 F1 from positive transfer).

Thanks for your rebuttal. You have addressed most of my concerns.

Thank you so much for taking the time to read our rebuttal and for your kind follow-up. We truly appreciate your constructive review and thoughtful questions, which helped us clarify key aspects of our work.

Reviewer_kuFpOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a method in the continual instruction tuning setting that combines two lines of work that have been applied to this setting: one based on clever regularized continual finetuning (eg. InfLoRA, O-LoRA, etc.) and another based on task-based routing (eg. EPI, SAPT). Specifically, it maintains online groups of tasks based on similarity in the corresponding task instructions, and routes subsequent received tasks to their corresponding groups (with a fallback of routing them to original backbone model in case of insufficient similarity to any). It attempts to group tasks together by similarity of the pure semantics of the instructions, attempting to ignore similarities in surface form of the instruction or the subject matter the task is based on. It does this by extracting task representations using the early layer activations (from a frozen LLM) over the instruction tokens of the task text, and comparing these by cosine similarity. The paper’s method "GroupRoute" maintains a small number of group-specific trainable parameters using the InK layers that scale the backbone model’s activations based on the outcome of the routing. The set of groups the framework maintains is set during training, and while the group prototype vectors can update at inference time, the set of groups cannot.

Critical strengths
  1. The paper proposes a simple methodology that introduces a small number of additional parameters per group that operate complementarily to existing regularized continual-finetuning based methods and shows both improved performance and reduced forgetting compared to a broad range of standard continual learning baselines.
  2. It also reports results over multiple sets of tasks (SuperNatural Instructions, TRACE, ProgressivePrompts), models (Gemma-2B, and LLaMA-2-7B, T5-large-encoder in appendices), and compares against several continual learning baselines.
  3. The paper reports a few interesting analyses including ablations (tables 2, 3) and effect of hyperparameter choice (Figure 3, Table 18, etc.)
  4. The work appears reproducible with many implementation details appearing in the appendices.
Critical weaknesses
  1. I am not entirely convinced about the novelty of this work. It does show through ablations that the method provides lift over InfLoRA, O-LoRA, etc. but the underlying LoRA-based techniques seem to be responsible for the majority of the gains over the naive Joint Training and Sequential Fine-tuning baselines. GroupRoute certainly works, but it is arguably only a small engineering advance that combines previous orthogonal ideas (regularized continual finetuning and gating) in a very obvious manner.

  2. It is unclear that the insight about segregating tasks with similar semantically instructions together is what is responsible for the success of GroupRoute. It is possible that the reduced load in terms of the number of tasks serviced by each activation scale region is what is responsible. Making this connection requires careful additional analysis (eg. comparing to a version of GroupRoute that randomly segregates tasks into groups of roughly equal sizes).

Questions for authors
  1. What is the practical significance of continual instruction tuning setting? Is there a realistic scenario in which one may receive a sequence of task specifications with the instruction snippet specifically pointed out and separated from the rest of the task description?

  2. Do we know that grouping tasks together by instruction modulo surface form is the right thing to do for optimal continual task learning? Isn’t it also possible that grouping tasks together by surface form/content they are based on may deliver good cross-task transfer? Are the empirical results in this paper sufficient to justify the focus on the former? I would also be interested to see how randomly segregating the task into groups performs (I am unsure how to set up such a measurement, but doing this fairly may involve changing the definition of k^s\hat{k}_s in InK to something independent of the instructions).

  3. Do you have any measurements of how the performance of GroupRoute (and other baselines) scales as a) the number of tasks encountered during training time becomes very large, and b) the number of tasks encountered during inference time (under a fixed set of groups) scales very large? Plotted curves showing this type of degradation would be informative about the limits of these methods.

4 comments
AuthorsOpenReview ↗

Response to Reviewer kuFp (1/2): Novelty & Practical Setting (W1, Q1)

We thank the reviewer for the detailed review and constructive questions.

W1: novelty and the source of gains.

We respectfully clarify a key point about our contributions. This paper introduces two distinct methods, not just a routing wrapper over prior LoRA techniques: (i) InK (Section 3.2), a new IA³-based continual learner that learns successive tasks in a single fixed-size adapter by conditioning activation scaling on the instruction with a Fisher penalty, using neither LoRA nor routing; and (ii) GroupRoute (Section 3.1), a general online instruction-guided grouping and routing framework that works with InK and also upgrades existing continual learners (InfLoRA, O-LoRA, DualPrompt).

Even on its own (using a single 1.94M adapter across all tasks, without LoRA or routing), InK already performs strongly: it raises Seq-IA³ from 13.30/10.51 to 75.19/77.45 AP on O3/O4 (Table 2) and from 29.75 to 43.50 AP on Gemma SuperNI-42 (47.78 on LLaMA, Table 15). That single shared adapter already outperforms every standalone baseline on SuperNI-42, including EPI (40.70), SAPT (36.99), InfLoRA (31.15), and O-LoRA (24.54), while using 20× fewer parameters than InfLoRA and O-LoRA (38.64M).

GroupRoute then provides large additional gains rather than a minor lift. Paired with InK, it achieves the highest AP on every benchmark in Table 2 (81.43/81.47 on O3/O4, 47.37 on SuperNI-42, 53.29 on TRACE). Applied to prior LoRA methods, it raises O-LoRA by +20.8 to +55.1 AP across all Gemma benchmarks (+24.2 to +75.8 on LLaMA) and raises InfLoRA on SuperNI-42 by +15.6 AP (31.15 → 46.74, cutting forgetting from 15.45 to 0.42).

Methodologically, GroupRoute is also not a simple combination of existing gating and regularized learners. Prior methods already combine continual learners with routing or gating (EPI, SAPT, GainLoRA), and our design addresses three specific limitations of existing methods:

  • Routing on the instruction rather than the whole prompt. Whole-prompt routers (such as EPI) entangle the task instruction with the input document and break down when different tasks share source texts (78.5% of SuperNI, Table 1). Isolating the instruction captures the requested operation directly, yielding a training-free router that achieves 100% task identification on original instructions and 99.4% to 99.7% under rewording (Table 1).
  • Extracting representations from the earliest layers rather than the final layer. Prior routers such as EPI follow the standard convention of using final-layer representations, where LLM hidden states increasingly specialize for next-token generation rather than task identity (Section 2.1). Across all three backbones (Figure 2), we show that instruction representations in the earliest layers (layers 1 to 3, under 12% of network depth) identify tasks substantially more accurately than final-layer representations while saving 88% to 94% of forward-pass layers.
  • Preserving pretrained capabilities on unseen tasks. Existing whole-prompt methods (EPI, SAPT, GainLoRA) apply their learned adapters to every query and degrade pretrained capabilities on out-of-stream queries (-2.50 to -14.59 ΔGA, Table 3). By centering instruction keys on a generic instruction mean (mapping the generic mean to the origin), GroupRoute derives a parameter-free fallback boundary (cos > 0.5, Equation 2) that routes unfamiliar queries to the frozen backbone and preserves general capabilities (+0.14 to +0.35 ΔGA, Table 3).

Q1: practical significance of continual instruction tuning and separating the instruction from the input instance.

Continual instruction tuning is an established setting (Yin et al., 2022; Scialom et al., 2022; Wang et al., 2023b; Zhao et al., 2024) that models a deployed LLM adapting sequentially as new tasks arrive, when joint retraining on all past tasks is too costly or past data cannot be retained due to privacy or licensing constraints, and where test queries arrive without task IDs. Within this setting, our method does not require pointing out a sub-snippet inside a task description; rather, the instruction is the full task description specifying the operation and output format, separated only from the input text (the specific document, passage, or sentence to be processed; Figure 1a, Section 2.1). In practice, these two parts are already separate: in LLM APIs and production pipelines, the task instruction is passed as the system prompt or template while the document is the user input; in standard datasets like SuperNI, each task's Definition is stored separately from each example's Input; and even in concatenated prompts, the instruction is simply the shared prefix preceding the input marker (Passage: or Input:).

(Continued in "Response to Reviewer kuFp (2/2)" below.)

AuthorsOpenReview ↗

Response to Reviewer kuFp (2/2): Instruction Grouping & Task Scaling (W2, Q2, Q3)

W2 + Q2: reduced task load, content-based grouping, and random grouping.

  • Reduced task load: The gain is not only due to per-group parameter isolation. In fact, GroupRoute + InK has fewer than one adapter per task (35 groups for 42 tasks on SuperNI-42, Table 7), whereas EPI, SAPT, and GainLoRA already allocate one full adapter per task (K = T = 42, the minimum possible load of 1 task/adapter). Yet GroupRoute + InK beats EPI, the strongest 42-adapter baseline, by +6.7 AP on Gemma SuperNI-42 and by +13.1 to +29.4 AP on every LLaMA benchmark (Tables 2 and 15). Moreover, even when we increase adapter load by lowering τ_join (Figure 3) to merge the 42 tasks into 13 groups (45.59 AP) or 6 groups (45.14 AP, ~7 tasks/adapter), or into one shared adapter (K = 1, single-adapter InK: 43.50 AP on Gemma and 47.78 on LLaMA), our method still outperforms every 42-adapter baseline. Reduced load alone only helps when routing is accurate; the gains come from semantic instruction routing and InK's instruction-conditioned sharing.

  • Grouping by input content or surface form (Q2): When tasks share a source corpus (each of the 42 tasks in SuperNI-42 shares one with at least one other task, Table 4), different tasks run incompatible operations on the same text (e.g., title generation vs. sentiment classification); grouping and routing by input content would merge these conflicting tasks and cause severe misrouting (47.5% to 65.6% whole-prompt accuracy on Gemma and LLaMA, Table 1), whereas instruction similarity only merges compatible tasks (e.g., TRACE's two NumGLUE math tasks or SuperNI-42's four Jeopardy tasks; Tables 8 and 9). Furthermore, online grouping requires a similarity threshold (τ_join) to decide which tasks share a group: with instructions, τ_join is calibrated once on a generic pool of 928 PromptSource wordings (Section 3.1), whereas whole-prompt or content representations depend on each benchmark's domain and length and offer no comparable way to derive τ_join.

  • Random grouping: In GroupRoute, grouping and routing both rely on instruction similarity: tasks with similar instructions (cos ≥ τ_join) are placed in the same group during training, so at test time a query can be routed to the group with the most similar instruction. If tasks are instead grouped at random, there is no way at test time to tell which group a query belongs to. Even if we form random groups of the same sizes during training and use our instruction router at test time, tasks that share the exact same instruction (such as 306–308 on SuperNI-42 or numglue_cm/ds on TRACE) are placed into different groups. Because their instructions are identical, at test time the router cannot tell them apart and sends all of their queries to just one task's adapter (which was never trained on the other tasks), whereas GroupRoute merges them and achieves 100% group routing accuracy across all 42 tasks and 336 rewordings.

Q3: scaling with the number of tasks during training and at inference time.

Unlike vision continual learning, where benchmarks often split one dataset into 50 or 100 small classification tasks, each task in continual instruction tuning is a full natural-language dataset trained with a large language model. Standard benchmarks in continual instruction tuning evaluate 8 tasks (TRACE) or 15 tasks (the long sequence of Progressive Prompts, O-LoRA, SAPT, and GainLoRA, whereas EPI's longest stream has 7 tasks). To evaluate scaling to much longer sequences, we constructed SuperNI-42 (42 tasks), which to our best knowledge is the longest task sequence evaluated in continual instruction tuning. As the stream scales from 8 and 15 tasks to 42 tasks, standalone baselines suffer from subspace saturation and rising forgetting (14.7 to 26.8 forgetting on LLaMA SuperNI-42) and whole-prompt routers face increasing content collisions, whereas GroupRoute + InK keeps forgetting near zero (0.41 on Gemma, 0.56 on LLaMA) and achieves its widest lead over baselines on the 42-task stream (Tables 2 and 15).

Question 3(b) appears to stem from a premise stated in the review summary, namely that "while the group prototype vectors can update at inference time, the set of groups cannot." We clarify that inference in GroupRoute is strictly read-only: neither the group set nor the task prototype vectors ever update at inference time, and no new tasks are merged into the groups during inference. At test time, a query whose best-matching prototype has cos > 0.5 is routed to that task's frozen group adapter, and any query with cos ≤ 0.5 is routed to the frozen pretrained backbone (Table 3). Because inference never modifies any prototype or adapter state, encountering additional tasks or queries at inference time causes no state accumulation and no degradation of the learned groups.

Thank you for your response. Thank you for clarifying misunderstandings I had about your work. I apologize for my confusion, but I think this also means the paper needs to be updated to make some of these points clearer to the reader and to make the main contributions, novelty and claims clearer. I am maintaining my rating but lowering my confidence in my assessment.

Thank you for taking the time to read our response and for your follow-up. We genuinely appreciate your feedback on presentation. In the submitted manuscript, we describe our two contributions (GroupRoute and InK) and explicitly note InK's IA³-based design in the Abstract, Section 1 (lines 94–119), Section 3.2, and Table 2, and we describe inference routing in Section 3.1. Since our goal is to make these distinctions unmistakable to every reader, could you let us know if there are specific statements or passages that felt confusing or seemed to suggest otherwise? Any concrete pointers would be very helpful as we refine the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.