GroupRoute: Instruction-Guided Grouping and Routing for Continual Learning

Continual instruction tuning requires language models to learn a sequence of tasks specified by natural-language instructions while maintaining performance on earlier tasks. When explicit task identifiers are unavailable at inference, the model must determine the requested operation from the prompt. Whole-prompt representations, however, can entangle the requested operation with input content, confusing tasks with similar inputs or shared source material. We show that instruction representations extracted from the earliest Transformer layers (layer indices 1–3) of frozen language models are sufficient for reliable task identification. On a shared-corpus benchmark, routing from the instruction alone achieves 99.4–99.7% accuracy even when instructions are reworded. Building on these insights, we introduce GroupRoute, an instruction-guided framework that groups related tasks as they arrive, guides training within each group, and routes queries to the corresponding group at inference. Task assignment and inference routing use frozen backbone representations and require no trainable routing parameters or additional training. GroupRoute routes queries that do not sufficiently match an observed task to the frozen backbone, preserving pretrained capabilities on unseen tasks, while routing related queries to learned groups enables positive transfer. We further introduce Instruction-Keyed Continual Learning (InK), which combines instruction-conditioned ${\text{IA}}^3$ scaling with a block-diagonal Fisher penalty to learn successive tasks with fixed adapter and consolidation storage within each group. Across the long-sequence benchmark, SuperNI-42, and TRACE, GroupRoute consistently improves performance and reduces forgetting for a range of continual learning methods, including InfLoRA, O-LoRA, DualPrompt, and InK. On SuperNI-42, a single InK adapter already outperforms all compared methods without GroupRoute on two of three backbones, and InK combined with GroupRoute exceeds all evaluated whole-prompt routing baselines on all three.