acceptodds
Under review as a conference paper at ICLR 2027

How Models Reason Matters: Task Circuit Replay for Continual Instruction Tuning

Abstract

Mitigating remains a major challenge in continual instruction tuning of large language models. Existing methods preserve past knowledge mainly through historical examples, model outputs, or selected internal representations. However, we argue that effective knowledge preservation should go beyond retaining such past information and instead preserve the underlying capabilities acquired from previous tasks. This shifts the focus toward how these capabilities are represented in the model. We argue that how the model computes for a task provides an important characterization of learned capabilities, as task-level computation patterns can consistently support task behavior across different inputs.Based on this perspective, we propose , a continual instruction-tuning framework that represents and preserves learned capabilities through task-relevant circuits. TCR identifies sparse task-level circuits spanning different model layers, and retains their characteristic responses together with a small number of historical examples. When learning subsequent tasks, these examples reactivate the corresponding circuits, whose responses are selectively preserved to maintain the capabilities they support. To facilitate task-specific circuit identification, TCR introduces Task-BOS to explicitly condition model computation. In this way, TCR preserves past knowledge as reusable task-level computation patterns rather than merely revisiting past observations. Experiments on TRACE and SuperNI demonstrate that TCR consistently reduces forgetting while improving average accuracy and backward transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Continual Instruction Tuning, Catastrophic Forgetting, Large Language Models, Mechanistic Interpretability, Circuit Replay

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eLoDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies catastrophic forgetting in continual instruction tuning and proposes Task Circuit Replay (TCR), a framework that aims to preserve previously learned capabilities by maintaining task-relevant internal computations rather than relying solely on historical examples or output-level knowledge distillation.

The proposed method consists of three main components. First, TCR constructs task-specific representations from a frozen pretrained model and injects them into the beginning-of-sequence (BOS) position to establish explicit task-conditioned computation. Second, it identifies sparse task-relevant components through contrastive gradient-based attribution, measuring the influence of task conditioning on model behavior. Third, during subsequent training stages, the method preserves the historical activations of selected components using a circuit-level regularization objective, together with conventional experience replay.

Critical strengths
  1. A coherent approach to preserving task-relevant internal computation. The main strength of this work is its attempt to formulate knowledge preservation at the level of task-conditioned computation. The Task-BOS mechanism provides an explicit conditioning signal, while contrastive attribution and selective activation replay translate this idea into a concrete training framework. Although component-level regularization and representation preservation have been explored in prior continual learning work, their integration with task-specific computational conditioning is a meaningful methodological contribution.

  2. Encouraging experimental performance accompanied by relevant diagnostic analyses. TCR achieves the highest overall performance in five of the six main experimental settings reported in Table 1. On TRACE, the overall performance improvements over the strongest competing baseline are 1.65 points for Qwen3-1.7B and 1.25 points for Qwen3-8B. The component ablations in Table 2 also provide useful evidence. Removing circuit replay decreases overall performance from 56.93 to 56.28, while replacing the selected circuit with random components reduces performance to 55.95.

Critical weaknesses
  1. Insufficient causal validation of the identified circuits. The main circuit selection relies on gradient-based attribution, while the causal interventions in Appendix C.2 use a different identification procedure. Thus, the causal relevance of the circuits selected by TCR remains insufficiently validated.

  2. Limited statistical evidence and efficiency analysis. Some improvements over strong baselines are marginal, and the main results rely on a single training seed. Moreover, the additional storage and computational costs of circuit replay are not sufficiently quantified.

Questions for authors
  1. Causal validation of TCR-selected circuits. Could the authors clarify whether the components used in the historical activation restoration experiment in Appendix C.7 are selected using exactly the same Task-BOS contrastive attribution procedure as Equations (8)–(10)? More importantly, could the authors provide direct activation patching or component ablation results for circuits selected by the full TCR procedure, preferably across multiple tasks or task sequences? Such evidence would help determine whether the main attribution method identifies causally important computation rather than components that are merely sensitive to task conditioning.

  2. Statistical reliability of the reported improvements. Could the authors report results from multiple independent training seeds, including means and standard deviations for OP, BWT, and FGT? This would be particularly important for settings where TCR improves over the strongest baseline by less than one point and for the relatively small contribution of circuit replay observed in Table 2(a). The results would help assess whether the improvements are robust to training randomness.

  3. Storage and computational efficiency. Could the authors quantify the total additional storage required for task prototypes, selected component identities, and reference activations? It would also be useful to report circuit discovery time, training overhead, and inference-time routing costs. If possible, a comparison against experience replay under a matched total storage or computational budget would help establish whether the proposed circuit-level preservation offers a favorable practical trade-off.

  4. Impact of task routing on end-to-end performance. Appendix C.5 reports a strong prototype routing accuracy of 99.87% on TRACE, which is encouraging. Could the authors additionally report the end-to-end task performance under predicted Task-BOS conditions versus oracle task identities, and analyze the effect of incorrect routing decisions? This distinction would help assess the practical robustness of the approach beyond task-identification accuracy alone.

5 comments
AuthorsOpenReview ↗

Response to Reviewer eLoD (1/3)

Thank you for your thoughtful and constructive feedback. We appreciate your recognition of the methodological contribution of integrating task-specific conditioning with circuit-level preservation, as well as your positive assessment of our experimental results and diagnostic analyses. We address each concern below.

Q1. Causal Validation of TCR-Selected Circuits

We thank the reviewer for highlighting the need for a clearer distinction between our causal analyses. We acknowledge that the original manuscript did not clearly clarify the different objectives of Appendices C.2 and C.7.

Specifically, Appendix C.2 investigates the causal relevance of task components using a separate identification procedure, whereas Appendix C.7 examines whether restoring historical circuit activations can recover forgotten capabilities during continual learning. Importantly, the circuits in Appendix C.7 are identified using the TCR attribution-based selection procedure employed in our continual learning experiments, rather than the separate procedure in Appendix C.2. The activation restoration results in Appendix C.7 therefore provide evidence that restoring historical computation in TCR-selected circuits can recover previously learned task performance.

Additional Causal Validation through Component Ablation. To further address the reviewer's concern that the identified components may be merely sensitive to Task-BOS conditioning rather than causally important for task execution, we conduct an additional component ablation experiment. We compare three selection strategies: (1) circuits identified by TCR's contrastive attribution, (2) randomly selected components, and (3) sensitivity-only components identified based solely on their activation sensitivity to Task-BOS, without gradient-based attribution.

Ablation ConditionΔ\Delta Accuracy
TCR-selected circuits−16.80−\textbf{16.80}
Random components−1.95−1.95
Sensitivity-only components−4.69−4.69

These results demonstrate that TCR's attribution procedure identifies components whose removal has a substantially larger impact on task performance than either random components or components selected solely by Task-BOS sensitivity. Together with the historical activation restoration results in Appendix C.7, these complementary interventions provide direct causal evidence that TCR-selected circuits contribute to task-relevant computation, rather than merely responding to Task-BOS conditioning.

Q2. Statistical reliability

We agree that evaluating performance variability across independent training runs is essential. We would also like to emphasize that the primary objective of TCR is to mitigate catastrophic forgetting, for which backward transfer (BWT) and forgetting (FGT) provide important complementary measures. As shown in Table 1, TCR achieves positive BWT across all six experimental settings and the lowest FGT in five settings (tied for the lowest in the remaining one). For example, on TRACE with Qwen3-1.7B, TCR improves BWT from 0.80 to 1.45 and reduces FGT from 1.43 to 0.19 compared with ER. These results demonstrate encouraging improvements in historical knowledge retention, even when the gains in overall performance (OP) are relatively modest.

Due to the computational cost of repeated sequential training, we first prioritize two representative benchmarks, TRACE and SuperNI-5, using Qwen3-1.7B. We conduct three independent training runs for TCR and three competitive baselines (ER, PaRSP, and CER), reporting the mean and standard deviation of OP, BWT, and FGT.

BenchmarkMethodOP ↑BWT ↑FGT ↓
TRACETCR56.86±0.34\textbf{56.86} \pm 0.341.47±0.58\textbf{1.47} \pm 0.580.84±0.25\textbf{0.84} \pm 0.25
ER55.56‾±0.55\underline{55.56} \pm 0.550.03‾±0.60\underline{0.03} \pm 0.601.40‾±0.42\underline{1.40} \pm 0.42
PaRSP51.86±2.6251.86 \pm 2.62−5.99±3.08−5.99 \pm 3.086.25±3.106.25 \pm 3.10
CER46.49±4.1246.49 \pm 4.12−10.91±4.66−10.91 \pm 4.6611.27±5.0111.27 \pm 5.01
SuperNI-5TCR50.02±0.13\textbf{50.02} \pm 0.130.43±0.04\textbf{0.43} \pm 0.040.10±0.01\textbf{0.10} \pm 0.01
ER49.90±0.1649.90 \pm 0.160.38‾±0.09\underline{0.38} \pm 0.090.14‾±0.06\underline{0.14} \pm 0.06
PaRSP50.01‾±0.08\underline{50.01} \pm 0.08−0.16±0.15−0.16 \pm 0.150.30±0.060.30 \pm 0.06
CER49.25±0.2649.25 \pm 0.26−0.50±0.38−0.50 \pm 0.380.85±0.210.85 \pm 0.21

On TRACE, TCR improves OP and BWT over ER by 1.30 and 1.45 percentage points, respectively, while reducing FGT by 0.56 points. On SuperNI-5, although TCR achieves comparable OP to PaRSP, it improves BWT by 0.58 points and reduces FGT by 0.20 points. These results provide further evidence that TCR's improvements in historical knowledge retention are not limited to the original single-seed evaluation.

AuthorsOpenReview ↗

Response to Reviewer eLoD (2/3)

Q3. Storage and computational efficiency

To address this concern, we conduct a comprehensive computational analysis covering task-vector construction, circuit discovery, comparisons with existing discovery methods, scalability, and end-to-end training overhead. All measurements are conducted on a single NVIDIA RTX PRO 6000 Blackwell GPU.

(1) Cost of task-vector construction and circuit discovery.

We first profile the complete offline preprocessing pipeline across five model scales, from Qwen3-0.6B to Qwen3-14B. The measurements use 32 task anchors and 32 prior anchors for task-vector construction, and 32 anchors for circuit discovery, with three independent timing repetitions.

ModelTask vector (s)Discovery + selection (s)Total (s)Peak GPU (GiB)
Qwen3-0.6B1.62±0.0101.62 \pm 0.0109.81±0.0339.81 \pm 0.03311.4211.423.203.20
Qwen3-1.7B1.60±0.0231.60 \pm 0.0239.99±0.1379.99 \pm 0.13711.6011.607.637.63
Qwen3-4B2.30±0.0182.30 \pm 0.01812.90±0.06412.90 \pm 0.06415.2015.2017.1217.12
Qwen3-8B2.89±0.0102.89 \pm 0.01013.59±0.02013.59 \pm 0.02016.4816.4832.8532.85
Qwen3-14B5.20±0.0135.20 \pm 0.01318.25±0.11118.25 \pm 0.11123.4523.4558.2858.28

The results show that TCR's complete offline preprocessing remains relatively inexpensive across the evaluated model sizes. Even at 14B parameters, task-vector construction and circuit discovery, including component selection and reference activation capture, require only 23.45 seconds in this setting.

(2) Comparison with existing circuit discovery methods.

We additionally compare TCR with Edge Attribution Patching (EAP), its integrated-gradients variant (EAP-IG), and Automated Circuit Discovery (ACDC).

All methods are evaluated using the same setting elaborated in Appendix.

ScaleTCR (s)EAP (s)EAP-IG (s)TCR peak (GiB)EAP / EAP-IG peak (GiB)
0.6B6.146.145.955.9520.3520.353.203.205.45/5.455.45 / 5.45
1.7B6.336.336.866.8625.1625.167.637.6312.03/12.0412.03 / 12.04
4B8.058.0514.8014.8060.3660.3617.1217.1229.23/29.2429.23 / 29.24
8B8.378.3722.4522.4591.2491.2432.8532.8549.86/49.8649.86 / 49.86
14B10.8010.8043.0943.09177.38177.3858.2858.2887.46/87.4787.46 / 87.47

At the 14B scale, TCR achieves 3.99× lower runtime than EAP and 16.43× lower runtime than EAP-IG, while reducing peak GPU memory by approximately 33.4% relative to EAP. The runtime advantage becomes more pronounced for the larger models.

For ACDC, the runs are truncated after approximately 732 seconds on models up to 8B without producing complete circuits, while the 14B run encounters an out-of-memory error.

(3) Scalability with the number of anchor examples.

We further investigate how circuit discovery scales with the number of attribution examples. On Qwen3-14B, increasing the number of anchors from 32 to 512 (16×) increases discovery, selection, and reference-capture time from 13.77 seconds to 226.68 seconds (16.46×), while peak GPU memory increases only from 29.53 GiB to 29.97 GiB.

This indicates approximately linear runtime scaling with the number of anchors and limited additional peak memory consumption under the evaluated configuration.

(4) Conditioning and replay overhead.

We additionally profile the computational cost of the replay objectives using single training exsample on Qwen3-1.7B.

Replay objectiveTime (s)Peak GPU (GiB)
ER-LM1.50±0.0581.50 \pm 0.0584.524.52
Task-BOS + LM1.59±0.0251.59 \pm 0.0254.534.53
Task-BOS + LM + logit KD1.65±0.0341.65 \pm 0.0344.724.72
Task-BOS + LM + circuit matching + KD3.56±0.0323.56 \pm 0.0325.055.05

The complete TCR replay objective requires 2.38× the time of standard ER-LM for the measured microbatches, with an additional 0.529 GiB of peak GPU memory.

(5) Additional storage efficiency. For Qwen3-1.7B on SuperNI-5, stored circuit and logit memories occupy approximately 329.19 MiB across five tasks. The task-routing encoder requires approximately 7.68 ms per query over 2,500 evaluated queries.

Overall, these experiments show that TCR enables relatively efficient component-level circuit identification across the evaluated model scales, while also making clear that activation matching introduces non-negligible recurring training overhead. We believe this more comprehensive analysis addresses the reviewer's central concern regarding computational efficiency and practical scalability.

AuthorsOpenReview ↗

Response to Reviewer eLoD (3/3)

Q4. Impact of task routing

We thank the reviewer for this valuable suggestion. To evaluate the impact of task routing on end-to-end performance, we compare oracle task identities with prototype-predicted Task-BOS under identical evaluation settings.

For Task-BOS, the overall performance decreases only from 56.3089 to 56.2977. This result indicates that predicted routing introduces slightly degradation in overall task performance.

We further analyze all evaluation samples and identify 198 incorrect routing decisions, primarily concentrated in Py150 (188), NumGLUE-ds (9), and MeetingBank (1), with no routing errors in the remaining five tasks. These results demonstrate that TCR maintains nearly identical end-to-end performance without requiring oracle task identities, supporting the practical robustness of prototype-based routing.

Reviewer_15R8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TCR, an instruction-tuning framework designed to mitigate catastrophic forgetting. First, TCR constructs a task-specific beginning-of-sequence (task-BOS) representation for each task and introduces a contrastive conditioning loss to encourage the model to effectively utilize these representations. Second, TCR identifies task-relevant model components by comparing two forward passes conditioned on the corresponding task-BOS and a reference BOS, respectively. Finally, TCR integrates these mechanisms into a unified training objective consisting of four components: SFT loss, replay loss, conditioning loss, and circuit loss. Experiments on three benchmarks demonstrate the effectiveness of the proposed approach.

Critical strengths
  1. The paper is well written, clearly organized, and easy to follow.
  2. The motivation is reasonable and well justified.
  3. The experimental evaluation is relatively comprehensive, providing empirical evidence for the effectiveness of the proposed method.
Critical weaknesses
  1. Insufficient analysis of task prototype construction. The construction of task prototypes relies heavily on an external corpus. However, the paper provides limited analysis of how the choice, quality, and distribution of this corpus affect the resulting prototypes and downstream performance. This raises concerns about the robustness of the proposed method to different prototype construction settings.
  2. Limited justification for BOS-based task conditioning. The authors motivate the injection of task representations into the BOS token based on the attention sink phenomenon. However, it remains unclear whether the observed improvements can be attributed to the unique properties of the BOS token or simply to the introduction of explicit task-conditioning signals. A comparison with alternative conditioning strategies or token positions would help disentangle these factors and strengthen the motivation.
  3. Potential confounding factors in the fixed random BOS ablation. The experimental setup of the fixed random BOS ablation is insufficiently described. In particular, it is unclear whether the observed performance gap arises from less effective task conditioning during training or inaccurate task routing during inference. These factors should be disentangled to better establish the contribution of the proposed task-BOS mechanism.
Questions for authors
  1. Qiu et al. [1] investigate the attention sink phenomenon and introduce a gated attention mechanism that substantially mitigates this behavior. This mechanism has been adopted in the Qwen3.5 model family. Given that TCR partially motivates BOS-based conditioning through the attention sink phenomenon, could the authors evaluate TCR on models equipped with gated attention, such as Qwen3.5? Such experiments would help clarify whether the effectiveness of BOS-based conditioning depends on the presence of attention sinks.
  2. Could the authors evaluate TCR on unseen or out-of-distribution (OOD) tasks? The current prototype router appears to always assign an input to one of the previously observed tasks, even when the input belongs to an unseen task. This forced assignment may introduce inappropriate task conditioning and potentially impair generalization. Would introducing a fallback mechanism that retains the original BOS for inputs with low routing confidence help mitigate this issue?

Reference

[1] Qiu et al. Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free. NeurIPS 2025, Best Paper.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 15R8 (1/2)

Thank you for your thoughtful and constructive feedback. We appreciate your positive assessment of the clarity of our presentation, the well-justified motivation, and the comprehensive experimental evaluation. We address each concern below.

Robustness of Task Prototype Construction.

We first want to clarify the different roles of the training data and the external corpus. As stated in Eq. (3), the task prototype is constructed from the task's own training data, we average the centered prompt representations and normalize the result. The external corpus enters only through Eq. (2), as a reference set for estimating the shared center cpriorc_{\text{prior}}, whose subtraction removes task-irrelevant language priors common to all tasks. Thus, the external corpus does not directly provide task-specific information.

To examine the sensitivity to the reference corpus, we construct task prototypes using three distinct corpora (Lima, Wikipedia, and C4). We evaluate the cosine similarity of the resulting prototypes relative to Lima, along with routing accuracy and the proportion of changed routing decisions on test prompts.

Prior corpusPrototype cosine to LimaRouting accuracyRouting labels changed
Lima (reference)1.0097.80%–
Wikipedia0.8297.87%0.21%
C40.8697.87%0.21%

Although different corpora induce some variation in prototype representations, routing decisions remain highly consistent, with only 0.21% of assignments changing relative to Lima. We further evaluate their impact on downstream continual learning by training the full TCR pipeline using each prototype set under identical experimental settings (TRACE, Qwen3-1.7B, and the same replay budget).

Prior corpusOP↑BWT↑FGT↓
Lima56.71‾\underline{56.71}+0.42‾+\underline{0.42}1.42‾\underline{1.42}
Wikipedia56.8156.81+0.66+\textbf{0.66}1.811.81
C456.91\textbf{56.91}+0.08+0.080.86\textbf{0.86}

The resulting overall performance varies by only 0.204 percentage across the three corpora, although some variation remains in forgetting-related metrics. These findings suggest that TCR is reasonably robust to the choice and distribution of the external reference corpus, particularly in terms of prototype-based routing and overall downstream performance.

Generalization to Unseen Tasks.

We appreciate the reviewer's concern that assigning unseen inputs to previously learned tasks may introduce inappropriate conditioning. To investigate this issue, we evaluate TRACE-trained models on two unseen benchmarks, GSM8K and IFEval, neither of which is used during continual training, prototype construction, or threshold calibration.

We compare forced nearest-prototype routing with a confidence-aware fallback that retains the original BOS when routing confidence falls below a threshold τ\tau. We report the equally weighted average of the two benchmark scores.

SettingOOD average
ER17.4617.46
TCR, forced nearest routing18.8718.87
TCR, confidence-aware fallback ($ \tau = 0.95$)20.54‾\underline{20.54}
TCR, conditioning disabled (original BOS)20.60\textbf{20.60}

Results show that continual training degrades OOD performance. Nevertheless, forcing a seen-task condition on unseen inputs is slightly harmful (18.87 vs. 20.60 without conditioning), and confidence-aware fallback recovers this loss (+1.66 over forced routing), reaching the level of disabling conditioning while leaving ID performance essentially unchanged (ID OP 56.318 vs. 56.298).

Our results validate the reviewer's suggestion that confidence-aware fallback can mitigate inappropriate task conditioning on unseen inputs while preserving performance on previously learned tasks. We will incorporate this mechanism and the corresponding evaluation into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 15R8 (2/2)

BOS-Based Conditioning and Attention Sinks.

Our motivation for BOS-based conditioning involves two related but distinct properties: (i) under causal computation, information at the first position can influence subsequent token representations, and (ii) pretrained Transformers often exhibit disproportionate attention to early tokens, commonly associated with attention sinks. Injecting task information at BOS provides a convenient conditioning interface without modifying the model architecture. Importantly, our design does not require the classical attention-sink phenomenon to persist. We investigate this distinction through two additional experiments.

Conditioning position. To determine whether the benefits arise merely from introducing explicit task-conditioning signals, we compare four injection positions, with each configuration trained and evaluated using its corresponding injection strategy.

Injection positionOP↑BWT↑FGT↓
BOS token56.62\textbf{56.62}+2.86+\textbf{2.86}0.56\textbf{0.56}
Prepended token55.3155.31−1.22−1.222.022.02
Second retained token55.9155.91+0.27+0.271.351.35
Last prompt token56.00‾\underline{56.00}+1.35‾+\underline{1.35}0.57‾\underline{0.57}

BOS-based conditioning achieves the best overall performance among the evaluated positions, outperforming alternative placements in both OP and BWT while achieving the lowest FGT. The differences across positions indicate that the placement of the conditioning signal influences downstream continual-learning performance, supporting our choice of BOS as an effective conditioning interface.

Gated attention (Qwen3.5). To investigate whether TCR remains effective when conventional attention sinks are mitigated, we extend our evaluation to Qwen3.5-2B, which incorporates gated attention mechanisms. We compare TCR against experience replay (ER) and two strong continual-learning baselines (CER and PaRSP):

Method (Qwen3.5-2B)OP↑BWT↑FGT↓
TCR59.11\textbf{59.11}+1.08+\textbf{1.08}0.78\textbf{0.78}
ER58.75‾\underline{58.75}−1.15‾-\underline{1.15}1.76‾\underline{1.76}
CER56.1156.11−3.47-3.474.424.42
PaRSP55.5555.55−6.97-6.976.096.09

The initial results show that TCR remains applicable to this architecture, achieving highest overall performance and lowest forgetting compared with three strong baselines. This result suggests that the applicability of BOS-based conditioning extends to architectures incorporating mechanisms that mitigate conventional attention sinks.

Conditioning-only control. We are additionally evaluating a variant that retains Task-BOS conditioning and the matched replay budget but disables circuit replay. This comparison is intended to isolate the contribution of circuit-level preservation from explicit task conditioning alone.

VariantOP↑BWT↑FGT↓
Task-BOS + ER (w/o circuit replay)55.4655.461.221.221.321.32
Full TCR56.30\textbf{56.30}1.74\textbf{1.74}1.02\textbf{1.02}

Enabling circuit replay improves OP by 0.839, BWT by 0.525, and reduces FGT by 0.299. Since both variants share the same task-conditioning mechanism and replay budget, these improvements demonstrate that circuit-level preservation provides additional benefits beyond explicit task conditioning and experience replay.

Taken together, these experiments support BOS as an effective conditioning position, demonstrate the applicability of TCR to architectures incorporating gated attention, and confirm that circuit replay contributes additional improvements beyond task conditioning alone.

Disentangling Task Conditioning and Inference Routing.

We agree that the Fixed Random BOS ablation in Table 2(a) may conflate the effectiveness of training-time task conditioning with potential errors in inference-time routing. To disentangle these factors, we conduct a controlled comparison between semantic Task-BOS and fixed random task vectors, each evaluated with oracle task assignments and prototype-based routing.

Importantly, both conditioning variants use the same semantic prototype router. This decouples the injected task representation from the retrieval mechanism, while oracle routing additionally eliminates routing errors.

Training conditioningOracle routingPrototype routing
Correct Task-BOS56.5156.5156.5156.51
Random Task-BOS56.3156.3156.3056.30

The results show that oracle and prototype-based routing achieve nearly identical performance, indicating that routing errors contribute negligibly to the observed performance gap. Meanwhile, semantic Task-BOS consistently outperforms fixed random vectors under both routing strategies. These findings suggest that the improvement primarily arises from the training-time conditioning mechanism rather than inaccurate inference-time routing.

We look forward to your response and would greatly appreciate any further comments, suggestions, or guidance you may have.

Reviewer_gsfbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Task Circuit Replay (TCR) to reduce forgetting during continual instruction tuning of LLMs. TCR adds task information at the first token (similar to a <bos> token) and uses the resulting activation changes to select internal components. During training, it replays old examples and matches selected activations to stored values. The authors' experiments on two model sizes from the qwen3 family with TRACE and two SuperNI task sequences show strengths of the method.

Critical strengths
  • The method shows empirical gains in 5/6 experimental settings (=least forgetting on these experiments).
  • TCR suggests a relatively cheap way to find the circuits (parameters that are responsible for a certain task or behavior) compared to previous works (https://arxiv.org/pdf/2405.17969) that require iterative ablation procedures. The strategy could make component selection more practical, although the paper provides no direct runtime comparison.
Critical weaknesses

(1) Narrow evaluation: The main experiments cover two sizes of one model family. The central component ablation uses only TRACE and the smaller main model. This scope provides limited evidence for generalization across model families & sizes.

(2) Error Bars are missing: The main results (tab 1,2) lack error bars across training runs, using just a single random seed for the experiments. It is difficult to assess the robustness of the reported gains (especially because some runs only show small improvements)

(3) Cost: I'm curious about the cost of this method while creating the task vector, circuit discovery, conditioning loss, etc, but the paper lacks cost reports for these. I'll continue this discussion in the following section.

Questions for authors
  • Does the component ablation give the same conclusion at 8B? What evidence supports the method’s effectiveness beyond the tested model family? I believe this would strengthen the paper substantially.
  • How do the main results and component ablation vary across independent training seeds? Please report means and SDs or SEs for Tables 1 and 2. Ideally over 2-3 seeds.
  • What time and peak GPU memory do task-vector construction, circuit discovery, and the conditioning loss require? In previous work, the cost of finding the circuit was the main concern in its real-world applicability. **This is my main question. **
6 comments
AuthorsOpenReview ↗

Response to Reviewer gsfb (1/2)

We thank the reviewer for the constructive feedback. We are glad that the reviewer acknowledges the computation efficiency of our circuit discovery method. We address the concerns below:

Q1. Narrow evaluation

We add experiments on two additional model families (Gemma 2 2B IT and Llama 3.2 3B Instruct) and a component ablation on one larger model (Qwen3-8B). We compare TCR with strong baselines (ER, CER, and PaRSP) on two datasets (TRACE and SuperNI-5).

Generalization across model families.

On both datasets, the proposed TCR beats or on par with the competitive baselines. This result shows that the benefit of TCR is not limited to Qwen models but can also generalizes to other model families.

MethodGemma 2 2B IT / SuperNI-5Gemma 2 2B IT / TRACELlama 3.2 3B / SuperNI-5Llama 3.2 3B / TRACE
ER50.7250.7255.9155.9151.20‾\underline{51.20}58.7458.74
CER50.4950.4956.2856.2850.4150.4159.41‾\underline{59.41}
PaRSP51.41\textbf{51.41}59.83\textbf{59.83}50.6050.6055.9155.91
TCR (Ours)51.27‾\underline{51.27}57.43‾\underline{57.43}51.48\textbf{51.48}61.96\textbf{61.96}

8B component ablation.

We additionally reproduce the component ablation study on Qwen3-8B using TRACE. As shown in the Table below, full TCR achieves the highest OP, BWT, and the lowest FGT among the evaluated variants.

Importantly, removing Circuit Replay reduces BWT from +1.04 to −0.69, while replacing the identified circuit with random components reduces BWT to −0.68 and increases forgetting to 1.37. Replacing the task-specific BOS representation with a fixed random vector also degrades performance.

These findings support that preserving historical circuit activations remains beneficial at a larger model scale and the improvements cannot be reproduced simply by matching activations at randomly selected components or using a fixed random BOS representation.

MethodOPBWTFGT
TCR (Full)64.98\textbf{64.98}+1.04+\textbf{1.04}0.76\textbf{0.76}
w/o Circuit Replay64.6464.64−0.69-0.691.141.14
ER63.3363.33−0.44‾-\underline{0.44}1.701.70
Fixed Random BOS64.12‾\underline{64.12}−0.48-0.480.95‾\underline{0.95}
Random Circuit63.3563.35−0.68-0.681.371.37

Taken together, these additional experiments strengthen the empirical evidence for the generalizability of TCR across model families and scales, as well as the contribution of the key design choices.

Q2. Error bars across independent training seeds.

We agree that evaluating performance variability across independent training runs is essential. We would also like to emphasize that the primary objective of TCR is to mitigate catastrophic forgetting, for which backward transfer (BWT) and forgetting (FGT) provide important complementary measures. As shown in Table 1, TCR achieves positive BWT across all six experimental settings and the lowest FGT in five settings (tied for the lowest in the remaining one). For example, on TRACE with Qwen3-1.7B, TCR improves BWT from 0.80 to 1.45 and reduces FGT from 1.43 to 0.19 compared with ER. These results demonstrate encouraging improvements in historical knowledge retention, even when the gains in overall performance (OP) are relatively modest.

Due to the computational cost of repeated sequential training, we first prioritize two representative benchmarks, TRACE and SuperNI-5, using Qwen3-1.7B. We conduct three independent training runs for TCR and three competitive baselines (ER, PaRSP, and CER), reporting the mean and standard deviation of OP, BWT, and FGT.

BenchmarkMethodOP ↑BWT ↑FGT ↓
TRACETCR56.86±0.34\textbf{56.86} \pm 0.341.47±0.58\textbf{1.47} \pm 0.580.84±0.25\textbf{0.84} \pm 0.25
ER55.56‾±0.55\underline{55.56} \pm 0.550.03‾±0.60\underline{0.03} \pm 0.601.40‾±0.42\underline{1.40} \pm 0.42
PaRSP51.86±2.6251.86 \pm 2.62−5.99±3.08−5.99 \pm 3.086.25±3.106.25 \pm 3.10
CER46.49±4.1246.49 \pm 4.12−10.91±4.66−10.91 \pm 4.6611.27±5.0111.27 \pm 5.01
SuperNI-5TCR50.02±0.13\textbf{50.02} \pm 0.130.43±0.04\textbf{0.43} \pm 0.040.10±0.01\textbf{0.10} \pm 0.01
ER49.90±0.1649.90 \pm 0.160.38‾±0.09\underline{0.38} \pm 0.090.14‾±0.06\underline{0.14} \pm 0.06
PaRSP50.01‾±0.08\underline{50.01} \pm 0.08−0.16±0.15−0.16 \pm 0.150.30±0.060.30 \pm 0.06
CER49.25±0.2649.25 \pm 0.26−0.50±0.38−0.50 \pm 0.380.85±0.210.85 \pm 0.21

On TRACE, TCR improves OP and BWT over ER by 1.30 and 1.45 percentage points, respectively, while reducing FGT by 0.56 points. On SuperNI-5, although TCR achieves comparable OP to PaRSP, it improves BWT by 0.58 points and reduces FGT by 0.20 points. These results provide further evidence that TCR's improvements in historical knowledge retention are not limited to the original single-seed evaluation.

AuthorsOpenReview ↗

Response to Reviewer gsfb (2/2)

Q3. Computational Efficiency and Scalability

We thank the reviewer for recognizing the potential computational efficiency of our circuit discovery strategy. We agree that both the cost of identifying task-relevant circuits and the additional overhead introduced during continual learning should be carefully evaluated.

Efficiency of circuit discovery. Unlike prior iterative ablation-based methods, the proposed TCR employs task-conditioned, contrastive gradient attribution to estimate the relevance of candidate components jointly. This avoids performing a separate ablation experiment for each candidate component, making circuit identification more computationally efficient.

We conduct a comprehensive computational analysis covering task-vector construction, circuit discovery, comparisons with existing discovery methods, scalability, and end-to-end training overhead. All measurements are conducted on a single NVIDIA RTX PRO 6000 Blackwell GPU.

(1) Cost of task-vector construction and circuit discovery.

We first profile the complete offline preprocessing pipeline across five model scales, from Qwen3-0.6B to Qwen3-14B with three independent timing repetitions.

ModelTask vector (s)Discovery + selection (s)Total (s)Peak GPU (GiB)
Qwen3-0.6B1.62±0.0101.62 \pm 0.0109.81±0.0339.81 \pm 0.03311.4211.423.203.20
Qwen3-1.7B1.60±0.0231.60 \pm 0.0239.99±0.1379.99 \pm 0.13711.6011.607.637.63
Qwen3-4B2.30±0.0182.30 \pm 0.01812.90±0.06412.90 \pm 0.06415.2015.2017.1217.12
Qwen3-8B2.89±0.0102.89 \pm 0.01013.59±0.02013.59 \pm 0.02016.4816.4832.8532.85
Qwen3-14B5.20±0.0135.20 \pm 0.01318.25±0.11118.25 \pm 0.11123.4523.4558.2858.28

The results show that TCR's complete offline preprocessing remains relatively inexpensive across model sizes. Even at 14B parameters, task-vector construction and circuit discovery, including component selection and reference activation capture, require only 23.45 seconds in this setting.

(2) Comparison with existing circuit discovery methods.

We additionally compare TCR with Edge Attribution Patching (EAP), its integrated-gradients variant (EAP-IG), and Automated Circuit Discovery (ACDC).

ScaleTCR (s)EAP (s)EAP-IG (s)TCR peak (GiB)EAP / EAP-IG peak (GiB)
0.6B6.146.145.955.9520.3520.353.203.205.45/5.455.45 / 5.45
1.7B6.336.336.866.8625.1625.167.637.6312.03/12.0412.03 / 12.04
4B8.058.0514.8014.8060.3660.3617.1217.1229.23/29.2429.23 / 29.24
8B8.378.3722.4522.4591.2491.2432.8532.8549.86/49.8649.86 / 49.86
14B10.8010.8043.0943.09177.38177.3858.2858.2887.46/87.4787.46 / 87.47

At the 14B scale, TCR achieves 3.99× lower runtime than EAP and 16.43× lower runtime than EAP-IG, while reducing peak GPU memory by approximately 33.4% relative to EAP. The runtime advantage becomes more pronounced for the larger models.

For ACDC, the runs are truncated after approximately 732 seconds on models up to 8B without producing complete circuits, while the 14B run encounters an out-of-memory error.

(3) Scalability with the number of anchor examples.

We further investigate how circuit discovery scales with the number of attribution examples. On Qwen3-14B, increasing the number of anchors from 32 to 512 (16×) increases discovery, selection, and reference-capture time from 13.77 seconds to 226.68 seconds (16.46×), while peak GPU memory increases only from 29.53 GiB to 29.97 GiB.

This indicates approximately linear runtime scaling with the number of anchors and limited additional peak memory consumption under the evaluated configuration.

(4) Conditioning and replay overhead.

We additionally profile the computational cost of the replay objectives using single training exsample on Qwen3-1.7B.

Replay objectiveTime (s)Peak GPU (GiB)
ER-LM1.50±0.0581.50 \pm 0.0584.524.52
Task-BOS + LM1.59±0.0251.59 \pm 0.0254.534.53
Task-BOS + LM + logit KD1.65±0.0341.65 \pm 0.0344.724.72
Task-BOS + LM + circuit matching + KD3.56±0.0323.56 \pm 0.0325.055.05

The complete TCR replay objective requires 2.38× the time of standard ER-LM for the measured microbatches, with an additional 0.529 GiB of peak GPU memory.

(5) Additional storage efficiency.

For Qwen3-1.7B on SuperNI-5, stored circuit and logit memories occupy approximately 329.19 MiB across five tasks. The task-routing encoder requires approximately 7.68 ms per query over 2,500 evaluated queries.

Overall, these experiments show that TCR enables relatively efficient component-level circuit identification across the evaluated model scales. We believe this more comprehensive analysis addresses the reviewer's central concern regarding computational efficiency and practical scalability.

AuthorsOpenReview ↗

Additional Experimental Results on Multi-Seed Training & Evaluation

We further conducted additional multi-seed experiments covering the main comparisons and component ablations. Specifically, we evaluated the main methods on Qwen3-8B using TRACE and SuperNI-5, and performed three-seed component ablations on Qwen3-1.7B across both benchmarks. All results below are reported as mean ±\pm standard deviation (SD) across independent training runs.

1. Multi-Seed Main Results on Qwen3-8B

BenchmarkMethodOP ↑BWT ↑FGT ↓
TRACETCR65.59±0.30\textbf{65.59} \pm 0.30−0.68±0.39\textbf{−0.68} \pm 0.391.40±0.15\textbf{1.40} \pm 0.15
ER65.47±0.6865.47 \pm 0.68−0.74‾±0.89\underline{−0.74} \pm 0.891.80‾±0.45\underline{1.80} \pm 0.45
PaRSP65.56‾±0.99\underline{65.56} \pm 0.99−1.77±0.48−1.77 \pm 0.482.03±0.532.03 \pm 0.53
CER56.53±4.2856.53 \pm 4.28−10.97±4.89−10.97 \pm 4.8911.06±4.9211.06 \pm 4.92
SuperNI-5TCR52.09±0.05\textbf{52.09} \pm 0.05−0.15‾±0.04\underline{−0.15} \pm 0.040.43±0.09\textbf{0.43} \pm 0.09
ER51.82±0.2951.82 \pm 0.29−0.31±0.47−0.31 \pm 0.470.70±0.220.70 \pm 0.22
PaRSP51.90‾±0.25\underline{51.90} \pm 0.25−0.87±0.19−0.87 \pm 0.191.05±0.261.05 \pm 0.26
CER51.79±0.6851.79 \pm 0.68−0.11±0.86\textbf{−0.11} \pm 0.860.61‾±0.76\underline{0.61} \pm 0.76

The results provide further evidence for the effectiveness of TCR in mitigating catastrophic forgetting. TCR achieves the lowest mean forgetting on both benchmarks while exhibiting lower FGT variability than ER and PaRSP. These additional experiments reinforce the evidence that TCR can preserve previously learned capabilities without substantially compromising overall performance.

2. Multi-Seed Component Ablations on Qwen3-1.7B

We additionally repeated the component ablation experiments across three independent training seeds to examine whether the contributions of individual design choices remain consistent across runs.

BenchmarkConfigurationOP ↑BWT ↑FGT ↓
TRACEFull TCR56.02±0.81\textbf{56.02} \pm 0.810.71±0.77\textbf{0.71} \pm 0.770.75±0.42\textbf{0.75} \pm 0.42
w/o Circuit55.86±0.3455.86 \pm 0.340.47±0.580.47 \pm 0.580.84±0.250.84 \pm 0.25
Random Circuit55.70±0.6755.70 \pm 0.670.29±0.600.29 \pm 0.600.93±0.540.93 \pm 0.54
Fixed Random BOS55.56±0.0955.56 \pm 0.090.24±0.210.24 \pm 0.211.06±0.181.06 \pm 0.18
ER55.56±0.5555.56 \pm 0.550.02±0.600.02 \pm 0.601.40±0.421.40 \pm 0.42
SuperNI-5Full TCR50.04±0.003\textbf{50.04} \pm 0.0030.57±0.100.57 \pm 0.100.057±0.052\textbf{0.057} \pm 0.052
w/o Circuit49.93±0.1849.93 \pm 0.180.31±0.130.31 \pm 0.130.151±0.0910.151 \pm 0.091
Random Circuit49.97±0.1549.97 \pm 0.150.46±0.090.46 \pm 0.090.060±0.0220.060 \pm 0.022
Fixed Random BOS50.02±0.1350.02 \pm 0.130.43±0.040.43 \pm 0.040.097±0.0120.097 \pm 0.012
ER50.04±0.0950.04 \pm 0.090.65±0.18\textbf{0.65} \pm 0.180.106±0.0980.106 \pm 0.098

The multi-seed ablations further support the contributions of the proposed components. While some component-level differences remain small relative to across-seed variability, the full method consistently achieves the lowest mean FGT across both ablation benchmarks.

Taken together, these additional multi-seed experiments strengthen the empirical evidence for TCR's effectiveness in mitigating catastrophic forgetting. Across all four reported settings, full TCR achieves the lowest average FGT, supporting the overall forgetting-reduction trend observed in our original experiments. The results also provide a clearer characterization of performance variability across independent training runs. We will incorporate these additional results into the revised manuscript and clearly distinguish the observed performance trends from claims of statistical significance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.