acceptodds
Under review as a conference paper at ICLR 2027

RESCUE: Repairing Language Model Errors to Sparse Circuits via Reinforcement Learning

Abstract

Large language models (LLMs) exhibit strong general capabilities that mechanistic interpretability has attributed to sparse computational circuits. However, existing circuit studies emphasize preserving functionality or explaining safety, leaving the mechanisms underlying failures across a broader range of tasks largely unexplored. Extending circuit analysis from abilities to errors, we explore the perspective that such failures may likewise arise from erroneous internal computations and that targeted tuning of the corresponding parameters can correct such errors while largely preserving other capabilities. Motivated by this insight, we introduce **RESCUE** (**R**easoning-**E**rror **S**parse-**C**ircuit **U**ncovering and **E**diting), a framework that localizes error-associated circuits and surgically repairs them for performance enhancement. General tasks typically involve multi-step reasoning and long-form generation, where early deviations can cause prefixes to drift from supervised references, leading SFT-based mask optimization to overlook circuits involved in generation-time errors. RESCUE therefore refines these masks through reinforcement learning with multiple masked-model rollouts, improving their relevance to observed task failures. Finally, RESCUE introduces a pruning technique and precisely fine-tunes error circuits to correct task failures, thereby translating error localization into a sparse and targeted model update. We validate RESCUE on heterogeneous repair sets across two domains: **(1) mathematical reasoning**, identifying a math error circuit of 1.40% density whose repair raises accuracy from ; and **(2) medical QA** where a similarly compact 1.44% circuit improves repair-set accuracy from .

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Interpretability, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NiKfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RESCUE, a method for repairing recurring LLM errors by identifying a small set of MLP projection rows and updating only their weights. It combines supervised mask search, rollout-based RL refinement, circuit pruning, and restricted fine-tuning. Experiments on Qwen3-8B and Llama-3.1-8B-Instruct report strong repair-set improvements and some gains on standard benchmarks while largely preserving non-target performance.

Critical strengths

1.The paper studies a useful problem: targeted error repair with minimal interference to existing capabilities. The large gains from suppressing a small fraction of MLP rows are noteworthy.

2.The evaluation includes two model families, mathematical and medical tasks, random-circuit controls, rephrasing, and sequential repair. The public GSM8K improvements are encouraging.

Critical weaknesses

1.The pattern-specific datasets are generated from shared solution templates, yet Table 1 reports full-set accuracy rather than separate held-out results. Direct transfer to rephrased inputs is also much weaker for several Qwen3-8B patterns (Table 2). The large repair-set gains therefore need more convincing independent evaluation.

2.Finding rows whose removal improves accuracy does not by itself show that they form the error-producing circuit. Also, the supervised-only baseline removes both RL and pruning, so the specific contribution of RL cannot be determined.

3.The baseline comparisons are limited, as RESCUE is not directly compared with existing circuit discovery methods, and the supervised-only baseline does not isolate the contributions of RL refinement and pruning.

Questions for authors

1.Can the authors report held-out repair accuracy separately from training/full-set accuracy, preferably also on different solution templates?

2.Can the authors separate the effects of RL refinement and pruning, and provide stronger evidence that the selected rows represent error-causing computations rather than effective intervention sites?

3.How was LoRA tuned? Its large degradation on Llama-3.1-8B-Instruct raises concerns about baseline fairness.

4.Could the authors provide comparisons with existing circuit discovery methods and separate ablations for RL refinement and pruning?

7 comments
AuthorsOpenReview ↗

[Part I] Responses to Weakness 1

Dear reviewer NiKf,

We sincerely thank the reviewer for the thoughtful and constructive comments. We have carefully considered each suggestion and provide our detailed point-by-point responses below.

For Weakness 1 and Question 1: Training and Held-out Repair Accuracies

Thank you for this comment. We additionally expanded the independent held-out set and evaluated RESCUE on it. The results confirm that RESCUE remains effective on held-out examples, as shown below:

1. Pattern-specific repair (Section 5.3): original model → circuit ablation

ModelError PatternTrain (%)Test (%)
Qwen3-8B-InstructMisleading Premise Back-solving1.3 → 96.32.1 → 93.6
Qwen3-8B-InstructDynamic State Tracking1.3 → 97.50.7 → 90.7
Qwen3-8B-InstructDropped-lowest Average2.5 → 93.80.0 → 95.7
Qwen3-8B-InstructDeclining Group Accumulation0.0 → 81.31.4 → 77.9
Qwen3-8B-InstructMultiplicative Increment Back-solving0.0 → 96.32.1 → 81.4
Llama-3.1-8B-InstructCumulative Target Gap1.3 → 98.80.0 → 94.3
Llama-3.1-8B-InstructScale Conversion26.3 → 95.024.3 → 92.1
Llama-3.1-8B-InstructPercentage Risk6.3 → 98.85.0 → 96.4
Llama-3.1-8B-InstructUnit Price and Total Cost23.8 → 100.023.6 → 96.4
Llama-3.1-8B-InstructFees and Change10.0 → 97.512.9 → 93.6

2. Heterogeneous GSM8K repair (Section 5.2): original model → RESCUE after circuit-restricted tuning

ModelTrain (%)Test (%)
Qwen3-8B-Instruct0.0 → 70.00.0 → 47.9
Llama-3.1-8B-Instruct6.3 → 81.34.3 → 69.3

3. MedMCQA repair (Section 5.6): original model → RESCUE after circuit-restricted tuning

ModelTrain (%)Test (%)
Qwen3-8B-Instruct2.5 → 78.82.9 → 77.9
Llama-3.1-8B-Instruct0.0 → 88.80.0 → 75.7

4. Direct transfer to rephrased unseen problems

We additionally rephrased the expanded independent test set and directly evaluated these examples using the previously localized circuits (Section 5.4).

ModelError PatternOriginal model Acc(%)Circuit ablation Acc(%)
Qwen3-8B-InstructMisleading Premise Back-solving23.643.6
Qwen3-8B-InstructDynamic State Tracking17.125.7
Qwen3-8B-InstructDropped-lowest Average2.919.3
Qwen3-8B-InstructDeclining Group Accumulation1.420.7
Qwen3-8B-InstructMultiplicative Increment Back-solving0.725.0
Llama-3.1-8B-InstructCumulative Target Gap23.660.7
Llama-3.1-8B-InstructScale Conversion30.754.3
Llama-3.1-8B-InstructPercentage Risk27.183.6
Llama-3.1-8B-InstructUnit Price and Total Cost40.787.1
Llama-3.1-8B-InstructFees and Change15.777.9

These held-out results demonstrate that RESCUE effectively repairs reasoning errors, with improvements extending to unseen examples. We will incorporate these results into the revised PDF.

AuthorsOpenReview ↗

[Part II] Responses to Weakness 2 and Question 2 & 3

For Weakness 2 and Question 2: Disentangling the Contributions of RL Refinement and Pruning, and Validating Error-Associated Circuits

1. Disentangling the Contributions of RL Refinement and Pruning

We acknowledge this limitation and have added supplementary ablation experiments to disentangle the contributions of RL refinement and pruning. The table reports GSM8K test accuracy (%) evaluated using lm-eval after the same circuit-restricted tuning procedure:

Localization pipelineQwen3-8B-InstructLlama-3.1-8B-Instruct
SFT87%73%
SFT + pruning88%74%
SFT + RL + pruning (RESCUE)91%77%

Pruning improves accuracy by 1 percentage point on both models, while adding RL refinement yields a further 3-point improvement on each. These results demonstrate the additional contribution of RL to circuit localization and downstream repair. We will incorporate this ablation into the revised PDF.

2. Validating Error-Associated Circuits

  • Intervention-Based Definition of Error-Associated Circuits Prior mechanistic interpretability studies such as SafeSeek [1] and SHIPS [2] have adopted an intervention-based attribution principle: attention heads or neurons whose ablation causes safety-aligned LLMs to exhibit unsafe behaviors are considered safety-related components. We follow the same principle to define error-associated circuits. Specifically, computations genuinely contributing to a reasoning error should exhibit a key property: their targeted suppression should reduce the corresponding errors while preserving correct behavior on clean inputs. The circuits identified by RESCUE satisfy this criterion and further support effective downstream repair.

  • Complementary Evidence from Ablation and Targeted Repair Our experiments provide complementary evidence for the validity of the identified circuits.

    • First, ablating RESCUE-localized circuits substantially reduces reasoning errors, whereas density-matched random circuits yield little improvement. For pattern-specific repair, RESCUE raises accuracy from 0–4% to 76–94% on Qwen3-8B-Instruct and from 1–36% to 92–95% on Llama-3.1-8B-Instruct, while random-circuit controls achieve only 0–13.5% mean accuracy.
    • Second, circuit-restricted tuning based on RESCUE achieves 91% and 77% GSM8K accuracy on the two models, respectively, outperforming validation-tuned LoRA (85% and 73%).

Together, these results show that the identified circuits not only exhibit error-mitigating effects upon suppression but also provide more effective targets for subsequent repair. This supports interpreting RESCUE-localized error-associated circuits as plausible proxies for the underlying computations contributing to reasoning failures.

For Question 3: LoRA Tuning Protocol

We thank the reviewer for this valuable suggestion. We conducted additional validation-based hyperparameter tuning for both models. We searched LoRA ranks {8, 16} and learning rates {1e-5, 5e-5, 1e-4, 2e-4, 3e-4}, resulting in ten configurations per model. Each configuration was trained for up to ten epochs, with evaluation after every epoch. LoRA alpha was set to twice the rank, dropout to 0.05, and adapters were applied to attention and MLP projections.

Checkpoints were selected based on repair validation accuracy, with clean validation accuracy used to break ties. Earlier checkpoints were preferred for exact ties. Across configurations, ties were resolved by preferring smaller ranks and then lower learning rates.

  • Llama GSM8K Degradation:

We evaluated the selected checkpoints on the complete GSM8K test set using lm-eval. The results are as follows:

ModelSelected rankSelected learning rateSelected epochLoRARESCUE (Ours)
Qwen3-8B-Instruct161e-4685%91%
Llama-3.1-8B-Instruct165e-51073%77%

After validation-based tuning, the Llama degradation no longer persists: LoRA achieves 73%, matching the original model (73%). However, it still falls short of RESCUE (77%). RESCUE also outperforms the tuned LoRA baseline on Qwen3-8B-Instruct. We will incorporate these results and the corresponding tuning protocol into the revised PDF.

AuthorsOpenReview ↗

[Part III] Responses to Weakness 3 and Question 4

For Weakness 3 and Question 4: Comparison with Existing Circuit Discovery Methods

Thank you for this suggestion. We agree that comparisons with established mask-based circuit localization methods are important. However, existing methods differ fundamentally from RESCUE in terms of what they localize and intervene on.

  • From Token-Level Attribution to Full-Generation Circuit Localization: Representative methods such as ACDC [3], Edge Pruning [4], and IBCircuit [5] primarily focus on identifying circuits responsible for only one-token prediction (e.g., predicting a binary "True" or "False" label in the Greater Than task). In contrast, RESCUE aims to localize error-associated circuits across the entire autoregressive generation process, addressing errors in complete, multi-step reasoning responses.

  • Limitations of Single-Token Circuit Attribution for Full-Generation Reasoning Repair: Full-generation reasoning repair is substantially more challenging than token-level circuit localization. Since autoregressive generation is sequential and context-dependent. Existing methods are not designed for full-generation intervention and therefore cannot directly address our reasoning-repair task.

Given the above differences, our newly proposed RESCUE framework is inherently difficult to compare directly with existing mask-based circuit localization methods.

Thank you again for your thoughtful and constructive feedback. We will address these points in the revised manuscript. We hope our clarifications and additional results help strengthen your confidence in our work and may merit a more favorable evaluation.

Best,

Authors

[1] SafeSeek: Universal Attribution of Safety Circuits in Language Models (ICML 2026)

[2] On the Role of Attention Heads in Large Language Model Safety (ICLR 2025 Oral)

[3] Towards Automated Circuit Discovery for Mechanistic Interpretability (NeurIPS 2023)

[4] Finding Transformer Circuits with Edge Pruning (NeurIPS 2024)

[5] IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck (ICML 2025)

Reviewer_SHiDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces RESCUE, a framework for identifying and repairing error-associated sparse circuits in LLMs. The method first locates candidate circuits through supervised mask optimization, then refines them using reinforcement learning with feedback from model-generated responses. The identified circuits are subsequently pruned and selectively fine-tuned to correct errors while preserving other capabilities. Experiments on Qwen3-8B and Llama-3.1-8B-Instruct demonstrate promising improvements on mathematical reasoning and medical QA tasks.

Critical strengths

The paper presents an interesting approach to identifying and repairing problematic circuits using RL-guided mask optimization.

The experimental results are encouraging, showing improvements over LoRA and the supervised-only baseline while largely preserving non-target capabilities.

Critical weaknesses

Limited evaluation diversity. The experiments mainly focus on GSM8K and MedMCQA, which do not adequately represent the diverse tasks and failure modes that LLMs encounter in real-world applications. Additional experiments across different task domains would provide stronger evidence for the generalizability of the proposed method.

Insufficient comparisons with related methods. The paper compares RESCUE primarily against LoRA and an internal supervised-only baseline. However, direct comparisons with established mask-based circuit localization methods are missing. Such comparisons are important for understanding whether the proposed approach offers meaningful advantages over existing circuit discovery and intervention techniques.

Questions for authors

Could the authors provide additional experiments on more diverse tasks beyond mathematical reasoning and medical QA to demonstrate the generalizability of RESCUE?

Could the authors compare RESCUE with at least one representative mask-based circuit localization method under comparable experimental settings?

5 comments
AuthorsOpenReview ↗

Responses to Weakness 1 & 2 and Question 1 & 2

Dear reviewer SHiD,

We thank the reviewer for the careful evaluation of our work and the insightful suggestions. We have addressed the comments individually and provide our responses and corresponding revisions below.

For Weakness 1 and Question 1: Cross-Domain Evaluation

We agree that evaluations on mathematical reasoning and medical QA alone provide limited evidence of RESCUE's generalizability. To address this concern, following the experimental setup in Section 5.6, we conducted additional cross-domain experiments on MMLU Professional Law using heterogeneous legal repair sets, comparing RESCUE with direct LoRA fine-tuning. The results are reported below:

ModelMethodRepair test accuracyCircuit density (%)MMLU Professional LawAggregate Evaluation
Qwen3-8B-InstructBase5.0%—50%66.63%
Qwen3-8B-InstructLoRA37.9%—49%67.00%
Qwen3-8B-InstructRESCUE40.0%1.66%52%66.63%
Llama-3.1-8B-InstructBase5.7%—49%62.38%
Llama-3.1-8B-InstructLoRA40.7%—48%61.50%
Llama-3.1-8B-InstructRESCUE45.0%1.55%50%62.50%

RESCUE outperforms LoRA on held-out legal repair accuracy by 2.1 and 4.3 percentage points, respectively, while modifying only 1.66% and 1.55% of eligible MLP rows. It also improves Professional Law accuracy while largely preserving broader capabilities. These results demonstrate the applicability of RESCUE beyond mathematical reasoning and medical QA. We will incorporate these experiments into the revised PDF.

For Weakness 2 and Question 2: Comparison with Existing Mask-Based Circuit Localization Methods

Thank you for this suggestion. We agree that comparisons with established mask-based circuit localization methods are important. However, existing methods differ fundamentally from RESCUE in terms of what they localize and intervene on.

  • From Single-Token Attribution to Full-Generation Attribution and Intervention: Representative methods such as ACDC [1], Edge Pruning [2], and IBCircuit [3] typically identify circuits responsible for only one-token prediction (e.g., predicting a binary "True" or "False" label in the Greater Than task). In contrast, RESCUE localizes and intervenes on error-associated circuits throughout the full autoregressive generation process, targeting errors in complete, multi-step reasoning responses.

  • Limitations of Single-Token Circuit Attribution for Full-Generation Reasoning Repair: Full-generation reasoning repair is substantially more challenging than token-level circuit localization. Because autoregressive generation is sequential and context-dependent, interventions at early generation steps can alter the subsequent reasoning trajectory. Existing methods are not designed for full-generation intervention and therefore cannot directly address our reasoning-repair task.

Given the above differences, our newly proposed RESCUE framework is inherently difficult to compare directly with existing mask-based circuit localization methods.

Thank you again for your thoughtful and constructive feedback. We will incorporate these clarifications and additional results into the revised manuscript. We hope they help address your concerns and would be grateful if you felt that they warrant a more favorable assessment of our work.

Best regards,

Authors

[1] Towards Automated Circuit Discovery for Mechanistic Interpretability (NeurIPS 2023)

[2] Finding Transformer Circuits with Edge Pruning (NeurIPS 2024)

[3] IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck (ICML 2025)

Looks good, can you update the new results and discussions to the submission as a revised version with new contents annotated blue?

Dear Reviewer SHiD,

Thank you again for your positive feedback and helpful suggestion. We have uploaded the revised manuscript, incorporating the additional experimental results and discussions, with all new and revised content highlighted in blue for your convenience.

We would be grateful if you could reconsider your rating in light of these updates and our detailed responses to your concerns. Thank you again for your time and thoughtful evaluation.

Best regards, The Authors

Reviewer_9wWeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RESCUE, a framework for repairing language models by identifying a small subset of MLP rows associated with errors. With model weights frozen, the method learns binary masks using corrected erroneous responses and originally correct responses, then refines the masks through reinforcement learning based on answer correctness and reasoning quality across multiple sampled responses. After pruning, only the selected parameter rows are fine-tuned. Experiments on Qwen3-8B and Llama-3.1-8B-Instruct across mathematical reasoning and medical question answering show that suppressing these rows substantially improves accuracy on the constructed repair sets, while subsequent targeted fine-tuning also improves performance on some public benchmarks.

Critical strengths

The paper shows that removing a small number of MLP rows can correct many errors in the constructed repair sets, even before any model weights are updated. This is an interesting finding because it suggests that some recurring errors can be corrected by changing only a small part of the model. The authors also show that fine-tuning these selected rows can improve performance, giving the analysis a practical use.

The authors test whether the improvements carry over to other questions. The same circuits improve accuracy when the original questions are reworded, without additional training or circuit search. After targeted fine-tuning, both models also perform better on the public GSM8K benchmark. These results are encouraging because they provide evidence that the method can help beyond the examples used for repair.

Critical weaknesses

I find the methodological novelty somewhat limited. Learning a mask to change a small part of a model while preserving its other capabilities is a familiar idea, with related approaches explored in continual learning and hallucination mitigation. The paper brings this idea into an error-circuit framework, but the distinction from earlier methods is not sufficiently clear. A more direct comparison would help establish what the proposed mask learning and RL refinement add beyond existing approaches.

The reported repair accuracy makes it difficult to assess how well the method handles unseen examples. Section 5.1 specifies training–test splits, but Table 1, Table 3, and the sequential repair experiments report “full-set accuracy.” It is unclear how much of the improvement comes from examples used during repair and how much carries over to held-out examples. The authors should clarify what “full-set” includes and report training and test accuracy separately. The gains on public benchmarks are encouraging, but separate held-out results are still needed to assess generalization within the constructed error patterns.

Questions for authors

Could you report separate training and held-out repair accuracies for the central experiments and identify the data used for mask-checkpoint selection, update acceptance, pruning, and tuning-checkpoint selection? Could rephrase transfer also be reported separately for underlying problems unseen during original localization? How many examples were evaluated on each public benchmark, and what uncertainty accompanies the main improvements? How were LoRA rank, learning rate, training duration, and checkpoints selected, and does the large Llama GSM8K degradation persist after validation-based tuning?

5 comments
AuthorsOpenReview ↗

[Part I] Responses to Weakness 1 & 2 and Question 1

Dear reviewer 9wWe,

We greatly appreciate the reviewer's time and valuable feedback. The suggestions have helped us further clarify and strengthen our work. Our detailed responses to each comment are presented below.

For Weakness 1: Methodological Novelty

We agree that learning masks to modify a sparse subset of model parameters while preserving other capabilities is not, by itself, a new idea. However, existing circuit discovery methods differ fundamentally from RESCUE in terms of what they localize and intervene on.

  • RESCUE extends single-token circuit attribution to full generation attribution: Representative methods such as ACDC [1], Edge Pruning [2], and IBCircuit [3] all identify circuits responsible for only one-token prediction (e.g., predicting a binary “True” or “False” label in the Greater Than task). In other words, their attribution is within the single token level. In contrast, RESCUE extends circuit attribution over the full autoregressive generation process (tons of tokens), targeting the correction of errors in long reasoning generations.

  • Limitations of Single-Token Circuit Attribution for Reasoning-Error Attribution: The full-generation reasoning repair setting addressed by RESCUE is substantially more challenging than token-level circuit localization. Existing methods are not designed for full-generation intervention and therefore cannot directly address our reasoning-repair task.

  • The Role of RL Refinement: Prior supervised mask-attribution methods such as SafeSeek [4] rely on fixed references, making them vulnerable to prefix drift and overfitting to reference trajectories. RL refinement leverages outcome feedback from multiple rollouts to identify more generation-relevant and generalizable circuits.

For Weakness 2 and Question 1: Training and Held-out Repair Accuracies and Data Used for Intermediate Decision

  • Training and Held-out Repair Accuracies: Thank you for highlighting this issue. We constructed a separate, expanded held-out set and evaluated RESCUE on it. The results further demonstrate that the repair gains remain effective on held-out data, as shown below:

    1. Pattern-specific repair (Section 5.3): original model → circuit ablation

    ModelError PatternTrain (%)Test (%)
    Qwen3-8B-InstructMisleading Premise Back-solving1.3 → 96.32.1 → 93.6
    Qwen3-8B-InstructDynamic State Tracking1.3 → 97.50.7 → 90.7
    Qwen3-8B-InstructDropped-lowest Average2.5 → 93.80.0 → 95.7
    Qwen3-8B-InstructDeclining Group Accumulation0.0 → 81.31.4 → 77.9
    Qwen3-8B-InstructMultiplicative Increment Back-solving0.0 → 96.32.1 → 81.4
    Llama-3.1-8B-InstructCumulative Target Gap1.3 → 98.80.0 → 94.3
    Llama-3.1-8B-InstructScale Conversion26.3 → 95.024.3 → 92.1
    Llama-3.1-8B-InstructPercentage Risk6.3 → 98.85.0 → 96.4
    Llama-3.1-8B-InstructUnit Price and Total Cost23.8 → 100.023.6 → 96.4
    Llama-3.1-8B-InstructFees and Change10.0 → 97.512.9 → 93.6

    2. Heterogeneous GSM8K repair (Section 5.2): original model → RESCUE after circuit-restricted tuning

    ModelTrain (%)Test (%)
    Qwen3-8B-Instruct0.0 → 70.00.0 → 47.9
    Llama-3.1-8B-Instruct6.3 → 81.34.3 → 69.3

    3. MedMCQA repair(Section 5.6): original model → RESCUE after circuit-restricted tuning

    ModelTrain (%)Test (%)
    Qwen3-8B-Instruct2.5 → 78.82.9 → 77.9
    Llama-3.1-8B-Instruct0.0 → 88.80.0 → 75.7

These held-out results demonstrate that RESCUE effectively repairs reasoning errors, with improvements extending beyond the training examples. We will incorporate these results into the revised PDF.

  • Data used for intermediate decisions:

    • Mask-checkpoint selection: Repair evaluation subset (SFT); repair and clean evaluation subsets (RL), subject to clean-performance and circuit-budget constraints.

    • RL update acceptance: Repair and clean evaluation subsets; updates are retained only if they satisfy the repair, clean-performance, and circuit-budget constraints, otherwise rolled back.

    • Pruning: Repair evaluation subset and clean subsets, including training and evaluation examples; row-group reopening must satisfy both repair and clean-performance tolerances.

    • Tuning-checkpoint selection: Repair evaluation accuracy, with clean evaluation accuracy used to break ties.

AuthorsOpenReview ↗

[Part II] Responses to Question 2 & 3

For Question 2: Held-out Repair Accuracies after rephrase

Thank you for this suggestion. We additionally rephrased the expanded independent test set and directly evaluated these examples using the previously localized circuits (Section 5.4). The results are reported below:

The table reports test accuracy (%) before and after circuit ablation.

ModelError PatternOriginal model Acc(%)Circuit ablation Acc(%)
Qwen3-8B-InstructMisleading Premise Back-solving23.643.6
Qwen3-8B-InstructDynamic State Tracking17.125.7
Qwen3-8B-InstructDropped-lowest Average2.919.3
Qwen3-8B-InstructDeclining Group Accumulation1.420.7
Qwen3-8B-InstructMultiplicative Increment Back-solving0.725.0
Llama-3.1-8B-InstructCumulative Target Gap23.660.7
Llama-3.1-8B-InstructScale Conversion30.754.3
Llama-3.1-8B-InstructPercentage Risk27.183.6
Llama-3.1-8B-InstructUnit Price and Total Cost40.787.1
Llama-3.1-8B-InstructFees and Change15.777.9

The transfer results on the independent test set indicate that RESCUE captures components associated with recurring reasoning failures rather than template-specific lexical mappings. We will incorporate these results into the revised PDF.

For Question 3: Sample Counts and Uncertainty on Benchmarks

  • Sample Counts on Benchmarks

We use the Language Model Evaluation Harness (lm-eval) for public-benchmark evaluation. The complete evaluation runs cover all examples in each benchmark's designated evaluation split. We verified the evaluated counts against the saved result files.The corresponding sample counts are reported below:

BenchmarkEvaluation splitEvaluated examples
GSM8KTest1,319
MATH-500Test500
MedMCQAValidation4,183
ARC-ChallengeTest1,172
HellaSwagValidation10,042
WinoGrandeValidation1,267
PIQAValidation1,838
TruthfulQA-MC1Validation817
MMLU High School MathematicsTest270
MMLU High School StatisticsTest216
BBH Multi-Step ArithmeticTest250
  • Uncertainty

We estimated 95% confidence intervals for GSM8K accuracy improvements using paired bootstrap resampling (10,000 resamples) over all 1,319 test examples. The results are reported below:

ModelOriginal accuracy (%)RESCUE accuracy (%)Improvement (pp)95% CI for improvement (pp)
Qwen3-8B-Instruct80.4490.75+10.31[+8.04, +12.05]
Llama-3.1-8B-Instruct73.0977.18+4.09[+1.97, +6.22]

Both models show positive GSM8K improvements, with 95% confidence intervals excluding zero.

AuthorsOpenReview ↗

[Part III] Responses to Question 4

For Question 4: LoRA Experimental Setup and Llama GSM8K Degradation

  • LoRA Experimental Setup:

We thank the reviewer for this valuable suggestion. We conducted additional validation-based hyperparameter tuning for both models. We searched LoRA ranks {8, 16} and learning rates {1e-5, 5e-5, 1e-4, 2e-4, 3e-4}, resulting in ten configurations per model. Each configuration was trained for up to ten epochs, with evaluation after every epoch. LoRA alpha was set to twice the rank, dropout to 0.05, and adapters were applied to attention and MLP projections.

Checkpoints were selected based on repair validation accuracy, with clean validation accuracy used to break ties. Earlier checkpoints were preferred for exact ties. Across configurations, ties were resolved by preferring smaller ranks and then lower learning rates.

  • Llama GSM8K Degradation:

We evaluated the selected checkpoints on the complete GSM8K test set using lm-eval. The results are as follows:

ModelSelected rankSelected learning rateSelected epochLoRARESCUE (Ours)
Qwen3-8B-Instruct161e-4685%91%
Llama-3.1-8B-Instruct165e-51073%77%

After validation-based tuning, the Llama degradation no longer persists: LoRA achieves 73%, matching the original model (73%). However, it still falls short of RESCUE (77%). RESCUE also outperforms the tuned LoRA baseline on Qwen3-8B-Instruct. We will incorporate these results and the corresponding tuning protocol into the revised PDF.

Thank you again for your careful evaluation and helpful suggestions. We will address each point in the revised manuscript. We hope our responses and additional experiments provide useful clarification, and we would be grateful for any favorable reassesment of our work you may find warranted.

Sincerely,

Authors

[1] Towards Automated Circuit Discovery for Mechanistic Interpretability (NeurIPS 2023)

[2] Finding Transformer Circuits with Edge Pruning (NeurIPS 2024)

[3] IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck (ICML 2025)

[4] SafeSeek: Universal Attribution of Safety Circuits in Language Models (ICML 2026)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.