- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RESCUE, a method for repairing recurring LLM errors by identifying a small set of MLP projection rows and updating only their weights. It combines supervised mask search, rollout-based RL refinement, circuit pruning, and restricted fine-tuning. Experiments on Qwen3-8B and Llama-3.1-8B-Instruct report strong repair-set improvements and some gains on standard benchmarks while largely preserving non-target performance.
Critical strengths
1.The paper studies a useful problem: targeted error repair with minimal interference to existing capabilities. The large gains from suppressing a small fraction of MLP rows are noteworthy.
2.The evaluation includes two model families, mathematical and medical tasks, random-circuit controls, rephrasing, and sequential repair. The public GSM8K improvements are encouraging.
Critical weaknesses
1.The pattern-specific datasets are generated from shared solution templates, yet Table 1 reports full-set accuracy rather than separate held-out results. Direct transfer to rephrased inputs is also much weaker for several Qwen3-8B patterns (Table 2). The large repair-set gains therefore need more convincing independent evaluation.
2.Finding rows whose removal improves accuracy does not by itself show that they form the error-producing circuit. Also, the supervised-only baseline removes both RL and pruning, so the specific contribution of RL cannot be determined.
3.The baseline comparisons are limited, as RESCUE is not directly compared with existing circuit discovery methods, and the supervised-only baseline does not isolate the contributions of RL refinement and pruning.
Questions for authors
1.Can the authors report held-out repair accuracy separately from training/full-set accuracy, preferably also on different solution templates?
2.Can the authors separate the effects of RL refinement and pruning, and provide stronger evidence that the selected rows represent error-causing computations rather than effective intervention sites?
3.How was LoRA tuned? Its large degradation on Llama-3.1-8B-Instruct raises concerns about baseline fairness.
4.Could the authors provide comparisons with existing circuit discovery methods and separate ablations for RL refinement and pruning?
7 comments
[Part II] Responses to Weakness 2 and Question 2 & 3
For Weakness 2 and Question 2: Disentangling the Contributions of RL Refinement and Pruning, and Validating Error-Associated Circuits
1. Disentangling the Contributions of RL Refinement and Pruning
We acknowledge this limitation and have added supplementary ablation experiments to disentangle the contributions of RL refinement and pruning. The table reports GSM8K test accuracy (%) evaluated using lm-eval after the same circuit-restricted tuning procedure:
| Localization pipeline | Qwen3-8B-Instruct | Llama-3.1-8B-Instruct |
|---|---|---|
| SFT | 87% | 73% |
| SFT + pruning | 88% | 74% |
| SFT + RL + pruning (RESCUE) | 91% | 77% |
Pruning improves accuracy by 1 percentage point on both models, while adding RL refinement yields a further 3-point improvement on each. These results demonstrate the additional contribution of RL to circuit localization and downstream repair. We will incorporate this ablation into the revised PDF.
2. Validating Error-Associated Circuits
-
Intervention-Based Definition of Error-Associated Circuits Prior mechanistic interpretability studies such as
SafeSeek[1] andSHIPS[2] have adopted an intervention-based attribution principle: attention heads or neurons whose ablation causes safety-aligned LLMs to exhibit unsafe behaviors are considered safety-related components. We follow the same principle to define error-associated circuits. Specifically, computations genuinely contributing to a reasoning error should exhibit a key property: their targeted suppression should reduce the corresponding errors while preserving correct behavior on clean inputs. The circuits identified by RESCUE satisfy this criterion and further support effective downstream repair. -
Complementary Evidence from Ablation and Targeted Repair Our experiments provide complementary evidence for the validity of the identified circuits.
- First, ablating RESCUE-localized circuits substantially reduces reasoning errors, whereas density-matched random circuits yield little improvement. For pattern-specific repair, RESCUE raises accuracy from 0–4% to 76–94% on Qwen3-8B-Instruct and from 1–36% to 92–95% on Llama-3.1-8B-Instruct, while random-circuit controls achieve only 0–13.5% mean accuracy.
- Second, circuit-restricted tuning based on RESCUE achieves 91% and 77% GSM8K accuracy on the two models, respectively, outperforming validation-tuned LoRA (85% and 73%).
Together, these results show that the identified circuits not only exhibit error-mitigating effects upon suppression but also provide more effective targets for subsequent repair. This supports interpreting RESCUE-localized error-associated circuits as plausible proxies for the underlying computations contributing to reasoning failures.
For Question 3: LoRA Tuning Protocol
We thank the reviewer for this valuable suggestion. We conducted additional validation-based hyperparameter tuning for both models. We searched LoRA ranks {8, 16} and learning rates {1e-5, 5e-5, 1e-4, 2e-4, 3e-4}, resulting in ten configurations per model. Each configuration was trained for up to ten epochs, with evaluation after every epoch. LoRA alpha was set to twice the rank, dropout to 0.05, and adapters were applied to attention and MLP projections.
Checkpoints were selected based on repair validation accuracy, with clean validation accuracy used to break ties. Earlier checkpoints were preferred for exact ties. Across configurations, ties were resolved by preferring smaller ranks and then lower learning rates.
- Llama GSM8K Degradation:
We evaluated the selected checkpoints on the complete GSM8K test set using lm-eval. The results are as follows:
| Model | Selected rank | Selected learning rate | Selected epoch | LoRA | RESCUE (Ours) |
|---|---|---|---|---|---|
| Qwen3-8B-Instruct | 16 | 1e-4 | 6 | 85% | 91% |
| Llama-3.1-8B-Instruct | 16 | 5e-5 | 10 | 73% | 77% |
After validation-based tuning, the Llama degradation no longer persists: LoRA achieves 73%, matching the original model (73%). However, it still falls short of RESCUE (77%). RESCUE also outperforms the tuned LoRA baseline on Qwen3-8B-Instruct. We will incorporate these results and the corresponding tuning protocol into the revised PDF.
[Part III] Responses to Weakness 3 and Question 4
For Weakness 3 and Question 4: Comparison with Existing Circuit Discovery Methods
Thank you for this suggestion. We agree that comparisons with established mask-based circuit localization methods are important. However, existing methods differ fundamentally from RESCUE in terms of what they localize and intervene on.
-
From Token-Level Attribution to Full-Generation Circuit Localization: Representative methods such as ACDC [3], Edge Pruning [4], and IBCircuit [5] primarily focus on identifying circuits responsible for only one-token prediction (e.g., predicting a binary "True" or "False" label in the Greater Than task). In contrast, RESCUE aims to localize error-associated circuits across the entire autoregressive generation process, addressing errors in complete, multi-step reasoning responses.
-
Limitations of Single-Token Circuit Attribution for Full-Generation Reasoning Repair: Full-generation reasoning repair is substantially more challenging than token-level circuit localization. Since autoregressive generation is sequential and context-dependent. Existing methods are not designed for full-generation intervention and therefore cannot directly address our reasoning-repair task.
Given the above differences, our newly proposed RESCUE framework is inherently difficult to compare directly with existing mask-based circuit localization methods.
Thank you again for your thoughtful and constructive feedback. We will address these points in the revised manuscript. We hope our clarifications and additional results help strengthen your confidence in our work and may merit a more favorable evaluation.
Best,
Authors
[1] SafeSeek: Universal Attribution of Safety Circuits in Language Models (ICML 2026)
[2] On the Role of Attention Heads in Large Language Model Safety (ICLR 2025 Oral)
[3] Towards Automated Circuit Discovery for Mechanistic Interpretability (NeurIPS 2023)
[4] Finding Transformer Circuits with Edge Pruning (NeurIPS 2024)
[5] IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck (ICML 2025)
[Part I] Responses to Weakness 1
Dear reviewer
NiKf,We sincerely thank the reviewer for the thoughtful and constructive comments. We have carefully considered each suggestion and provide our detailed point-by-point responses below.
For
Weakness 1andQuestion 1: Training and Held-out Repair AccuraciesThank you for this comment. We additionally expanded the independent held-out set and evaluated RESCUE on it. The results confirm that RESCUE remains effective on held-out examples, as shown below:
1. Pattern-specific repair (
Section 5.3): original model → circuit ablation2. Heterogeneous GSM8K repair (
Section 5.2): original model → RESCUE after circuit-restricted tuning3. MedMCQA repair (
Section 5.6): original model → RESCUE after circuit-restricted tuning4. Direct transfer to rephrased unseen problems
We additionally rephrased the expanded independent test set and directly evaluated these examples using the previously localized circuits (
Section 5.4).These held-out results demonstrate that RESCUE effectively repairs reasoning errors, with improvements extending to unseen examples. We will incorporate these results into the revised PDF.