- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes BCTI, which localizes components causally supporting factual recall via bidirectional causal tracing, then directly minimizes their restoration scores with localized LoRA adapters under retention constraints.
Critical strengths
- Distinguishing "the answer is suppressed" from "the mechanism that can restore the answer is weakened" is conceptually important.
- The overall method and framework design are well motivated.
Critical weaknesses
- In Table 2, the answer-only objective achieves a 98.6% causal-restoration reduction, identical to full BCTI. This suggests that a plain answer-level objective may by itself be sufficient to disrupt the audited causal pathway.
- Table 1 lacks the causal-restoration reduction metric for the baselines.
- Redundant presentation: Figure 4 largely restates Table 1, and Figure 5 largely restates Table 2.
- Several basic definitions are missing. The clean/corrupted/patched runs and the corruption procedure itself are never defined, and some metrics are used before being introduced, which forces the reader to consult prior literature. A more intuitive explanation of why minimizing L_causal leads to forgetting would also help readability. The Table 2 column header "Causal restoration" would be clearer as "Causal-restoration reduction". As written it is easy to misreadr.
Questions for authors
See weaknesses
3 comments
The comparison confines answer-only to conditions designed around BCTI, which may prevent it from discovering alternative strategies. Its poorer performance is therefore understandable, especially against explicitly guided BCTI. This supports BCTI’s interpretability and targeted effectiveness in this setting, but not the intrinsic superiority of its causal-mechanistic guidance. In addition, Table 1 still lacks complete results and requires further refinement. I would like to keep my score.
Thank you for your follow-up and for recognizing BCTI’s interpretability and targeted effectiveness. We would like to clarify the purpose of the answer-only ablation and the Table 1 updates already reported.
- Your concern about whether the “conditions designed around BCTI” explain this advantage is important; our comprehensive ablations are designed precisely to address it. Answer-only receives the same guidance on update locations, isolating the causal loss’s added benefit and following the earlier request for “similar retention levels” and “the same update locations.” Separately, replacing BCTI-selected update locations with ROME-selected update locations reduces answer reduction to 0.6%. Thus, both components are tested separately, and the shared setup controls for update locations rather than giving BCTI an exclusive advantage.
- Regarding Table 1, our previous response already reported supplementary causal-restoration results for GA+KL and NPO-LoRA, alongside their retention metrics. We also explicitly acknowledged that the remaining baseline evaluations were ongoing and would be included in the final version.
We hope this clarifies why the controlled ablation directly addresses the causal objective’s contribution and how we are completing Table 1. Thank you again for helping us strengthen the evidence and its presentation. We welcome further discussion of any remaining concerns.
Thank you for recognizing the conceptual importance of distinguishing answer suppression from weakening the mechanisms that can restore the answer, and for your positive assessment of the motivation behind our overall method and framework design. Your detailed observations have helped us sharpen the controlled comparisons, broaden the causal audit of baseline methods, and improve the clarity of the presentation.
1. Identical causal-restoration reduction for answer-only and full BCTI
You correctly observed that the original Table 2 reports the same mean causal-restoration reduction (98.6%) for full BCTI and answer-only. This observation motivated a controlled comparison at similar retention levels. We trained answer-only at the same BCTI-selected update locations and stopped when validation retain KL first reached BCTI's target. At this point, maximum retain-answer drop is 11.7 ± 6.2% for answer-only and 12.5 ± 10.7% for BCTI; target-answer reduction is 64.1 ± 18.4% versus 87.8 ± 7.8%, and causal-restoration reduction is 84.9 ± 16.8% versus 98.6 ± 1.5%. This shared validation retain-KL criterion highlights the contribution of the causal objective. In the causal-only ablation, target-answer scores on held-out prompts decrease by 83.4 ± 9.1% and causal restoration decreases by 97.9 ± 1.9%, demonstrating that the restoration effect provides an effective forgetting signal. The direct- and benign-relearning tests further show slower recovery and lower answer scores for BCTI, with particularly clear differences in causal scores and under benign relearning.
2. Missing causal-restoration reduction for baselines in Table 1
Thank you for suggesting the additional causal-restoration evaluations for baselines. We conducted post-hoc audits of the existing GA+KL and NPO-LoRA checkpoints using their original training results and checkpoint selections. Their causal-restoration reductions are 100.00 ± 0.01% and 69.3 ± 34.1%, respectively, compared with 98.6 ± 1.5% for BCTI. Retention provides essential context: GA+KL has retain KL of 0.655 ± 0.395 and maximum retain-answer drop of 77.0 ± 17.2%; BCTI achieves 0.006 ± 0.004 and 12.5 ± 10.7%, respectively. NPO-LoRA and BCTI each update approximately 2.10M parameters. The expanded table therefore enables a joint comparison of causal-restoration reduction and retention outcomes.
The expanded baseline table will be included in the updated PDF.
3. Redundant presentation in Figures 4 and 5
Thank you for identifying this redundancy. Your suggestion allows us to devote more space to new recovery evidence. We will remove Figures 4 and 5, which overlap with the tabulated comparisons, and use the space for the direct- and benign-relearning trajectories, reporting both answer and causal scores.
4. Definitions, corruption procedure, intuition, and metric naming
Thank you for these precise suggestions. Making the intervention easier to follow should also make the connection between the causal loss and forgetting more accessible. We will introduce the clean, corrupted, and patched runs before the metric is used and describe the corruption operation explicitly. For intuition, in “The capital of France is …”, the clean run uses the original prompt; the corrupted run perturbs the tokens identifying France; and the patched run restores a selected clean activation within the corrupted run. An increase in the score of “Paris” after patching measures the restorative contribution of that state under this intervention. Our causal loss penalizes this patching-induced increase, connecting the operational metric directly to the optimization objective. Finally, following your naming suggestion, we will rename the Table 2 column “Causal-restoration reduction” to specify that the column reports the decrease in the restoration effect.
We hope these controlled and extended evaluations, together with the clearer presentation, address the main issues you raised. Thank you again for helping us strengthen both the evidence and its interpretation. We welcome and look forward to further discussion of any remaining questions. The above discussions and corresponding updates will be reflected in the updated PDF, which we are currently preparing.