- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This article studies knowledge editing methods based on activation distribution (such as SAKE) and finds that there is a gap between representation space alignment and output behavior: the layer with the best representation alignment cannot make the model output reflect the target knowledge (behavior under implementation), while increasing intervention intensity can improve the success rate of editing, but it will cause the output to deviate from the output distribution of the model when "new knowledge is explicitly given in context" (behavior overshoot) and damage locality. The author proposes BECAS based on this: using the transmission direction of SAKE, taking the output distribution under multiple contextual conditions as a reliable behavior reference, and jointly searching for intervention layers and strengths for each editor, so as to align the target comprehensive behavior, achieve sufficient editing effect, and maintain locality. In the CounterFact and zsRE experiments on GPT2-XL, BECAS improved generalization and locality while maintaining an editing success rate of approximately 93%, and significantly reduced behavioral differences.
Critical strengths
- The diagnostic analysis layer by layer and intensity by intensity clearly demonstrates the two phenomena of "behavior under implementation" and "behavior overshoot", providing an intuitive and useful perspective for the selection of intervention intensity in activated knowledge editing.
- The method is simple and easy to implement, without the need to train new transmission maps, and has significant improvements in locality and behavior alignment compared to SAKE.
Critical weaknesses
- The experiment was only conducted on an older small model, GPT2-XL, and the candidate layer was also selected for it. It was not validated on modern models such as Llama and Qwen, nor did it consider more practical scenarios such as batch or sequence editing. Therefore, there is a great doubt about whether the conclusion can be generalized, which is the main reason why I tend to reject it.
- There is a certain degree of cyclicality in the evaluation. The main advantages of BECAS are reflected in two indicators: behavioral JS and overshoot, which are its direct optimization goals; At the same time, each editor uses neighborhood queries for calibration, which are close to the distribution of queries evaluating locality. Therefore, the significant improvement in these indicators cannot fully demonstrate that the editing quality has truly improved.
- The baseline comparison is not complete enough. The paper explicitly lists ICE as a comparative method in the experimental setup, but its results are not presented in the results table; The behavioral reference used by BECAS comes from the output distribution under contextual conditions. Without this comparison, it is difficult to determine whether the benefits come from activating the intervention itself or borrowing the behavioral reference. In addition, recent methods such as AlphaEdit mentioned in related work have not been included in the comparison.
Questions for authors
- How effective is BECAS on larger and newer models such as Llama-8B or Qwen series? Does the selection of candidate layers require re diagnosis for each model?
- Why is there no ICE in the results table? It uses the same behavioral reference as BECAS, and this comparison is crucial for understanding the sources of revenue.
- How are neighborhood queries used for calibration and evaluation of locality divided? Are they from the same neighborhood set?
- What is the additional computational cost for each editor to perform layer and intensity searches?
3 comments
Response to Reviewer dHUD
Q1: Cross-model
Appendix C.2 of the submitted paper reports single-fact editing results on LLaMA-3-8B-Instruct. On this model, BECAS improves both "Gen." and "Loc." relative to SAKE. We will move this comparison into the main text and specify the candidate layers used for LLaMA-3-8B-Instruct. Performance on the Qwen family remains to be evaluated.
When transferring BECAS to a new model, we recommend running a layer-wise diagnostic once to identify its candidate layers, since layer indices and responses to activation interventions can differ across architectures. The resulting candidate set is reused across edits on that model. For each edit, BECAS selects an intervention layer from that set and tunes the intervention strength.
Q2: More baseline
We thank the reviewer for pointing out this omission: while we included ICE in our experimental setup, we failed to report the corresponding results. We now report ICE alongside SAKE and BECAS on the same CounterFact/ZsRE edits on GPT2-XL, using identical held-out evaluation prompts and metric definitions.
| Method on CounterFact | EFF.↑ | Gen.↑ | Loc.↑ | B.JS↓ | OS↓ |
|---|---|---|---|---|---|
| ICE | 99.87 | 71.62 | 56.90 | 26.21 | 15.37 |
| SAKE | 96.20 | 87.54 | 84.67 | 27.79 | 43.46 |
| BECAS | 93.37 | 93.13 | 90.84 | 15.92 | 7.67 |
| Method on ZsRE | EFF.↑ | Gen.↑ | Loc.↑ | B.JS↓ | OS↓ |
|---|---|---|---|---|---|
| ICE | 98.90 | 78.41 | 61.87 | 26.71 | 18.61 |
| SAKE | 95.47 | 86.27 | 87.31 | 28.08 | 42.13 |
| BECAS | 92.72 | 92.01 | 90.32 | 10.82 | 11.24 |
ICE and BECAS both use model outputs conditioned on the new fact, but they use those outputs differently. ICE uses a context-conditioned teacher for a weight update based on the rewrite prompt. BECAS uses a reference built from multiple contextual queries to select an activation intervention. The comparison therefore reflects differences in both the intervention mechanism and the information available during editing.
We agree that AlphaEdit would broaden the comparison with recent editing methods. We do not yet have AlphaEdit results under the matched evaluation protocol, so the revised table reports the directly comparable ICE results and the behavioral-loss ablation. A matched comparison with AlphaEdit remains open.
Q3: Neighborhood queries for calibration and evaluation
Yes. As described in Appendix A.3, for each edit we sample one query from its CounterFact neighborhood pool for locality calibration and reserve the remaining neighborhood queries for evaluation. In the revised 1,000-edit run, we evaluate four remaining neighborhood queries per edit and verify that none overlaps with the calibration query after text normalization.
Q4: Additional computational cost
We benchmarked 100 paired single-fact edits on GPT2-XL using one NVIDIA RTX A6000, FP32, and six CPU threads. Each method ran independently. Timings exclude model loading, warm-up, and final quality evaluation.
| Method | Mean time per edit (s) | P95(s) | Peak GPU memory (GiB) |
|---|---|---|---|
| SAKE @L42, | 4.714 | 5.476 | 5.98 GiB |
| L=42, continuous | 5.612 | 6.013 | 5.98 GiB |
| ICE | 4.188 | 4.297 | 7.05 GiB |
| BECAS | 11.098 | 11.673 | 5.99 GiB |
Compared with fixed-layer SAKE, BECAS adds 6.384 seconds per edit (95% paired confidence interval [6.243, 6.530]). BECAS performs the search once per edit and reuses the selected intervention for subsequent queries. Another time-consuming aspect is that BECAS also needs to construct transport mappings for the two candidate layers separately.
Thank you again.
Thank you for the detailed rebuttal and the additional experiments. In light of this, I am raising my score from 1 to 2.
Response to Reviewer dHUD
Thank you for your thoughtful review and constructive feedback.
W1: Cross-model and more practical editing scenarios
Cross model: Refer to Q1. Appendix C.2 of the submitted paper reports single-fact editing results on LLaMA-3-8B-Instruct.
More practical editing scenarios: Batch and sequential editing are both important practical settings. We are currently evaluating BECAS in multi-edit settings(ongoing) and will report the latest results before the discussion deadline if time permits.
W2: behavioral JS and overshoot improvement cannot fully demonstrate that the editing quality has truly improved.
Behavioral JS and overshoot alone are insufficient to establish editing quality, given their relationship to the calibration objectives. The "Gen." and "Loc." reported in Table 1 (can also refer to Q2) of the main paper are evaluation results on held-out paraphrase and neighborhood queries, respectively. These queries were excluded from calibration and from the per-edit selection of intervention layer and strength. Across both datasets, BECAS improves generalization to unseen paraphrases and locality on unseen neighborhood queries relative to SAKE and ICE. These gains on held-out task metrics provide evidence of editing effectiveness beyond the behavioral alignment metrics.
As described in Appendix A.3, calibration uses the original rewrite query, one sampled paraphrase, and one sampled neighborhood query per edit. The remaining paraphrase and neighborhood queries are reserved for evaluation. The calibration and evaluation neighborhood queries come from the same edit-specific pool, so the locality results measure preservation on unseen queries within that neighborhood. We will state this split alongside Table 1 and refer to Appendix A.3.
W3: More baseline and where benefits come from
More baseline: Refer to Q2. We now report ICE on the same CounterFact/ZsRE edits, using identical held-out evaluation prompts and metric definitions.
Benefits come from activating the intervention itself or borrowing the behavioral reference. ICE and BECAS both use model outputs conditioned on the new fact, but they use those outputs differently. ICE uses a context-conditioned teacher for a weight update based on the rewrite prompt. BECAS uses a reference built from multiple contextual queries to select an activation intervention. The comparison therefore reflects differences in both the intervention mechanism and the information available during editing.
Table 3 in main paper includes a direct ablation, "BECAS w/o Lbeh". Following the evaluation protocol in Appendix A.3, both variants are evaluated on held-out queries for the same edits. The comparison isolates the effect of including the behavioral reference in calibration while retaining the other calibration objectives. The results demonstrate that, within the BECAS framework, the behavioral reference aligment contributes to the performance(Gen, Loc) of BECAS.