- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the problem of benign relearning in knowledge graph embedding unlearning and proposes a framework named Kernel-Anticipated Unlearning (KAU). KAU leverages tangent kernels to approximate how continued training affects forgotten facts through shared entity embeddings. It incorporates the predicted post-update scores into the forgetting objective, using sampled retained and stand-in edges to model future updates while preserving retained knowledge. The authors also introduce an evaluation protocol that measures relearning leakage relative to a retrained oracle. Experiments on two knowledge graph datasets with three embedding backbones demonstrate that KAU achieves lower relearning leakage and better retained utility than the evaluated baselines.
Critical strengths
S1. The paper addresses an important and underexplored problem of relearning in knowledge graph embeddings. The proposed evaluation protocol accounts for knowledge recovery in the retrained oracle, providing a meaningful measure of excess relearning.
S2. The proposed framework makes effective use of the structural properties of knowledge graph embeddings. The experiments cover multiple backbones and include extensive comparisons and ablation studies that support the effectiveness of the proposed method.
Critical weaknesses
W1. The evaluation does not sufficiently examine the model's ability to learn new knowledge during continued training. It remains unclear whether the improved relearning robustness comes at the cost of reduced learning plasticity.
W2. The experimental evaluation relies primarily on synthetic update scenarios and a single random deletion setting, leaving the generalizability of the proposed method to more realistic knowledge graph updates and deletion requests insufficiently explored.
Questions for authors
Q1. In Section 5.2, the evaluation focuses on the recovery of forgotten facts during continued training, while retained and test performance are mainly reported at release. Although KAU achieves lower relearning leakage, could this improvement come at the cost of reduced learning plasticity? Could the authors report the performance on newly introduced triples and retained knowledge after continued training, compared with the updated retrained oracle? This would help clarify whether KAU can selectively prevent forgotten facts from returning while preserving its ability to acquire new knowledge.
Q2. In Appendix C.3, the update configurations are constructed partly by sampling retained neighbors of the forgotten probe triples' endpoints. This creates a strong structural overlap between the update data and forgotten facts. How representative is this setting of ordinary knowledge graph updates? Would KAU maintain its advantages under updates sampled independently of the forgotten facts or constructed from naturally evolving knowledge graphs?
Q3. The experiments use a uniformly sampled forget set containing 10% of the training triples, as described in Section 5.1 and Appendix C.1. How would KAU perform under different deletion settings, such as smaller forget sets or entity-centric deletion requests? These settings may lead to different degrees of parameter sharing between forgotten and retained facts.
Q4. In Appendix D.5, Table 15 shows that the predicted and realized per-fact recovery have a relatively low Spearman correlation of 0.207, with a noticeable discrepancy between their average values. Although the authors discuss the ranking-based interpretation of the response, how consistent is this prediction quality across different backbones and update configurations? Could the authors further clarify why this approximation remains effective for optimizing the anticipated forgetting objective?
Q5. Appendix C.1 describes a sealed probe of 1,000 forgotten triples used for relearning evaluation, while Appendix C.3 states that the unlearner never sees this probe. Could the authors clarify whether these triples participate in the unlearning objective or are entirely excluded from the unlearning procedure? This distinction is important for understanding the evaluation protocol.
8 comments
W2: "The experimental evaluation relies primarily on synthetic update scenarios and a single random deletion setting [...]"
Q2: "[...] How representative is this setting of ordinary knowledge graph updates? Would KAU maintain its advantages under updates sampled independently of the forgotten facts or constructed from naturally evolving knowledge graphs?"
We answer the part of W2 on update scenarios here, together with Q2, and the part on deletion settings in comment 3, together with Q3.
Response: Yes. KAU has the lowest leakage under both kinds of update, in every setting we ran.
Updates sampled independently of the forgotten facts. Each update is a uniform sample of 10%, 25% or all of the retained graph plus held-out triples as new knowledge, at two learning rates, or two such 10% updates in sequence (five or seven configurations per setting, seven seeds), in all six settings of Table 1. Averaged over all configurations, KAU's leakage is 46%, 18%, 58%, 68%, 54% and 85% below the better baseline (Table R2), and below GraphDPO's in every seed in five settings (paired sign-flip test, p = 0.016) and in 6 of 7 seeds in the sixth. Under the worst-case rule of Table 1, KAU is again the lowest in all six settings, on CoDEx-L by 71%, 62% and 86% in every seed. On a second set of 750 forgotten facts, disjoint from the probe and used for no selection, its leakage in the same runs is 40% to 90% below the better baseline.
At comparable forgetting depth. On FB15k-237 TransE and TransH, KAU's Table 1 releases forget more deeply than GraphDPO's, and leakage grows with depth (Figure 3, Appendix D.6). We therefore gave both methods the same search of 56 release settings per backbone on one seed, selected for each the setting with the lowest worst-case graph-wide leakage in two depth bands by a rule fixed in advance, and evaluated the selected releases on 13 further seeds. KAU leaks 26% to 41% less than GraphDPO in all five comparisons (p ≤ 0.005; Table R2b), at higher retained MRR. On the second set of forgotten facts, which the selection did not use, the same KAU releases also leak less in all five comparisons (by 14% to 48%).
Table R2. Leakage under graph-wide updates at the Table 1 releases (seven seeds) as KAU / GraphDPO / ZeroFisher, with the seeds in which KAU is below GraphDPO. Worst case: the rule of Table 1. Mean: over all configurations.
| Probe, worst case | Probe, mean | Second set, mean | |
|---|---|---|---|
| FB15k-237 TransE | 0.0165 / 0.0198 / 0.0376 (5/7) | 0.0114 / 0.0212 / 0.0373 (7/7) | 0.0080 / 0.0140 / 0.0305 (7/7) |
| FB15k-237 TransH | 0.0213 / 0.0227 / 0.0399 (3/7) | 0.0201 / 0.0246 / 0.0391 (6/7) | 0.0102 / 0.0175 / 0.0289 (6/7) |
| FB15k-237 RotatE | 0.0126 / 0.0367 / 0.0213 (7/7) | 0.0063 / 0.0341 / 0.0150 (7/7) | 0.0105 / 0.0397 / 0.0174 (7/7) |
| CoDEx-L TransE | 0.0083 / 0.0290 / 0.0843 (7/7) | 0.0073 / 0.0224 / 0.0691 (7/7) | 0.0245 / 0.0449 / 0.0801 (7/7) |
| CoDEx-L TransH | 0.0162 / 0.0426 / 0.1075 (7/7) | 0.0168 / 0.0367 / 0.0935 (7/7) | 0.0143 / 0.0417 / 0.0989 (7/7) |
| CoDEx-L RotatE | 0.0096 / 0.0817 / 0.0678 (7/7) | 0.0086 / 0.0732 / 0.0563 (7/7) | 0.0081 / 0.0833 / 0.0838 (7/7) |
Table R2b. FB15k-237 at comparable forgetting depth (13 seeds not used for the selection). Depth is the forgetting rate at release; leakage is the per-seed maximum over three graph-wide updates.
| KAU depth | GraphDPO depth | Leakage, KAU / GraphDPO | KAU lower by (seeds; p) | |
|---|---|---|---|---|
| TransE | 68.9 | 61.5 | 0.0085 / 0.0138 | 39% (10/13; 0.004) |
| TransE | 84.0 | 77.3 | 0.0127 / 0.0213 | 40% (12/13; 0.0005) |
| TransH | 66.7 | 71.6 | 0.0125 / 0.0188 | 34% (11/13; 0.001) |
| TransH | 78.4 | 71.6 | 0.0139 / 0.0188 | 26% (12/13; 0.003) |
| TransH | 78.4 | 96.6 | 0.0139 / 0.0236 | 41% (12/13; 0.0005) |
A naturally evolving knowledge graph. ICEWS05-15 is a temporal knowledge graph of dated political events. We trained on all facts observed before 2013, deleted 10% of them, and used as the update the real stream of facts that appeared afterwards, so nothing in this update is constructed by us. KAU leaks 0.0071, against 0.0135 for GraphDPO and 0.0151 for ZeroFisher. This is 47% below the better baseline, and below GraphDPO in every seed of each evaluated configuration, while KAU retains the most knowledge (a retained MRR of 0.149, with the oracle at 0.154). GraphDPO's forgetting is fully undone by the stream in 14 of its 21 runs, and KAU's in none (Table R3).
Table R3. ICEWS05-15 with TransE (seven seeds, three update configurations).
| KAU | GraphDPO | ZeroFisher | Oracle | |
|---|---|---|---|---|
| Leakage, under the update where each method leaks most | 0.0071 | 0.0135 | 0.0151 | n/a |
| Forgetting depth at release (100 = oracle) | 106 | 84 | 113 | 100 |
| Retained MRR at release | 0.149 | 0.126 | 0.123 | 0.154 |
| Runs, out of 21, in which the forgetting is fully undone | 0 | 14 | 0 | n/a |
| Test MRR after the update | 0.112 | 0.114 | 0.090 | 0.112 |
Other training procedures. Under SGD, Adagrad and self-adversarial training, KAU has the lowest leakage in all seven cases that we ran.
How representative is the main protocol? Recovery flows through the entity embeddings that are shared with the update (Appendix D.4, Table 14). When an update avoids the entities of the evaluated forgotten facts, these facts do not come back for KAU or GraphDPO. In practice such sharing is the rule, since 97% to 99% of all retained edges touch an entity of some forgotten fact, so ordinary updates reach these parameters too. The updates of our main protocol concentrate on them (82% to 100% of their edges touch a probe entity, against about 55% for a uniform sample) and therefore act as a stress test. The paper also varies this protocol: the 48 configurations of Appendix D.1 change the number of new edges, the number of neighbours, the learning rate and whether random retained edges away from the probe are added, and Appendix D.2 uses an update made of never-trained edges only.
KAU's advantage is thus not specific to the update class of our protocol: it holds under graph-wide updates in all six settings, at comparable forgetting depth, and on a real temporal stream. These results will be in a new appendix section of the updated draft, and the graph-wide and ICEWS results will be reported next to Table 1.
Thank you for the constructive review and the clear questions. We answer them in the order of the review, and where a weakness and a question raise the same point we answer them together.
W1: "The evaluation does not sufficiently examine the model's ability to learn new knowledge during continued training. It remains unclear whether the improved relearning robustness comes at the cost of reduced learning plasticity."
Q1: "[...] Could the authors report the performance on newly introduced triples and retained knowledge after continued training, compared with the updated retrained oracle? [...]"
Response: Thank you for raising this. It is the natural question for any method that resists relearning, and the answer is that KAU's robustness does not come at the cost of plasticity. We now show this directly. We continued training on newly introduced triples only, with half of them held out, in all six dataset-backbone combinations. We then compared every model with the identically updated retrained oracle and with the original model that never unlearned (Table R1).
Table R1. Learning new facts (seven seeds). The gain is the improvement of the MRR on the new triples during the update, divided by the oracle's improvement, so that 1.00 means learning exactly as much as the oracle. Distances are differences in MRR on the new triples after the update. The last column is the leakage of the forgotten facts (Eq. 1) in the same runs.
KAU learns new facts at the oracle's rate. Its gain on the new triples is 89% to 111% of the oracle's. The original model that never unlearned gains 86% to 94% under the same update, and KAU gains at least as much as this model in all six combinations. Unlearning with KAU therefore costs no plasticity.
After learning, KAU sits at the oracle's level. Its MRR on the new triples differs from the oracle's by only 0.001 to 0.009 in every combination, which is the closest of the three methods in every seed (GraphDPO is 0.013 to 0.077 away, and ZeroFisher 0.157 to 0.258). On the held-out new triples, KAU is within 0.006 of the oracle in all six combinations. As for retained knowledge, KAU's MRR on held-out retained triples after the update is above GraphDPO's in all six combinations and in every seed. It is 0.005 to 0.017 below the oracle's, where GraphDPO is 0.023 to 0.103 below. KAU's test MRR after the update is within 0.003 of the oracle's.
KAU stays selective while it learns. In the same runs, its forgotten facts rise only 0.001 to 0.023 beyond the oracle's, against 0.012 to 0.035 for GraphDPO and 0.040 to 0.135 for ZeroFisher. This is the lowest value in all six combinations. New knowledge is acquired while the forgotten knowledge stays forgotten.
The same holds for new facts that are unrelated to the forgotten ones. When the new triples share no entity with the evaluated forgotten facts, KAU learns at 94% to 105% of the oracle's rate and again ends closest to the oracle. In this case no method relearns the forgotten facts, as the mechanism of our Introduction and Appendix D.4 predicts: recovery travels through entity embeddings that are shared with the update, and this update shares none.
One remark on the baselines: their larger raw gains do not indicate better plasticity. They start below the oracle and end far from it (GraphDPO by up to 0.077 MRR and ZeroFisher by up to 0.258), whereas KAU ends at the oracle's level. KAU is the method whose learning matches that of the retrained model. These results will be in a new appendix section of the updated draft.