- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies personal knowledge updates, examining how a model can use new facts when user attributes, such as occupation or location, change while preserving other personal information and general knowledge.
SPIKE maps representations of the initial and updated knowledge graphs to activation intervention vectors, which are injected into selected attention heads. The base model remains frozen during updating, and the training objective balances accuracy on new facts, knowledge retention, and intervention magnitude.
Critical strengths
1.Private entities and updates to multiple attributes differ from traditional editing of public facts. The paper designs its method around these differences rather than merely changing the dataset.
2.SPIKE performs consistently across four datasets, with substantial advantages over some editing methods when updating multiple facts about the same person. The comparison between attention head and FFN interventions also supports the design choice.
3.Using the difference between the initial and updated knowledge as the intervention signal better reflects the requirements of updating than encoding new facts alone.
Critical weaknesses
1.The personalization setting remains somewhat idealized. The experiments first train the model on the initial personal knowledge to an accuracy above 99% before evaluating updates. This helps isolate the updating problem, but real models often have incomplete knowledge of a user’s initial information.
2.The representation of differences between old and new facts appears to contradict the motivation that private entities lack meaningful representations. The paper argues that private entities lack meaningful representations in model parameters, yet the method learns an update representation that encodes the difference between old and new facts.
If an entity has never appeared before, where does its old fact representation come from? These two claims appear logically inconsistent.
3.The information available at inference time and the fairness of comparisons with retrieval-augmented methods need clarification. SPIKE uses structured knowledge and representations of its updates. Key questions include whether the system must identify the person and relation associated with a query, whether it must retrieve relevant triples, and how it selects the intervention vector.
If the system can already retrieve the relevant updates accurately, strong baselines should include context injection and retrieval-augmented methods using the same triples and retrieval results.
Questions for authors
See weakness.
3 comments
Response to Reviewer gFou (2)
W3.
Information required at inference. We thank the reviewer for these important questions. We answer the three questions in turn. (1) Identifying the person and relation: required. The Accuracy evaluation in Table 1 assumes that it is known which person and which relation the question is about. (2) Triple retrieval: no triple is retrieved at question time. Since Δ is computed from triples alone, independently of the question, it can be precomputed and stored whenever the KG is updated (Appendix A.11). (3) Intervention vector selection: the stored Δ for the (person, relation) pair is selected.
SPIKE therefore assumes that the update needed for a question is known exactly. This is the same level of information as the perfect retrieval assumed by the SC baseline, and we now state this assumption explicitly in the problem setting (Section 3.1).
Relaxing this assumption. We drop the assumption that the relation of the query is known and add a step that selects the relation corresponding to the given query. The relation is first selected by exact string matching, i.e., by checking whether the query is identical to the template of one of the person's relations (Table 12); unless exactly one template matches, the init-LLM is prompted to select the relation. On Qwen2.5-7B-Instruct / PerInfoKG, this yields (Acc, Loc, Ret, Tot) = (99.2, 96.3, 91.5, 95.6), compared with (99.4, 96.8, 91.9, 95.9) when the relation is known (Table 1), with all metrics changing by at most 0.5 points. This indicates that the known-relation assumption can be replaced by a simple selection step with little loss in performance. The details and results of this experiment are added to Appendix A.9.
Comparison with context injection using the same information. This baseline is already included. SC assumes perfect retrieval and provides only the triples needed for the question as context, and SC+3-Shot additionally provides three examples (Section 5.1.2). Both therefore use the same triples and the same retrieval results as SPIKE.
The difference lies in where the information is placed. Context injection puts personal information into the prompt of every question, so personal context is attached even to general-knowledge questions, which can lower Retention (e.g., SC+3-Shot reaches a Retention of only 34.3 on Qwen / PeaCoK-Ex). It also exposes sensitive raw records in the inference prompt. SPIKE injects updates as internal representations of the model, so personal context is not added to the prompt and raw records do not appear in it.
Since SC uses oracle retrieval, its performance is an upper bound for any context-injection or retrieval-augmented method that uses the same triples. On Qwen2.5-7B-Instruct / PerInfoKG, SPIKE without knowing query relevant relation (Total 95.6) still exceeds SC (51.2) and SC+3-Shot (83.0), both of which are given the oracle triples.
Thank you for your rebuttal. I will maintain my original score.
Response to Reviewer gFou
W1
We thank the revier for pointing this out. As the reviewer mentioned, our main experiments assume that the init-LLM knows more than 99% of the initial personal knowledge, in order to isolate the update problem. To evaluate the case where initial knowledge is incomplete, we added the following setting on PerInfoKG. For each user, about half (8–9) of the changed attributes are treated as missing and removed from the init KG, and the init-LLM is trained on the init KG without these attributes. The resulting init-LLM has an accuracy of 78.8%; on the 8 missing attributes it answers about 3 correctly on average, which are guesses based on the same relations learned for other users. The upd KG is used unchanged, so all changed attributes are injected regardless of whether they are missing.
The missing attributes therefore have no initial value in either the LLM parameters or the KG. For these attributes, we compute Δ using a zero vector as the KG embedding and, as the text embedding, a sentence in which the old value is replaced with "no information." Results on Qwen2.5-7B / PerInfoKG:
Even with about half of the changed attributes missing from the initial KG, Total stays at 95.5, close to the 95.9 without missing attributes. This shows that SPIKE can inject updates even when the old fact is absent from both the LLM and the KG.
W2.
Thank you for raising this insightful point. SPIKE's representation of the old fact, H^txtinit , is not the LLM-internal lookup key used by ROME-style methods. It is obtained from KG embeddings learned on KGinit (TransE, R-GCN), text embeddings, and a learnable alignment module. Moreover, H^txtinit is computed per (s, r, o) triple rather than as a key bound to a single subject, and it serves only as the reference point for computing Δ with respect to the updated triple. SPIKE therefore does not rely on the contextual stability of lookup keys that locate-and-edit methods assume, which we show does not hold for personal entities (Appendix A.2). In addition, in the missing-attribute experiment in our W1 response, where the initial information is absent from both the LLM and the KG (zero vector for the KG embedding, "no information" for the text), SPIKE still achieves an Accuracy of 98.82 on Qwen / PerInfoKG.