acceptodds
Under review as a conference paper at ICLR 2027

Knowledge Property Evaluation and Injection In Large Language Models

Abstract

Knowledge editing corrects isolated facts in large language models, yet facts do not live in isolation: symmetric, inverse and transitive relations constrain many facts at once, and an edited model that violates them holds inconsistent knowledge. We formalise this gap as property learning: encoding a relational property through editing so that the model applies it to facts it was never shown. We introduce PropEdit-Bench, spanning geospatial and social domains, which unlike existing benchmarks assesses property abstraction directly: properties are injected on one set of entities and tested on a disjoint one. Property-aware metrics complement completion with binary judgements in both polarities, making the evaluation robust to one-to-many answers. Across LLM families and scales, a small number of property-instantiating edits, applied before the new fact, raise compliance over standard editing in every model, property and domain for individually edited facts, by up to relative. Completion and binary evaluations frequently disagree, exposing relations that models recognise but do not produce. Factual correction and relational structure are thus distinct capabilities, calling for editing methods that explicitly target the structure underlying edited knowledge.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Knowledge Editing, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_buxfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formalizes property learning in knowledge editing: relational properties such as symmetry, inverse relations, and transitivity are injected through examples on one entity set and evaluated on disjoint entities. It introduces PropEdit-Bench, property-compliance and bipolar binary metrics, and a detailed study of supervision scale, batching, sequential editing, and preservation of earlier edit keys. The experiments use AlphaEdit across Llama and Qwen models in geospatial and social domains.

Critical strengths
  1. Treating relational structure as a separate objective from isolated fact correction exposes a real weakness of existing editing benchmarks.

  2. Disjoint supervision and evaluation entities reduce direct memorization, while completion and positive/negative binary questions reveal failures that a single answer format would miss.

  3. The study varies model family and scale, domain, supervision amount, injection mode, and preservation constraints, and it openly reports specificity and efficacy costs rather than presenting compliance gains alone.

Critical weaknesses
  1. All main experiments use AlphaEdit, so the results may reflect the geometry and failure modes of this specific locate-then-edit method. The paper does not establish whether property learning behaves similarly for other parameter-modifying or memory-based editors.

  2. Table 2 reports the supervision-scale point with the highest PCR in each cell. Without a separate validation protocol for selecting the number of property pairs, these maxima can be optimistic and make cross-condition comparisons difficult to interpret.

  3. Training and test examples retain the same relation names and domain. The update may generalize through relation-specific lexical or representation similarity without acquiring a domain-independent notion of symmetry or inversion. This concern is reinforced by specificity losses, occasional efficacy collapse, and relatively small absolute gains in high-baseline settings.

Questions for authors
  1. Was the supervision scale selected on a held-out validation set? If not, how do conclusions change when a single scale is fixed before evaluating the test set or when selection bias is corrected?

  2. Do the main findings reproduce with other editors and under cross-relation or cross-domain transfer, where the property must generalize beyond the same relation token and domain?

1 comment
AuthorsOpenReview ↗

Response to Reviewer buxf

Thank you for the review. We answer each weakness with the matching question.

W1 / Q2 (editors). All main experiments use AlphaEdit.

Our scope is locate-then-edit editing, in which facts are written into the weights in closed form. AlphaEdit and MEMIT are the two representative editors of this family that write facts in batches, as our property phase requires. ROME writes one fact per rank-one update and cannot run it.

  • A second parameter-modifying editor. MEMIT is added on the three models (Table 4, App. A.5). For Llama3-8B on symmetry it gains +10.5 [7.8, 13.3] PCR points over Base, and a budget-matched control without property instances matches it, as with AlphaEdit. The pattern is not tied to the null-space projection of AlphaEdit.
  • Memory-based editors. Such editors keep the new fact outside the weights and supply it at inference. The new in-context control evaluates this case (Table 3, App. A.4): with the fact in the prompt, PCR rises from 21-35 to 56-74, and to 90-100 when ten supervised pairs precede it. The property is thus within reach of the models, and the limit we report is one of parameter-modifying editing. In-context provision remains a reference point and not a substitute: it requires the fact and instances of the property to be available and retrieved for every query, and its prompt grows with the number of facts and relations to cover, which becomes intractable at scale.

W2 / Q1. The supervision scale was selected on the test set.

No claim rests on that selection any more. Table 2 reports a single scale fixed in advance (n = 100), averaged over five independent draws of the supervised pairs, with a 95% bootstrap half-width on every number (App. A.1, Eq. 5). App. A.1 corrects for selection in two ways: the scale chosen on half of the test facts with the gain measured on the other half, and a max-T adjusted test. For Llama3-8B the gain holds under all three (+9.9 [7.1, 12.9] at n = 100; +5.2 to +10.9 cross-fitted; adjusted p < 0.001). Llama3-70B gains up to +1.7 [0.3, 3.5], and Qwen3-32B shows no gain.

W3 / Q2 (transfer). Supervision and test share relation names and domain.

Three new analyses answer this point.

  • Cross-relation supervision (Table 5, App. A.6). With pairs from other relations only, PCR on the held-out relation still rises for Llama3-8B (+6.6 [3.3, 9.9] on symmetry, +4.1 [0.7, 7.7] on reversibility). The gain therefore reaches relations absent from the supervision.
  • Budget-matched control (Table 2, App. A.2). Unpaired facts of the same relations produce the same PCR gain as pairs (difference between -0.6 and +1.5 points, significant in none of 12 settings), and they do so across relations as well. What transfers is that a new fact is more reliably written (efficacy 72.1 to 89.5 on the held-out relation).
  • What is specific to the property (App. A.3). Complete pairs add +2.0 [0.3, 3.7] points of binary discrimination on the counterpart, carried by symmetry, while its generation is unchanged.

This is the answer to RQ1: editing conveys a property in recognition but not yet in generation. Cross-domain transfer (Geo to Wikidata) was not run and is listed as future work (Sec. 7).

Specificity losses, efficacy collapse, small gains at high baselines. These are reported next to every result: Tables 10 and 12 give generalisation and specificity, and low efficacy is flagged in red. For some draws of the supervised pairs the efficacy of Qwen3-32B collapses, and it does so with unpaired facts as well (App. A.1). Gains are small where the baseline is high (at most +1.7 for Llama3-70B), and the text says so.

Reviewer_7sdEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies a gap in knowledge editing: whether knowledge edits can help language models to learn relational properties, including symmetry, reversibility, and transitivity. It introduces PropEdit-Bench and a two-phase protocol: property examples are edited first, new facts are then injected. The paper evaluates if the language model can correctly answer the unedited relations of the new facts.

Critical strengths
  1. This paper studies an important problem. Successfully editing a fact does not guarantee that the model will correctly handle its logical consequences. Evaluating this gap is useful for understanding the reliability of knowledge editing.

  2. The overall protocol is fair and easy to understand. Separating the property supervision entities from evaluation entities helps distinguish the ability of learning relation rather than facts.

  3. The evaluation considers both completion and binary judgment. The experiments also include different models, several supervision sizes, and editing regimes.

Critical weaknesses
  1. The claim of the paper is a little stronger. The experiments are mainly under AlphaEdit, and improved results sometimes disappears with sequential editing. The authors should better consider the claim that “property learning should be a first-class objective”.

  2. The method’s improvement is not sufficiently justified. It has an additional editing phase where the Base model does not. Budget-matching controls are needed to distinguish the true effect of the property learning process.

  3. The difference between existing papers is unclear. DepEdit, RuleEdit, ChainEdit already study relational dependencies beyond isolated facts. The authors should explain their differences more directly and include more baselines if necessary.

  4. The generality of the method needs further support. Table 2 selects only the best supervision scale. The authors should clarify whether this process has an independent validation set, and report budget-matching results under repeated runs.

[1] Evaluating Dependencies in Fact Editing for Language Models: Specificity and Implication Awareness.

[2] RuleEdit: Towards Rule-Level Knowledge Generalization to Mitigate Over-Editing in Large Language Models.

[3] ChainEdit: Propagating Ripple Effects in LLM Knowledge Editing through Logical Rule-Guided Chains.

Questions for authors
  1. Can the authors reconsider the claim of property learning as first-class objective?

  2. How does the proposed method perform under budget-matching settings that do not have property pairs? How’s the performance on held-out tasks?

  3. How does the problem setting differ from DepEdit and RuleEdit? What about their performance?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 7sdE

Thank you for the review. We answer the weaknesses as follows:

W1 / Q1. The claim that property learning should be a first-class objective is too strong.

We agree, and we rewrote it (abstract, Sec. 1, 4, 6.1 and 7). The paper no longer asserts the claim. It measures how far current editors achieve property learning, and states what the evidence supports:

  • supervision raises compliance on disjoint pairs, by up to +9.9 [7.1, 12.9] points, with two editors;
  • a budget-matched control reproduces this gain, and its main source is a higher efficacy of the new fact, which edits of the same relations suffice to obtain;
  • the property instances add a specific gain in recognition (+2.0 [0.3, 3.7] points of binary discrimination, carried by symmetric counterparts), while the generation of counterparts is unchanged;
  • the same models produce the counterpart when the facts are stated in their prompt (PCR of 90-100).

Editing therefore conveys a property in recognition but not yet in generation. This is what makes property learning a distinct and open target for editing methods, and it is now a finding, not a premise. The loss of the gain under sequential editing is stated in the summary of RQ2 and analysed in App. D.

W2 / Q2. Budget-matched controls; performance on held-out tasks.

Budget-matched control. The new condition Volume (Sec. 5.2; rows "volume" of Tables 2 and 11) writes 2n facts drawn from the property pool with the relation mix of the n pairs, never a fact together with its counterpart, in one update like Batch.

  • On Geo, Batch minus Volume lies between -0.6 and +1.5 PCR points at n = 100 and is significant in none of the 12 settings.
  • On Wikidata, the first-token accuracy of counterparts rises by +1.6 [0.8, 2.4] with pairs and +1.8 [0.9, 2.8] with unpaired facts.
  • The control is not favoured by the entities it touches: on the new facts sharing no entity with the supervision of either condition, Batch minus Volume is -0.5 [-1.0, +0.1] (App. A.2).
  • What the pairs add over the control appears in binary judgements (+2.0 [0.3, 3.7]; App. A.3).

Held-out tasks. We added a leave-one-relation-out experiment (Table 5, App. A.6). The pairs come from the other relations only (symmetry: two of borders, close, far; reversibility: north/south for east/west and conversely), and new facts of the held-out relation are tested. For Llama3-8B, PCR on the held-out relation rises by +6.6 [3.3, 9.9] on symmetry and +4.1 [0.7, 7.7] on reversibility. Unpaired facts of the training relations do as well: what transfers across relations is the gain in efficacy. Qwen3-32B shows no gain.

W3 / Q3. Difference with DepEdit, RuleEdit and ChainEdit; their performance.

Sec. 3 now positions the three works:

  • DepEdit evaluates whether an edit updates the facts it implies through If-Then rules, on the edited entities.
  • RuleEdit edits rule-level knowledge so that it generalises to the instances the rule derives.
  • ChainEdit propagates an edit along chains of facts linked by logical rules mined from a knowledge graph.

In the three, the rule or the implied facts are available to the method, and the consequences are checked on the edited entities. Our protocol gives no rule and never writes the counterpart of a new fact. A property is shown only through instances on other entity pairs, and we test whether the editor applies it to a new pair. A method that derives the counterpart from a rule answers our test by construction. These methods therefore measure a different ability, and we did not run them as baselines.

What access to the information brings in our protocol is measured by the new in-context control (Table 3, App. A.4): with the new fact in the prompt, PCR is 56-74, against 21-35 after editing, and it reaches 90-100 when ten instances of the property precede the fact.

W4. Best-scale selection, validation set, repeated runs.

The table that selected the best scale is gone. Table 2 reports Batch and its control at n = 100, fixed in advance, as means over five independent draws of the supervised pairs (four for the control), with a 95% bootstrap half-width on every number (App. A.1, Eq. 5). App. A.1 adds a cross-fitted estimate (scale chosen on half of the test facts, gain measured on the other half) and a selection-adjusted test.

  • Llama3-8B keeps its gain under all three: +9.9 [7.1, 12.9] at n = 100, +5.2 to +10.9 cross-fitted, adjusted p < 0.001.
  • Llama3-70B gains up to +1.7 [0.3, 3.5]. Qwen3-32B shows no gain.
  • Re-running a configuration changes PCR by at most 0.8 points, and the draw of the pairs by up to 13.6.

Presentation.

Each paragraph of Sec. 6 states one finding and each research question ends with a boxed summary. Conditions carry one name in the text, the tables and the figures. Broken references and legacy labels are fixed.

AuthorsOpenReview ↗

Follow-up on our response to 7sdE

Dear Reviewer 7sdE, as the discussion period is coming to an end, we wanted to follow up on our response, which we hope addresses your concerns. We would be glad to clarify anything further before the deadline.

Thanks for your detailed response. However, it still needs a major rebvision to meet the bar of ICLR.

Reviewer_VA7pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript frames property learning as knowledge-editing objective. Relational properties should be encoded through editing so that an edited model produces the counterpart of a newly edited fact, even for entities never seen during property supervision.

The authors release ProprEdit-Bench and propose a property-compliance rate together with a two-polartiy binary evaluation.

Critical strengths

This manuscript provide useful framing and a sound core design idea. Evaluating counterparts on entities disjoint from property supervision is the right control for separating abstration from memorization, and it addresses a limitation of BAKE, RippleEdit, and similar benchmarks.

Honest reportin gof the negative results is strength. The manuscript reports specificity costs, the failure of joint injection, and the loss of gains under sequential editing.

Critical weaknesses
  • The results are selected on the test set, with no variance estimates.

  • PCR may be confounded with efficacy. PCR multiples the score on b by the scroe on \pi(b), so any increase in how well b itself is learned raises PCR even if counterpart behavior is unchanged.

  • There are no control for nono-property edits. The treatment adds n extra edits before injecting b. Many effects could arise from editing.

  • Methodological scope is narrow. ONly one editor (AlphaEdit) is evaluated. There is no comparison with BIRD, MEMIT, fine-tuning, or in-context/retriefval editors, where the property could be supplied as a rule.

  • Several presentation and consistency issue, e.g., broken references, undefined legacy labels (RQ/RQJ), etc.

Questions for authors

Please refer weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer VA7p (1/2)

Thank you for the review. Your methodological points were included in the revision.

W1. Results selected on the test set, no variance estimates.

Table 2 no longer reports the best scale. Batch and its control are reported at n = 100, fixed before looking at the test facts, as means over five independent draws of the supervised pairs. Every number carries a 95% bootstrap half-width, over facts and over draws where several exist; App. A.1 defines the procedure (Eq. 5). It also adds two selection-aware estimates: the scale chosen on a random half of the test facts with the gain measured on the other half (200 splits), and a max-T sign-flip test over the scales.

  • Llama3-8B, symmetry: +9.9 [7.1, 12.9] with property keys discarded, +4.3 [1.4, 7.1] with keys kept. Cross-fitted: +5.2 to +10.9. Adjusted p < 0.001.
  • Llama3-70B: up to +1.7 [0.3, 3.5], adjusted p between 0.04 and 0.39. Qwen3-32B: no gain (p >= 0.21).
  • Re-running a configuration changes PCR by at most 0.8 points. The draw of the pairs changes it by up to 13.6, which is why draws are averaged.

W2. PCR may be confounded with efficacy.

It is, and the revision measures by how much. PCR multiplies the score of the edited fact by that of its counterpart, and supervision raises the first factor: for Llama3-8B on symmetry, efficacy goes from 72.1 to 95.2. PCR recomputed with the efficacy of Batch and the counterpart scores of Base gives +6.5 of the +9.9 points (App. A.2). We therefore:

  • report the conditional compliance RQ^th, which conditions on the edited fact being learned (Eq. 9, Table 19): 27.9 for Base, 30.5 with Batch, 31.2 with the budget-matched control;
  • measure the effect of the pairs on the counterpart alone, pooled over 30 settings (App. A.3): its completion is the same with pairs and with unpaired facts (difference within [-0.10, +0.11] points of first-token accuracy), and its binary discrimination is higher with pairs by +2.0 [0.3, 3.7] points;
  • state in Sec. 5.4 that PCR grows with efficacy, and give two references for a counterpart score: unedited models reach 17-36 on Geo, and models given the new fact in their prompt reach 68-84.

W3. No control for non-property edits.

We added it as the condition Volume (Sec. 5.2; rows "volume" of Tables 2 and 11): 2n facts drawn from the property pool with the relation mix of the n pairs, never a fact together with its counterpart, written in one update like Batch and followed by the same new facts.

  • On Geo, Batch minus Volume lies between -0.6 and +1.5 PCR points at n = 100 and is significant in none of the 12 settings (five draws for Batch, four for Volume).
  • On Wikidata (12 settings, one draw), the first-token accuracy of counterparts rises by +1.6 [0.8, 2.4] with pairs and +1.8 [0.9, 2.8] with unpaired facts. PCR does not move for the Llama models, because Base already writes the new fact almost perfectly there (efficacy above 97). For Qwen3-32B on symmetry, the unpaired facts of this draw lower efficacy (79-81 against 98 with pairs).
  • The control is not favoured by the entities it touches. On the new facts that share no entity with the supervision of either condition, Batch minus Volume is -0.5 [-1.0, +0.1] (App. A.2).

The effects you listed as alternatives thus account for the gain in PCR, and the control is what allows the paper to say so. What is specific to the pairs is the +2.0 points of binary discrimination, carried by symmetry: with the pairs, a model more often rejects the wrong relation for the counterpart (-2.1 [-3.9, -0.4] affirmations of the negative question). PCR does not register this effect. The binary evaluation in both polarities does (Sec. 6.3).

AuthorsOpenReview ↗

Response to Reviewer VA7p (2/2)

W4. Only AlphaEdit; no BIRD, MEMIT, fine-tuning, in-context or retrieval editors.

Our scope is locate-then-edit editing, in which facts are written into the weights in closed form. AlphaEdit and MEMIT are the two representative editors of this family that write facts in batches, as our property phase requires. ROME writes one fact per rank-one update and cannot run it.

  • MEMIT is added on the three models (Table 4, App. A.5). The pattern is the same: for Llama3-8B on symmetry, +10.5 [7.8, 13.3] over Base, matched by the control. The control matches or exceeds Batch in every MEMIT setting but one (Llama3-8B, reversibility: +4.0 [1.9, 6.2] for Batch, through efficacy).
  • In-context editing is added as a control (Table 3, App. A.4). No weight is changed and the new fact is stated in the prompt. PCR rises from 21-35 to 56-74, and to 90-100 when ten supervised pairs precede the fact. This is what an in-context or retrieval-based editor obtains once the fact is retrieved, and it shows that the property is within reach of the same models. We use it as a reference point and not as an alternative to editing: it requires the fact and instances of the property to be available and retrieved for every query, and its prompt grows with the number of facts and relations to cover, which becomes intractable at scale.
  • BIRD writes the reverse of each edited fact. In our protocol the counterpart is never written, so BIRD, or an editor that is given the rule, would be supplied with the answer we test for. Sec. 3 discusses these methods as ones that enforce a known rule rather than learn it from instances.

W5. Presentation.

Broken references and the legacy labels RQ/RQJ are fixed. Each condition has one name in the text, the tables and the figures, and each paragraph of Sec. 6 states one finding. The x-axis of Figures 2 and 3 is now the number of supervised pairs.

The gain of supervision survives a fixed scale, repeated draws, a second editor and supervision from other relations. The control you asked for separates two effects: a higher efficacy of the new fact, the main source of the gain in PCR, and a property-specific gain in recognition that only binary judgements show. The in-context control shows that the property is within reach of these models, so its generation after editing is an open objective. This is what the paper now claims, and the protocol with its two controls is what makes it measurable.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer VA7p, as the discussion period is coming to an end, we wanted to follow up on our response, which we hope addresses your concerns. We would be glad to clarify anything further before the deadline.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviewer buxf

Thank you for your reply. We appreciate that your main concerns have been addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.