- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper formalizes property learning in knowledge editing: relational properties such as symmetry, inverse relations, and transitivity are injected through examples on one entity set and evaluated on disjoint entities. It introduces PropEdit-Bench, property-compliance and bipolar binary metrics, and a detailed study of supervision scale, batching, sequential editing, and preservation of earlier edit keys. The experiments use AlphaEdit across Llama and Qwen models in geospatial and social domains.
Critical strengths
-
Treating relational structure as a separate objective from isolated fact correction exposes a real weakness of existing editing benchmarks.
-
Disjoint supervision and evaluation entities reduce direct memorization, while completion and positive/negative binary questions reveal failures that a single answer format would miss.
-
The study varies model family and scale, domain, supervision amount, injection mode, and preservation constraints, and it openly reports specificity and efficacy costs rather than presenting compliance gains alone.
Critical weaknesses
-
All main experiments use AlphaEdit, so the results may reflect the geometry and failure modes of this specific locate-then-edit method. The paper does not establish whether property learning behaves similarly for other parameter-modifying or memory-based editors.
-
Table 2 reports the supervision-scale point with the highest PCR in each cell. Without a separate validation protocol for selecting the number of property pairs, these maxima can be optimistic and make cross-condition comparisons difficult to interpret.
-
Training and test examples retain the same relation names and domain. The update may generalize through relation-specific lexical or representation similarity without acquiring a domain-independent notion of symmetry or inversion. This concern is reinforced by specificity losses, occasional efficacy collapse, and relatively small absolute gains in high-baseline settings.
Questions for authors
-
Was the supervision scale selected on a held-out validation set? If not, how do conclusions change when a single scale is fixed before evaluating the test set or when selection bias is corrected?
-
Do the main findings reproduce with other editors and under cross-relation or cross-domain transfer, where the property must generalize beyond the same relation token and domain?
Response to Reviewer buxf
Thank you for the review. We answer each weakness with the matching question.
W1 / Q2 (editors). All main experiments use AlphaEdit.
Our scope is locate-then-edit editing, in which facts are written into the weights in closed form. AlphaEdit and MEMIT are the two representative editors of this family that write facts in batches, as our property phase requires. ROME writes one fact per rank-one update and cannot run it.
W2 / Q1. The supervision scale was selected on the test set.
No claim rests on that selection any more. Table 2 reports a single scale fixed in advance (n = 100), averaged over five independent draws of the supervised pairs, with a 95% bootstrap half-width on every number (App. A.1, Eq. 5). App. A.1 corrects for selection in two ways: the scale chosen on half of the test facts with the gain measured on the other half, and a max-T adjusted test. For Llama3-8B the gain holds under all three (+9.9 [7.1, 12.9] at n = 100; +5.2 to +10.9 cross-fitted; adjusted p < 0.001). Llama3-70B gains up to +1.7 [0.3, 3.5], and Qwen3-32B shows no gain.
W3 / Q2 (transfer). Supervision and test share relation names and domain.
Three new analyses answer this point.
This is the answer to RQ1: editing conveys a property in recognition but not yet in generation. Cross-domain transfer (Geo to Wikidata) was not run and is listed as future work (Sec. 7).
Specificity losses, efficacy collapse, small gains at high baselines. These are reported next to every result: Tables 10 and 12 give generalisation and specificity, and low efficacy is flagged in red. For some draws of the supervised pairs the efficacy of Qwen3-32B collapses, and it does so with unpaired facts as well (App. A.1). Gains are small where the baseline is high (at most +1.7 for Llama3-70B), and the text says so.