acceptodds
Under review as a conference paper at ICLR 2027

From aggregate scores to cross-probe claims: Auditing knowledge-editing rankings

Abstract

Knowledge-editing benchmarks rank editors by averaging scores over several kinds of test question, or probes. A higher average shows one editor is ahead on the benchmark's chosen mix, not on every probe: when editors trade wins across probes, some reweighting reverses their ranking. Over two released benchmarks and a preregistered factorial we ask how often that happens and what the disagreements track: cross-probe disagreements are common, most fail at least one check tied to how the evaluation was built or scored, and the rest divide into real editor failures whose size the design cannot calibrate and a small residue no check here touches. One pair shows what such an average is made of: HalluEditBench's Generalization Score puts ROME 20.3 points above LoRA on Mistral-7B, yet LoRA leads by 21.2 on the rephrase question, because after LoRA's edit the model usually no longer answers yes/no questions in the required form (13.2% format compliance against ROME's 93.4%, from the same unedited model). Across that benchmark 12 of 17 rejections disappear without its open-answer rephrase questions†; on KnowEdit, one cell scores probe types on disjoint items, and two paired reversals survive every check the release allows. In the factorial, three large reversals between likelihood-ranked choice and token-F1 generation disappear under four other scores, including the teacher-forced accuracy KnowEdit applies. But which score shows a reversal is not stable across model and configuration: at EasyEdit's own FT-L configuration one survives two of three, and on the release's own backbone teacher-forced accuracy yields a corrected reversal where token F1 yields none. Splitting KnowEdit's pooled Reasoning label yields reversals real in sign whose magnitude a ceiling-against-chance comparison cannot calibrate. We release an audit reporting, per pair, the verdict, how far the ranking is from flipping, the largest reversal the data cannot exclude, and which confounds apply.

Primary area:
datasets and benchmarks
Keywords:
knowledge editing, benchmark evaluation, ranking robustness, cross-probe consistency, Pareto dominance, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_aXdLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether an editor that performs better overall also performs better across different evaluation probes. By utilizing HalluEditBench, KnowEdit, and a controlled experiment with 533 facts, the authors show that editor rankings can reverse depending on the probe, scoring method, and evaluation setup. They also introduce an audit tool to identify these inconsistencies and assess the robustness of ranking claims.

Critical strengths

The released-data analysis contains compelling examples. The KnowEdit analysis is also valuable, and statistical reporting is generally careful.

Critical weaknesses
  1. The experiments do not convincingly identify why rankings reverse. The authors show that reversals disappear under alternative scoring methods, but this does not establish that the original rankings were misleading. Different scoring rules measure different aspects of performance. Without an independent assessment of answer quality, it is unclear which ranking is more meaningful. This is my biggest concern because it limits the paper's main interpretation.
  2. The findings have limited generalizability. The controlled experiments mainly evaluate single edits on GPT-J-6B, with only one template per response format. They do not establish whether the conclusions hold across more diverse prompts, sequential editing, or multi-hop benchmarks such as MQuAKE-Remastered.
Questions for authors
  1. Can the ROME–FT-L comparison be reproduced on the exact crossed recent records and released prompts? If that is not feasible, will you revise the inference that implementations are the remaining explanation?
  2. What criterion distinguishes a scoring artifact from a legitimate change in measured utility, particularly for the joint rescoring result in Appendix J.4?
Reviewer_UCYjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether knowledge-editing rankings built on composite scores also hold across individual probe classes, using Gail–Simon and intersection-union tests for qualitative interaction plus four attribution checks (label pooling, ceiling/chance, item sets, target-as-gold credit). It re-analyses HalluEditBench and KnowEdit, runs a preregistered single-edit factorial on GPT-J-6B with KnowEdit's questions, and packages the procedure as an audit reporting verdicts, flip weights and minority-direction bounds.

Critical strengths

S1. ROME's 20.3-point Generalization lead over LoRA on Mistral-7B coexists with LoRA leading by 21.2 on rephrase, traced to post-edit yes/no format compliance (93.4% vs 13.2%); S2. On different-subject prompts the gradient editors rank the edit target above the prompt's own gold answer on 90.2%, 97.7% and 99.9% of cases against 6.4% for ROME.

Critical weaknesses

W1. The framing is post hoc. The registered hypothesis — format reverses editors, question type does not — returned only partial support, and both the registered specificity test and the answer-identity split went against it. §4.2 and H.4 state that the audit framing was adopted after those results. The preregistration is real, but it protects the contrast and the counts rather than the claim the paper now makes. §1 separately disclaims novelty for the test, Proposition 1, the leakage measure and the checks, and §7 describes three results as recoveries of known ones. W2. The controlled arms are not the editors they are named after. Three of the five editors are the authors' own implementations, described in §4.1 as over-trained: restoring EasyEdit's stopping rule cuts FT-M's repetition from 0.588 to 0.197 and its neighbourhood leakage from 0.976 to 0.651. AdaLoRA at the published rate diverged, so no result describes EasyEdit's published LoRA editor. The factorial also does not reproduce KnowEdit's two surviving reversals; its editors behave as the mirror image of the release's (§3). W3. The audit's prospective value is undemonstrated. Both prospective applications are uninformative by the paper's own account: Qwen2.5-7B-Instruct returns non-diagnostic, and HELM MMLU finds no failure among 45 model pairs. M2 carries no information about which pairs reverse (likelihood ratio 1.0, B.7), so the demonstrated output is the retrospective re-analysis of the two benchmarks that happened to release per-probe data.

Questions for authors

Q1. All four scores that place the gradient editors above ROME on restatement generation ignore what follows the first answer span, so is there any score that penalises repetition and still ranks them above ROME?

Reviewer_WuyPOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper distinguishes a ranking under a benchmark's chosen probe weights from the stronger claim that one editor has an advantage across probes. It reanalyzes HalluEditBench and KnowEdit, applies qualitative-interaction tests, and packages an audit reporting ranking reversals, sensitivity to reweighting, uncertainty about undetected reversals, and attribution flags. A controlled study evaluates five editors on 533 KnowEdit-derived facts using GPT-J-6B, with subsequent checks involving scoring rules, editor configurations, and other backbones.

Critical strengths
  1. The HalluEditBench example makes the motivation tangible: ROME's aggregate advantage over LoRA coexists with a substantial disadvantage on rephrasing, while their post-edit compliance with yes/no answers differs sharply. This shows why a single generalization score can obscure distinct behaviors.

  2. The study combines released results with paired controlled comparisons, rescoring, configuration checks, and analyses of dependence. All three primary measurement reversals survive clustering by relation, providing useful support for their existence on the original measurement scale.

Critical weaknesses
  1. Section 4.2 explains that the original claim—format changes reverse rankings while semantic changes do not—was not sustained, and the audit framing followed those results. This is a legitimate research development. However, the final paper does not sufficiently establish what the combined analyses add beyond documenting familiar forms of evaluation sensitivity. The related-work discussion acknowledges that several findings recover known phenomena, while both prospective audit applications are uninformative. The manuscript needs a clearer, demonstrated account of what an evaluator learns or can decide by using the combined audit beyond inspecting disaggregated results and standard sensitivity checks.

  2. M2 fires on all 84 HalluEditBench pairs. This does not invalidate its intended role as a warning about effect-size interpretation, but makes the prevalence of flags weak evidence of diagnostic usefulness. The leakage judgment changes with the prompt family for 7 of 11 reversals, and the experiment intended to vary leakage within an editor family fails to form its planned design. On Llama-2, M4 fires for one prompt family even though calibration increases the choice advantage. The authors appropriately describe these as flags rather than causes. Nevertheless, the paper provides limited evidence of how these warnings help an evaluator interpret a disagreement, beyond identifying additional sensitivities.

Questions for authors
  1. What is the most important new conclusion supported by the revised audit framing, beyond the individual sensitivity findings? Could one worked case show how the audit leads to a justified interpretation or evaluation decision that a probe-level table and standard sensitivity analysis would leave unresolved?

  2. Could the main paper summarize, for each central claim, the final supporting comparison, scoring rule, implementation, correction family, and unresolved alternative explanation? Please make the distinction between registered and final audit counts easy to follow, without requiring readers to reconstruct the sequence of experiments.

  3. For the central GPT-J and Llama-2 examples, what capability should each score be taken to measure? What evidence distinguishes an inappropriate scoring penalty from a real loss in the ability to produce a usable answer? Please explain what the evaluator should conclude when first-answer scoring and teacher-forced accuracy resolve different reversals on different backbones.

1 comment

Thank you for the substantial additional analyses. Identifying the padding defect provides a concrete example of diagnostic value beyond probe-level significance tests, and the distinction between recall and usability clarifies the interpretation. I will raise my rating.

Reviewer_cQ1nOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how knowledge editing benchmarks rank various editors in order to understand what is hidden behind the average score attributed to different knowledge editing approaches on that benchmark. The objective seems to be to explore whether the average scores over probes assigned to the editing methods reveal a true ranking of knowledge editing or some underlying quirk of the dataset construction. For example, the paper finds that in HalluEditBench, the generalization score attributed to ROME is higher than the generalization score attributed to LoRA, however this discrepancy mostly seems to be the result of LoRA breaking the expected format of the answers, and not necessarily because it is a less general approach to editing Mistral-7B. The paper then goes from this motivating example to conduct a larger scale analysis on 2 benchmarks to see whether the score disagreements (on various editing approaches) are robust to evaluation construction. As a controlled experiment, the paper also creates a variant of the KnowEdit benchmark to isolate how much of the score variance is explained by answering format (multiple choice, free response, yes/no) or type of probe (whether it's a restatement or consequence). The compare 5 different knowledge editing approaches and find that across probes, restatements tend to give higher scores than consequences, and multiple choice gives higher scores than free response, before concluding with some recommendations on reporting claims about knowledge editing methods across probes.

Critical strengths

The paper properly motivates why this is an important question as in general knowledge editing methods are chosen by glancing at an overall benchmark score, and the paper shows how those scores can be misleading and the paper's controlled edits are also well motivated and allow to make clearer judgements about how each knowledge editing method actually performs once you control for question type or answer format. The paper also makes a list of recommendations for benchmarks that are interpreted as providing rankings on different knowledge editing methods across cross-probe claims.

Critical weaknesses

The biggest weakness of the paper is that it is unclear about how to translate the findings on the specific controlled edit benchmark, to existing public knowledge editing benchmarks that are not controlled. The paper can be improved by going into details about what claims if any can be substantiated by these benchmarks and in what degree of confidence. The audit section was intended to partly address that but it is poorly written, seems bolted on and does not demonstrate effectiveness on any included application of it. Overall, the paper seems to have taken an interesting observation on a benchmark, run some controlled experiments by changing the benchmarking and isolating different factors but this is not a strong enough contribution. A stronger story would be a more detailed analysis exploring, for the rankings that hold or the rankings that flip, what about the knowledge editing technique leads to the difference.

Questions for authors

The conclusions that can be derived from the results in table 1 were already alluded to in the introduction and in the abstract by the observation of LoRA vs ROME editing Mistral 7-B on HalluEditBench, what additional insights, that generalize beyond the specific construction of editing GPT-J-6B on the controlled KnowEdit, do the results provide?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.