- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines whether an editor that performs better overall also performs better across different evaluation probes. By utilizing HalluEditBench, KnowEdit, and a controlled experiment with 533 facts, the authors show that editor rankings can reverse depending on the probe, scoring method, and evaluation setup. They also introduce an audit tool to identify these inconsistencies and assess the robustness of ranking claims.
Critical strengths
The released-data analysis contains compelling examples. The KnowEdit analysis is also valuable, and statistical reporting is generally careful.
Critical weaknesses
- The experiments do not convincingly identify why rankings reverse. The authors show that reversals disappear under alternative scoring methods, but this does not establish that the original rankings were misleading. Different scoring rules measure different aspects of performance. Without an independent assessment of answer quality, it is unclear which ranking is more meaningful. This is my biggest concern because it limits the paper's main interpretation.
- The findings have limited generalizability. The controlled experiments mainly evaluate single edits on GPT-J-6B, with only one template per response format. They do not establish whether the conclusions hold across more diverse prompts, sequential editing, or multi-hop benchmarks such as MQuAKE-Remastered.
Questions for authors
- Can the ROME–FT-L comparison be reproduced on the exact crossed recent records and released prompts? If that is not feasible, will you revise the inference that implementations are the remaining explanation?
- What criterion distinguishes a scoring artifact from a legitimate change in measured utility, particularly for the joint rescoring result in Appendix J.4?
Thank you for the substantial additional analyses. Identifying the padding defect provides a concrete example of diagnostic value beyond probe-level significance tests, and the distinction between recall and usability clarifies the interpretation. I will raise my rating.