- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes eHoReN, based on a simple observation: each edit only modifies one row in the external memory, but existing implementations repeatedly compute many things that are not affected by this edit. The authors divide these redundancies into three parts: editing computation, training states, and historical evaluation, and optimize them separately. The goal is to make continuously updating knowledge in LLMs and checking whether previous knowledge is affected faster and more memory-efficient.
Critical strengths
- Clear motivation and simple idea: The motivation is clear, and the core idea is simple and intuitive.
- Solid engineering: The method is solid and reasonable from an engineering perspective. The authors identify several engineering issues in methods like HoReN and fix them.
- Efficiency: The method improves efficiency with almost no performance loss, making model editing much more scalable.
Critical weaknesses
- Novelty: Although the engineering is solid, most of the improvements are engineering-level fixes, and the novelty is limited.
- Generality: The efficiency improvements mainly apply to methods like HoReN and are not general to all editing methods. Although the experiments on GRACE provide some support for generality, not all editing methods have these issues.
- Experimental results: I noticed that the main experiments (Table 1) only include Llama-3.1 8B / ZsRE and Qwen2.5 7B / WikiBigEdit. Also, the results on ZsRE and WikiBigEdit seem quite different. AlphaEdit performs much better than other methods on WikiBigEdit, but the paper does not explain this difference well.
- Missing evaluations: The paper does not report the Persona and multi-hop reasoning scores provided by the original WikiBigEdit benchmark.
Questions for authors
- Could the authors provide more detailed results for Llama-3.1 8B / WikiBigEdit and Qwen2.5 7B / ZsRE?
- Could the authors explain why the results on ZsRE and WikiBigEdit are so different?
- Could the authors provide the Persona and multi-hop reasoning scores on WikiBigEdit?