- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies performance degradation in lifelong knowledge editing for LLMs. The authors identify inference-time memory retrieval and execution as important bottlenecks in existing residual-memory editing methods. To address these issues, they propose RECON, which replaces hash-based retrieval with semantic routing and introduces Retrieval-Conditioned Norm Calibration to mitigate interference caused by write collisions in shared residual parameters. Experiments demonstrate improvements in OOD generalization and long-horizon sequential editing.
Critical strengths
-
The paper offers a useful perspective on the performance bottlenecks of residual-memory editing by focusing on memory readout. This provides insights into the limitations of existing methods and potential directions for improvement. RECON is also lightweight, requiring no changes to the original write process or additional training of the retrieval module.
-
The experiments cover multiple models, benchmarks, and editing horizons, with promising results in OOD generalization and long-horizon sequential editing.
Critical weaknesses
-
Although the ablation studies show the effectiveness of semantic routing, the paper lacks a systematic analysis of when and why the original hash-based routing fails, and to what extent these failures contribute to editing performance degradation.
-
The authors observe a correlation between write collisions and residual column norms. However, the current theoretical and empirical evidence does not sufficiently establish whether norm inflation is a key cause of editing performance degradation, nor does it directly demonstrate how the proposed calibration mitigates interference among different edits.
-
Semantic routing requires storing semantic representations for all edits and performing similarity-based retrieval. At larger editing scales, its retrieval latency is substantially higher than that of hash-based routing. Although the authors suggest approximate nearest-neighbor search as a potential acceleration strategy, they do not evaluate it in practice, leaving the trade-off between computational overhead and performance gains unclear.
Questions for authors
-
I would like to see a more detailed analysis of the relationship between retrieval errors and editing failures, particularly for the original hash-based routing. Comparisons across different semantic representations and similarity metrics would also help establish the necessity of the proposed routing design. Similarly, further analysis of whether norm inflation is a key factor in editing degradation would strengthen the motivation for Norm Calibration.
-
The largest gains from Semantic Routing are observed on Temporal, which contains at most 75 sequential edits. In contrast, its contribution appears relatively limited in the 15,000-edit setting on ZsRE. Does this suggest that Semantic Routing is primarily beneficial under distribution shift rather than at longer editing horizons? How would the method perform when long editing sequences and OOD queries are considered together?