- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RankReclaim, a capacity-aware framework for lifelong model editing that aims to accommodate more sequential edits under a fixed capacity budget. Its central idea is to measure the effective subspace capacity occupied by previous edits, rather than simply summing their requested ranks, and to manage this capacity through distortion-aware reclamation and constrained update placement.
RankReclaim consists of three components. First, a committed-capacity ledger estimates the occupied capacity using the numerical rank of stacked input factors, accounting for subspace overlap among edits. Second, priced reclamation evaluates candidate low-rank consolidations using stored probes and selects a truncation rank that keeps the accumulated displacement of previous edits within a prescribed tolerance. Third, overlap-budgeted placement constrains the component of each new update outside the currently occupied subspace, encouraging subspace reuse and limiting additional capacity consumption.
The authors evaluate the framework in a constructed single-layer numerical setting using editing streams derived from zsRE, CounterFact, and MQuAKE. The results suggest that measuring effective rank improves capacity utilization and that displacement-aware reclamation introduces less perturbation than fixed-rank consolidation. However, the evaluation does not involve actual editing of large language models, leaving the practical effectiveness of RankReclaim for lifelong knowledge editing insufficiently established.
Critical strengths
- A meaningful perspective on measuring capacity consumption in lifelong model editing.
The paper identifies an important limitation of conventional rank-based capacity accounting: the sum of requested edit ranks can overestimate actual capacity consumption when different edits occupy overlapping subspaces. By measuring the numerical rank of jointly committed updates, RankReclaim provides a more informative characterization of capacity utilization. This distinction is conceptually meaningful because it suggests that additional edits may be accommodated without proportionally increasing the occupied subspace dimension. The experimental results also support the effectiveness of this accounting mechanism in the authors' numerical setting.
- A coherent framework integrating capacity measurement, reclamation, and allocation.
Rather than addressing capacity estimation or low-rank compression in isolation, RankReclaim combines effective-rank accounting, displacement-aware reclamation, and overlap-constrained update placement into a unified framework. In particular, introducing an explicit displacement criterion for rank reduction is a useful design choice compared with selecting a fixed truncation rank without considering its effect on previous edits. This integration offers a systematic approach to managing limited editing capacity over a sequence of updates.
Critical weaknesses
- The experimental evaluation is insufficient to establish the method's effectiveness for real lifelong knowledge editing.
The most significant limitation is that the experiments are conducted in a simplified single-layer numerical setting rather than on actual large language models. Although the constructed editing streams are derived from established knowledge-editing datasets, their subspace overlap patterns may not faithfully represent the relationships among knowledge edits in real LLMs, where information is distributed across multiple layers and parameters exhibit complex superposition and nonlinear interactions. Consequently, the observed improvements in numerical capacity utilization may not generalize to realistic model-editing scenarios.
More importantly, admitting more edits does not necessarily imply preserving more knowledge or achieving better editing performance. The paper primarily demonstrates improvements in capacity utilization and consolidation-induced displacement, but does not provide sufficient end-to-end evidence of improved editing success, knowledge retention, locality, or resistance to interference. Without comprehensive evaluations on actual language models, it remains unclear whether the proposed capacity-management mechanisms translate into meaningful gains in lifelong editing quality. This substantially weakens the paper's central practical claim.
- The proposed preservation criterion and online setting leave important methodological and practical concerns unresolved.
The priced-reclamation mechanism measures the impact of consolidation through a limited set of stored probes. However, small output displacement on these probes does not guarantee that previous edits remain correct across semantically diverse inputs. Moreover, this criterion primarily captures perturbations introduced by consolidation, rather than the full range of interference caused by subsequent edits, conflicts among edited facts, or nonlinear effects in deep models. Therefore, the proposed displacement constraint should not be interpreted as a reliable guarantee against knowledge forgetting.
There is also a concern regarding the claimed online setting. As described in Appendices A.9 and A.12, the displacement tolerance (\delta) is determined using response statistics from the entire editing stream before the editing process begins. This introduces a dependence on future edit requests, which would generally be unavailable in a strictly online lifelong editing scenario. The authors should clarify whether this setting assumes advance access to the full editing stream and evaluate how the method performs when the tolerance must be selected without future information.
Finally, RankReclaim introduces additional computational operations, including SVD-based capacity estimation and the evaluation of candidate reclamation decisions. Yet the paper does not provide a sufficiently detailed analysis of runtime or memory overhead. Given that scalability is essential for lifelong editing of large models, the lack of practical efficiency measurements further limits the assessment of the framework's applicability.
Questions for authors
see weakness
4 comments
(continued)
Summarized W2 (3): Runtime and memory overhead across width bands
A2 (3): Following your suggestion, we further measure the per-arrival cost (including wall-clock time and peak memory) of our method against Allocation-only at the four width bands (i.e., 12, 16, 24 and 32) on zsre, on one CPU thread. Table R3 shows that the per-arrival cost of our method stays below 25 ms and 7 MB at the four width bands, where the widest band of 32 takes 21.83 ms and 6.52 MB per arrival against 2.89 ms and 3.35 MB for Allocation-only. These results indicate that our method is practical to deploy on long edit streams.
Table R3: Per-arrival cost across width bands.
| Width band | Allocation-only (time) | Our method (time) | Allocation-only (memory) | Our method (memory) |
|---|---|---|---|---|
| 12 | 1.26 ms | 6.43 ms (×5.1) | 1.83 MB | 2.97 MB |
| 16 | 1.57 ms | 9.01 ms (×5.7) | 2.08 MB | 3.68 MB |
| 24 | 2.13 ms | 13.75 ms (×6.5) | 2.62 MB | 4.90 MB |
| 32 | 2.89 ms | 21.83 ms (×7.6) | 3.35 MB | 6.52 MB |
Following your suggestions, we have added these results to Appendix A.21 (Lines 1653–1667), and pointed to them from Appendix A.7 (Lines 938–946).
Best Wishes,
The Authors of Paper 52763
Thank you for your response. However, it does not address my concerns. In addition, your submission appears to contain a substantial amount of AI-generated content, particularly in the figures. I therefore maintain my recommendation to reject this paper. I also urge the authors to refrain from submitting papers containing such extensive AI-generated content in the future, as this only places an additional burden on the research community.
Second-Round Response to Reviewer kQeY
Dear Reviewer kQeY,
Thank you for your timely response and for your valuable comments on our work.
We sincerely appreciate your recognition that the paper offers a meaningful perspective on measuring capacity consumption in lifelong model editing, and that RankReclaim integrates capacity measurement, displacement-aware reclamation and overlap-constrained placement into a coherent framework.
We will incorporate the experiments and revisions from the rebuttal period into the main text, including the comparison on a pretrained LLM, the online estimate of the displacement tolerance, and the runtime and memory analysis.
Thank you again for your time and effort in reviewing our paper.
Best regards,
The Authors of Paper 52763
Dear Reviewer kQeY:
Thank you for recognizing the method novelty and the method design of our work. We have responded point by point to your concerns regarding the experiments on a pretrained LLM, the scope of the displacement criterion, the online estimate of the displacement tolerance, and the runtime and memory overhead. In the marked copy appended to the revised PDF (pages 33–64), the revisions made in response to your comments are tagged with @Reviewer kQeY. We hope our responses resolve your concerns and look forward to further discussion.
Summarized W1: Extend main comparison to a pretrained LLM layer with LoRA edits
A1: Following your suggestion, we further repeat the main comparison on a layer of a pretrained LLM (i.e., the MLP down-projection at layer 5 of Llama-3-8B, with 14336 inputs and 4096 outputs) with LoRA edits from counterfact, against three representative baselines (including adapter merging, null-space placement and MEMIT), in terms of qual, succ, loc, comm, slack, disp and served. The per-layer budget keeps the ratio of the reference layer to its input width (i.e., d=2389), and the stream runs until its requested ranks total three times that budget (i.e., 7167), which takes 2880 edits. The overlap is present on this layer as well, where the LoRA input factors of the whole stream commit 4472 directions against a requested total of 7167. Table R1 shows that our method serves 1842 corrections within a committed count of 2388, against 941 for the null-space arm at 2389, and pays a displacement of 1.2291 against 2.5974 for adapter merging. These results indicate that the overlap and the gains of our method carry over to a pretrained LLM.
Table R1: Main comparison on counterfact with Llama-3-8B, at a budget of 2389.
Following your suggestions, we have added these results to Appendix A.19 (Lines 1617–1639), and pointed to them from Section 5.2 (Lines 357–369).
Summarized W2 (1): Scope of the displacement criterion beyond stored probes
A2 (1): We respectfully clarify that the displacement tolerance is enforced on the stored probes, and the change a consolidation causes on other inputs is bounded as well. First, on an input x outside the stored probes, the residual left by a consolidation moves the response by at most the first discarded singular value σq+1 times the input norm ∥x∥, as stated in Section 4.3. Second, as you noted, the criterion prices consolidation, and interference from later edits lies outside its scope.
Following your suggestions, we have revised Section 4.3 (Lines 235–244) to make this clearer.
Summarized W2 (2): Extend to online estimate of the displacement tolerance
A2 (2): Following your suggestion, we further replace the full-stream displacement tolerance with an online estimate (i.e., the median over the arrivals seen so far), and compare the two estimates (including full-stream and online) across three streams (including zsre, counterfact and mquake) in terms of four readings (including qual, comm, disp and served). Table R2 shows that the online estimate stays within one served correction and 0.0119 in qual of the full-stream estimate on the three streams. These results indicate that with the online estimate, our method does not require the stream in advance.
Table R2: Full-stream and online estimates of the displacement tolerance.
Following your suggestions, we have added these results to Appendix A.22 (Lines 1670–1687), and pointed to them from Section 5.1 (Lines 343–346) and Appendix A.9 (Lines 1040–1045).