acceptodds
Under review as a conference paper at ICLR 2027

RankReclaim: Counting the Capacity an Edited Layer Has Actually Spent

Abstract

Lifelong model editing corrects a deployed model many times, and each correction arrives as a low-rank update to one layer of fixed width, the resource every later correction draws on. Editors decide when the layer is full by summing the ranks the corrections request, yet corrections share input directions: on one stream the requested total reaches by the eighteenth arrival while only directions are committed, and the consolidations that return width are executed before their cost to earlier corrections is read. We propose RankReclaim, which counts the directions a layer has actually committed instead of summing requested ranks, and prices every consolidation before it executes. A committed-capacity ledger reads the numerical rank of the stacked input factors and attributes to each correction the directions its removal would return, testing admission against the measured count. Priced reclamation evaluates a candidate truncation on the stored probes of every surviving correction and executes it only at the smallest rank whose accumulated displacement stays inside a tolerance, quoting the cost of returned width before it is paid. Overlap-budgeted placement splits each arrival's fit against the committed span and caps the share drawn from the free width, holding every arrival's draw at or under a set budget. The three components share one committed basis per layer and act on the one count the ledger produces. On three edit streams run to three times the per-layer budget, the requested-rank account serves , and corrections, the measured count alone , and at zero displacement, and RankReclaim , and . Among eight arms, RankReclaim is the only reclaiming arm that holds the committed count at the budget, , and of , and the only one that reads the displacement of every consolidation before executing it, returning width at to of the displacement of a fixed-rank consolidation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
lifelong model editing, low-rank weight updates, layer capacity accounting, edit consolidation, numerical rank, continual knowledge editing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kQeYOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RankReclaim, a capacity-aware framework for lifelong model editing that aims to accommodate more sequential edits under a fixed capacity budget. Its central idea is to measure the effective subspace capacity occupied by previous edits, rather than simply summing their requested ranks, and to manage this capacity through distortion-aware reclamation and constrained update placement.

RankReclaim consists of three components. First, a committed-capacity ledger estimates the occupied capacity using the numerical rank of stacked input factors, accounting for subspace overlap among edits. Second, priced reclamation evaluates candidate low-rank consolidations using stored probes and selects a truncation rank that keeps the accumulated displacement of previous edits within a prescribed tolerance. Third, overlap-budgeted placement constrains the component of each new update outside the currently occupied subspace, encouraging subspace reuse and limiting additional capacity consumption.

The authors evaluate the framework in a constructed single-layer numerical setting using editing streams derived from zsRE, CounterFact, and MQuAKE. The results suggest that measuring effective rank improves capacity utilization and that displacement-aware reclamation introduces less perturbation than fixed-rank consolidation. However, the evaluation does not involve actual editing of large language models, leaving the practical effectiveness of RankReclaim for lifelong knowledge editing insufficiently established.

Critical strengths
  1. A meaningful perspective on measuring capacity consumption in lifelong model editing.

The paper identifies an important limitation of conventional rank-based capacity accounting: the sum of requested edit ranks can overestimate actual capacity consumption when different edits occupy overlapping subspaces. By measuring the numerical rank of jointly committed updates, RankReclaim provides a more informative characterization of capacity utilization. This distinction is conceptually meaningful because it suggests that additional edits may be accommodated without proportionally increasing the occupied subspace dimension. The experimental results also support the effectiveness of this accounting mechanism in the authors' numerical setting.

  1. A coherent framework integrating capacity measurement, reclamation, and allocation.

Rather than addressing capacity estimation or low-rank compression in isolation, RankReclaim combines effective-rank accounting, displacement-aware reclamation, and overlap-constrained update placement into a unified framework. In particular, introducing an explicit displacement criterion for rank reduction is a useful design choice compared with selecting a fixed truncation rank without considering its effect on previous edits. This integration offers a systematic approach to managing limited editing capacity over a sequence of updates.

Critical weaknesses
  1. The experimental evaluation is insufficient to establish the method's effectiveness for real lifelong knowledge editing.

The most significant limitation is that the experiments are conducted in a simplified single-layer numerical setting rather than on actual large language models. Although the constructed editing streams are derived from established knowledge-editing datasets, their subspace overlap patterns may not faithfully represent the relationships among knowledge edits in real LLMs, where information is distributed across multiple layers and parameters exhibit complex superposition and nonlinear interactions. Consequently, the observed improvements in numerical capacity utilization may not generalize to realistic model-editing scenarios.

More importantly, admitting more edits does not necessarily imply preserving more knowledge or achieving better editing performance. The paper primarily demonstrates improvements in capacity utilization and consolidation-induced displacement, but does not provide sufficient end-to-end evidence of improved editing success, knowledge retention, locality, or resistance to interference. Without comprehensive evaluations on actual language models, it remains unclear whether the proposed capacity-management mechanisms translate into meaningful gains in lifelong editing quality. This substantially weakens the paper's central practical claim.

  1. The proposed preservation criterion and online setting leave important methodological and practical concerns unresolved.

The priced-reclamation mechanism measures the impact of consolidation through a limited set of stored probes. However, small output displacement on these probes does not guarantee that previous edits remain correct across semantically diverse inputs. Moreover, this criterion primarily captures perturbations introduced by consolidation, rather than the full range of interference caused by subsequent edits, conflicts among edited facts, or nonlinear effects in deep models. Therefore, the proposed displacement constraint should not be interpreted as a reliable guarantee against knowledge forgetting.

There is also a concern regarding the claimed online setting. As described in Appendices A.9 and A.12, the displacement tolerance (\delta) is determined using response statistics from the entire editing stream before the editing process begins. This introduces a dependence on future edit requests, which would generally be unavailable in a strictly online lifelong editing scenario. The authors should clarify whether this setting assumes advance access to the full editing stream and evaluate how the method performs when the tolerance must be selected without future information.

Finally, RankReclaim introduces additional computational operations, including SVD-based capacity estimation and the evaluation of candidate reclamation decisions. Yet the paper does not provide a sufficiently detailed analysis of runtime or memory overhead. Given that scalability is essential for lifelong editing of large models, the lack of practical efficiency measurements further limits the assessment of the framework's applicability.

Questions for authors

see weakness

4 comments

Dear Reviewer kQeY:

Thank you for recognizing the method novelty and the method design of our work. We have responded point by point to your concerns regarding the experiments on a pretrained LLM, the scope of the displacement criterion, the online estimate of the displacement tolerance, and the runtime and memory overhead. In the marked copy appended to the revised PDF (pages 33–64), the revisions made in response to your comments are tagged with @Reviewer kQeY. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1: Extend main comparison to a pretrained LLM layer with LoRA edits

A1: Following your suggestion, we further repeat the main comparison on a layer of a pretrained LLM (i.e., the MLP down-projection at layer 5 of Llama-3-8B, with 14336 inputs and 4096 outputs) with LoRA edits from counterfact, against three representative baselines (including adapter merging, null-space placement and MEMIT), in terms of qual, succ, loc, comm, slack, disp and served. The per-layer budget keeps the ratio of the reference layer to its input width (i.e., d=2389d = 2389), and the stream runs until its requested ranks total three times that budget (i.e., 7167), which takes 2880 edits. The overlap is present on this layer as well, where the LoRA input factors of the whole stream commit 4472 directions against a requested total of 7167. Table R1 shows that our method serves 1842 corrections within a committed count of 2388, against 941 for the null-space arm at 2389, and pays a displacement of 1.2291 against 2.5974 for adapter merging. These results indicate that the overlap and the gains of our method carry over to a pretrained LLM.

Table R1: Main comparison on counterfact with Llama-3-8B, at a budget of 2389.

Methodqual ↑succ ↑loc ↑commslack ↓disp ↓served ↑
Adapter-merging0.00600.00.393014821512.59742880
Null-space0.13190.04410.8375238900.0941
MEMIT0.17330.09760.5535238900.0947
Our method0.19640.10000.588923881491.22911842

Following your suggestions, we have added these results to Appendix A.19 (Lines 1617–1639), and pointed to them from Section 5.2 (Lines 357–369).


Summarized W2 (1): Scope of the displacement criterion beyond stored probes

A2 (1): We respectfully clarify that the displacement tolerance is enforced on the stored probes, and the change a consolidation causes on other inputs is bounded as well. First, on an input xx outside the stored probes, the residual left by a consolidation moves the response by at most the first discarded singular value σq+1\sigma_{q+1} times the input norm ∥x∥\lVert x \rVert, as stated in Section 4.3. Second, as you noted, the criterion prices consolidation, and interference from later edits lies outside its scope.

Following your suggestions, we have revised Section 4.3 (Lines 235–244) to make this clearer.


Summarized W2 (2): Extend to online estimate of the displacement tolerance

A2 (2): Following your suggestion, we further replace the full-stream displacement tolerance with an online estimate (i.e., the median over the arrivals seen so far), and compare the two estimates (including full-stream and online) across three streams (including zsre, counterfact and mquake) in terms of four readings (including qual, comm, disp and served). Table R2 shows that the online estimate stays within one served correction and 0.0119 in qual of the full-stream estimate on the three streams. These results indicate that with the online estimate, our method does not require the stream in advance.

Table R2: Full-stream and online estimates of the displacement tolerance.

StreamEstimatequal ↑commdisp ↓served ↑
zsreFull-stream0.0833161.003113
zsreOnline0.0833161.095713
counterfactFull-stream0.1905150.992915
counterfactOnline0.2024140.953015
mquakeFull-stream0.0750160.981115
mquakeOnline0.0750150.892716

Following your suggestions, we have added these results to Appendix A.22 (Lines 1670–1687), and pointed to them from Section 5.1 (Lines 343–346) and Appendix A.9 (Lines 1040–1045).

(continued)

Summarized W2 (3): Runtime and memory overhead across width bands

A2 (3): Following your suggestion, we further measure the per-arrival cost (including wall-clock time and peak memory) of our method against Allocation-only at the four width bands (i.e., 12, 16, 24 and 32) on zsre, on one CPU thread. Table R3 shows that the per-arrival cost of our method stays below 25 ms and 7 MB at the four width bands, where the widest band of 32 takes 21.83 ms and 6.52 MB per arrival against 2.89 ms and 3.35 MB for Allocation-only. These results indicate that our method is practical to deploy on long edit streams.

Table R3: Per-arrival cost across width bands.

Width bandAllocation-only (time)Our method (time)Allocation-only (memory)Our method (memory)
121.26 ms6.43 ms (×5.1)1.83 MB2.97 MB
161.57 ms9.01 ms (×5.7)2.08 MB3.68 MB
242.13 ms13.75 ms (×6.5)2.62 MB4.90 MB
322.89 ms21.83 ms (×7.6)3.35 MB6.52 MB

Following your suggestions, we have added these results to Appendix A.21 (Lines 1653–1667), and pointed to them from Appendix A.7 (Lines 938–946).


Best Wishes,

The Authors of Paper 52763

Thank you for your response. However, it does not address my concerns. In addition, your submission appears to contain a substantial amount of AI-generated content, particularly in the figures. I therefore maintain my recommendation to reject this paper. I also urge the authors to refrain from submitting papers containing such extensive AI-generated content in the future, as this only places an additional burden on the research community.

AuthorsOpenReview ↗

Second-Round Response to Reviewer kQeY

Dear Reviewer kQeY,

Thank you for your timely response and for your valuable comments on our work.

We sincerely appreciate your recognition that the paper offers a meaningful perspective on measuring capacity consumption in lifelong model editing, and that RankReclaim integrates capacity measurement, displacement-aware reclamation and overlap-constrained placement into a coherent framework.

We will incorporate the experiments and revisions from the rebuttal period into the main text, including the comparison on a pretrained LLM, the online estimate of the displacement tolerance, and the runtime and memory analysis.

Thank you again for your time and effort in reviewing our paper.

Best regards,

The Authors of Paper 52763

Reviewer_3kGGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RankReclaim manages sequential low-rank updates by counting overlapping directions, consolidating updates under a cumulative displacement limit, and constraining new update placement. Controlled single-layer experiments evaluate update admissions, consolidation damage, and retained editing quality.

Critical strengths
  • Counting overlapping directions admits more updates under the same budget without consolidation. This isolated comparison supports a useful improvement over summing requested ranks.
  • Consolidation accounts for accumulated displacement on all surviving probes. It achieves lower displacement and higher final quality than unpriced fixed-rank consolidation across all three streams, supporting the value of checking merge damage before execution.
Critical weaknesses
  • The capacity count depends on input factors, while model changes depend on the product of input and output factors. Equivalent factorizations can receive different counts even after row normalization. Its connection to actual storage or functional capacity remains insufficiently justified.
  • More admissions produce inconsistent gains in retained quality. Allocation without reclamation achieves higher final quality in 16 of 20 tested settings. The displacement limit covers consolidation, leaving interference from new edits partly uncontrolled and the benefit to long-term retention unclear.
Questions for authors
  • What operational resource does the capacity count measure, and how should equivalent factorizations with different numerical ranks be handled?
  • Could you evaluate each edit alone, within its merged update, and within the complete live set to separate installation error, consolidation loss, and later interference?
4 comments

Dear Reviewer 3kGG:

Thank you for recognizing the method effectiveness and the design value of our work. We have responded point by point to your concerns regarding the meaning of the capacity count, the gains in retained quality, the handling of equivalent factorizations, and the separation of the losses at stream end. In the marked copy appended to the revised PDF (pages 33–64), the revisions made in response to your comments are tagged with @Reviewer 3kGG. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1: Connection of capacity count to actual storage and functional capacity

A1: We respectfully clarify that the capacity count connects to both the functional capacity and the actual storage of the layer. First, for functional capacity, the count is the number of input directions in which the layer's response has already been changed by live corrections. The output factors decide what each change writes, while the input factors decide on which inputs each change acts, and a correction whose input factor is orthogonal to these committed directions leaves the layer's response unchanged on every input inside their span. Second, for actual storage, the summed update of the live set, written in one shared basis of its KK committed directions, takes K(n+m)K(n+m) parameters, with the input width nn and the output width mm, up to the components below the spectral tolerance θ\theta. We address equivalent factorizations in our response to your Q1.

Following your suggestions, we have revised Appendix A.4 (Lines 868–872) to make this clearer.


Summarized W2: Gains of more admissions in retained quality

A2: We respectfully clarify how admissions, retained quality and interference from new edits relate in our method, in three parts. First, our method trades part of the quality of consolidated corrections for more admissions under the same budget. Table R1 shows that across three streams (including zsre, counterfact and mquake), over three stream seeds, our method serves 13.3, 15.0 and 14.7 corrections against 8.7, 8.7 and 13.3 for Allocation-only (i.e., the measured count without reclamation), with a qual that is lower on zsre and higher on counterfact and mquake. Second, this trade-off accounts for the higher final quality of Allocation-only in 16 of 20 settings, because Allocation-only performs no consolidation and its served corrections are not displaced by consolidation, whereas our method returns width by consolidating live corrections within the displacement tolerance. Third, the displacement tolerance covers consolidation, and the interference from new edits is quantified in our response to your Q2, which evaluates each edit alone, within its merged update and within the complete live set to separate installation error, consolidation loss and later interference.

Table R1: Served corrections and quality at stream end, over three stream seeds.

StreamMethodServedqual ↑
zsreAllocation-only8.7 ± 0.60.1673 ± 0.0010
zsreOur method13.3 ± 0.60.0850 ± 0.0046
counterfactAllocation-only8.7 ± 1.20.1852 ± 0.0146
counterfactOur method15.0 ± 1.00.1894 ± 0.0076
mquakeAllocation-only13.3 ± 0.60.0658 ± 0.0057
mquakeOur method14.7 ± 0.60.0743 ± 0.0115

Following your suggestions, we have revised Section 5.3 (Lines 406–421), and updated the Limitations in Appendix A.0 (Lines 700–701).


Summarized Q1: Operational resource of the capacity count and handling of equivalent factorizations

A3: We respectfully clarify that the capacity count measures the number of input directions in which the layer's response has already been changed by live corrections, as detailed in our response to your W1. For equivalent factorizations, the same update keeps the same row space under any factorization, and the different numerical ranks you noted arise when singular values of the stacked input factor lie near the spectral tolerance θ\theta. We handle this by carrying each arrival in a canonical form, with the right singular vectors of its update as the input factor and the output factor absorbing the scale, such that the input factor keeps orthonormal rows and hence the unit row norm of Appendix A.4. We further show in Proposition 1 that, with every arrival in canonical form, the committed count after every event is a function of the target updates of the stream and not of the factorizations in which they were supplied. To check whether this choice of factorization changes the committed count and the corrections served, we compare the original and canonical factorizations across three streams (including zsre, counterfact and mquake). Table R2 shows that the committed count and the corrections served stay unchanged on the three streams.

(Continued)

Table R2: Original and canonical factorizations.

StreamFactorizationcommserved ↑
zsreOriginal1613
zsreCanonical1613
counterfactOriginal1515
counterfactCanonical1515
mquakeOriginal1615
mquakeCanonical1615

Following your suggestions, we have repeated the readings with each arrival in this canonical form, revised Section 4.2 (Lines 185–195) and Appendix A.7 (Lines 919–928) accordingly, added these results to Appendix A.23 (Lines 1690–1706), and pointed to them from Appendix A.4 (Lines 852–877).


Summarized Q2: Separation of installation error, consolidation loss and later interference

A4: Following your suggestion, we further evaluate each served correction alone, within its merged update and within the complete live set, and separate three losses (including installation error, consolidation loss and later interference) for two methods (including Allocation-only and our method) across three streams (including zsre, counterfact and mquake). Table R3 shows that later interference is the largest of the three losses at stream end for both methods on the three streams.

Table R3: Installation error, consolidation loss and later interference.

StreamMethodInstallation error ↓Consolidation loss ↓Later interference ↓
zsreAllocation-only0.00.00.6667
zsreOur method0.13460.35580.3942
counterfactAllocation-only0.00.00.5000
counterfactOur method0.08330.13330.5167
mquakeAllocation-only0.00.00.9038
mquakeOur method0.11670.26670.5167

Following your suggestions, we have added these results to Appendix A.24 (Lines 1725–1742), and pointed to them from the Limitations in Appendix A.0 (Lines 690–693).


Best Wishes,

The Authors of Paper 52763

Thank you for the detailed responses and additional experiments. The canonical factorization substantially addresses my concern about representation dependence, and the loss decomposition is helpful. However, my main concern about retained editing quality remains: compared with Allocation-only, the gains are modest on CounterFact and MQuAKE, while quality decreases substantially on zsRE. The additional results therefore do not yet establish that increased admissions translate into a convincing overall retention benefit. I appreciate the improvements, but will maintain my current rating.

AuthorsOpenReview ↗

Second-Round Response to Reviewer 3kGG

Dear Reviewer 3kGG:

Thank you for your timely reply, and we are glad that the canonical factorization and the loss decomposition address your concerns. As you noted, the average retained quality of our method is comparable to or slightly higher than Allocation-only on counterfact and mquake and lower on zsre, which is the cost of serving 2 to 7 more edits under the same budget, and we will state this trade-off in the Limitations. We understand that your main concern lies in whether our contribution stands beyond this reading, and below we restate our task, problem, method and experiments.

  • Task. Lifelong model editing updates a deployed LLM many times, for example to correct an outdated fact, without retraining it. Each edit is written as a low-rank update into one layer, and the capacity of this layer is limited by its width. The input is a stream of edits that arrive one at a time, and the output is an edited layer that admits as many edits as its capacity allows while preserving the earlier edits.
  • Problem. Existing editors estimate the used capacity of a layer by summing the ranks that edits request, but different edits often share directions, and the layer is regarded as full while part of its capacity remains unused.
  • Method. Our method measures the capacity a layer has actually used rather than summing the requested ranks, and admits more edits under the same budget. To realize this, we face three challenges, and each challenge is addressed by one module of our method. Table R4 shows that removing each module weakens its corresponding effect.

Table R4: Challenge, core design, and effect of removing each module.

ChallengeCore designWithout this module
Overlapping edits are counted more than onceCommitted-capacity ledger: counts only the capacity that the edits actually occupyServed ↑ 13 → 6
Merging edits to release capacity may damage earlier editsPriced reclamation: merges edits only when the damage to earlier edits stays within a toleranceQuality ↑ 0.1905 → 0.0595
A new edit may consume the remaining capacity at onceOverlap-budgeted placement: directs new edits to reuse occupied directions before drawing on the remaining capacityDrawn share ↓ 26.4% → 59.9%
  • Experiments. We conduct experiments on 3 edit datasets (i.e., zsre, counterfact and mquake) with a synthetic layer and a pretrained LLM (i.e., Llama-3-8B), compare against 7 baselines and 3 ablation variants, and use both quality metrics (e.g., quality, edit success and locality) and capacity metrics (e.g., committed count, displacement and edits served). In total, we conduct 10 types of experiments, reported in about 23 tables and 8 figures across the paper and this rebuttal, where the first three support our core claim (i.e., admitting more edits under the same budget, rather than improving the quality of each edit) and the other seven make the evaluation more thorough.
    • Experiment 1. Comparison with 7 baselines on 3 datasets, showing that our method makes full use of the layer capacity.
    • Experiment 2. Ablation study, showing the necessity of each module.
    • Experiment 3. Capacity tracking along the edit stream, showing that our method measures the used capacity correctly as edits arrive.
    • Experiment 4. Evaluation across four layer widths, showing that the gains hold for different layer sizes.
    • Experiment 5. Sensitivity and robustness analysis on hyperparameters, seeds and the online setting, showing the robustness of our method.
    • Experiment 6. Verification of the displacement bound, showing that the predicted damage safely bounds the actual damage.
    • Experiment 7. Boundary regimes, showing the conditions under which our method takes effect.
    • Experiment 8. Evaluation on a pretrained LLM, showing that the gains carry over to a realistic setting.
    • Experiment 9. Loss decomposition, showing where the loss of earlier edits comes from.
    • Experiment 10. Cost analysis, showing that the additional time and memory remain low.

We hope this restatement clarifies our contribution, and we look forward to your further feedback.

Best Wishes,

The Authors of Paper 52763

Reviewer_bvQHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies lifelong low-rank model editing and proposes RankReclaim, which counts the directions a fixed-width layer has committed instead of summing requested ranks. A committed-capacity ledger reads the numerical rank of the stacked input factors, priced reclamation truncates a consolidation at the smallest rank whose accumulated displacement stays inside a tolerance, and overlap-budgeted placement caps each arrival's free-width draw. Experiments use three edit streams seeded from zsre, counterfact and mquake at one layer of 96 inputs and 64 outputs, with seven baselines and three ablation arms.

Critical strengths

S1: Counting committed directions rather than requested ranks is a clean reframing. Eq. (3) and Eq. (4) make slack and per-correction attribution exact, and Figure 2 shows 48 requested against 30 committed directions by the eighteenth arrival.

S2: The ledger alone raises corrections served from 6, 7 and 6 to 9, 8 and 13 at zero displacement in Table 2, and RankReclaim reaches 13, 15 and 15 while keeping the committed count at 16, 15 and 16 of 16.

S3: Priced reclamation returns width at 26% to 36% of the fixed-rank displacement in Table 2, and the overlap-budgeted placement keeps the draw at or under the budget at all six points of Table 13. Appendix A.17 transparently lists every cell where another arm reads better.

Critical weaknesses

W1: All streams run at one layer of 96 inputs and 64 outputs, per Section 5.1. Repeating the main comparison of Table 1 on a layer of a pretrained LLM, with LoRA edits from zsre or counterfact, would show that the overlap and the gains carry over to a realistic setting.

W2: Every reading comes from one deterministic run, per Appendix A.13. Some gaps are small, such as quality 0.0750 against 0.0625 on mquake in Table 8. Reporting means and spread over several stream seeds would make the comparisons in Tables 1 and 2 easier to trust.

W3: Table 6 lists the added ledger SVD, group factorizations and probe evaluations only by their scaling. Reporting wall-clock time and memory against Allocation-only at the four width bands of Table 9 would help readers judge the per-arrival overhead of RankReclaim when it is deployed on long edit streams.

Questions for authors

Q1: Table 1 reads edit success 0.0 on zsre and mquake, and Appendix A.12 finds Allocation-only ahead in 16 of 20 rows of Table 7. How many served corrections are still reproduced at stream end?

Q2: The displacement tolerance is computed over the full stream before the loop, per Appendix A.9. How would an online estimate behave?

Q3: Table 17 reads quality 0.0 at d = n. Please explain this, and define the two primary streams used in Tables 3 and 9.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bvQH

Dear Reviewer bvQH:

Thank you for recognizing the task novelty, the method effectiveness, and the clear presentation of our work. We have responded point by point to your concerns regarding the experiments on a pretrained LLM, the results over multiple stream seeds, the runtime and memory cost, the retention of served corrections, the online estimate of the displacement tolerance, and the details of the evaluation settings. In the marked copy appended to the revised PDF (pages 33–64), the revisions made in response to your comments are tagged with @Reviewer bvQH. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1: Extend main comparison to a pretrained LLM layer with LoRA edits

A1: Following your suggestion, we further repeat the main comparison on a layer of a pretrained LLM (i.e., the MLP down-projection at layer 5 of Llama-3-8B, with 14336 inputs and 4096 outputs) with LoRA edits from counterfact, against three representative baselines (including adapter merging, null-space placement and MEMIT), in terms of qual, succ, loc, comm, slack, disp and served. The per-layer budget keeps the ratio of the reference layer to its input width (i.e., d=2389d = 2389), and the stream runs until its requested ranks total three times that budget (i.e., 7167), which takes 2880 edits. The overlap is present on this layer as well, where the LoRA input factors of the whole stream commit 4472 directions against a requested total of 7167. Table R1 shows that our method serves 1842 corrections within a committed count of 2388, against 941 for the null-space arm at 2389, and pays a displacement of 1.2291 against 2.5974 for adapter merging. These results indicate that the overlap and the gains of our method carry over to a pretrained LLM.

Table R1: Main comparison on counterfact with Llama-3-8B, at a budget of 2389.

Methodqual ↑succ ↑loc ↑commslack ↓disp ↓served ↑
Adapter-merging0.00600.00.393014821512.59742880
Null-space0.13190.04410.8375238900.0941
MEMIT0.17330.09760.5535238900.0947
Our method0.19640.10000.588923881491.22911842

Following your suggestions, we have added these results to Appendix A.19 (Lines 1617–1639), and pointed to them from Section 5.2 (Lines 357–369).


Summarized W2: Tables 1 and 2 over multiple stream seeds

A2: Following your suggestion, we further repeat the comparisons of Tables 1 and 2 over three stream seeds (i.e., the original seed and two new seeds for each of zsre, counterfact and mquake), and report the mean and spread (i.e., sample standard deviation) of each reading. Table R2 shows that our method still holds the comm at or near the budget on the three streams, reading 16.0 ± 0.0, 15.3 ± 0.6 and 16.0 ± 0.0 against at most 11.0 for both merging arms.

Table R2: Three-seed results of Table 1.

Methodzsre commzsre served ↑counterfact commcounterfact served ↑mquake commmquake served ↑
Adapter-merging11.0 ± 0.019.7 ± 2.98.7 ± 0.620.7 ± 0.69.0 ± 1.020.3 ± 1.5
Model-merging11.0 ± 0.019.7 ± 2.98.7 ± 0.620.7 ± 0.69.0 ± 1.020.3 ± 1.5
Null-space15.7 ± 0.67.3 ± 0.616.0 ± 0.06.7 ± 0.615.7 ± 0.66.3 ± 0.6
Orthogonal-adapter16.0 ± 0.06.0 ± 0.016.0 ± 0.07.0 ± 1.016.0 ± 0.06.0 ± 1.0
ROME14.7 ± 0.614.7 ± 0.615.7 ± 0.615.7 ± 0.615.0 ± 0.017.3 ± 0.6
MEMIT15.7 ± 0.65.3 ± 1.216.0 ± 0.07.0 ± 0.016.0 ± 0.06.0 ± 0.0
Norm-bounded15.7 ± 0.66.7 ± 0.615.7 ± 0.68.0 ± 0.015.0 ± 0.013.3 ± 0.6
Our method16.0 ± 0.013.3 ± 0.615.3 ± 0.615.0 ± 1.016.0 ± 0.014.7 ± 0.6

Table R3 shows that the small quality gap you noted holds in the three-seed means, where on mquake the qual of our method reads 0.0743 ± 0.0115 against 0.0658 ± 0.0057 for the measured count without reclamation. These results indicate that the comparisons in Tables 1 and 2 remain consistent across stream seeds.

(continued)

Table R3: Three-seed results of Table 2.

StreamMethodserved ↑returneddisp ↓qual ↑
zsreRequested-rank account5.7 ± 0.60.0 ± 0.00.0 ± 0.00.0887 ± 0.0047
zsreMeasured count, no reclamation8.7 ± 0.60.0 ± 0.00.0 ± 0.00.1673 ± 0.0010
zsreFixed-rank consolidation, unpriced15.7 ± 0.615.7 ± 1.23.8306 ± 0.04250.0317 ± 0.0036
zsreOur method13.3 ± 0.68.0 ± 1.00.9532 ± 0.07060.0850 ± 0.0046
counterfactRequested-rank account7.3 ± 0.60.0 ± 0.00.0 ± 0.00.1834 ± 0.0062
counterfactMeasured count, no reclamation8.7 ± 1.20.0 ± 0.00.0 ± 0.00.1852 ± 0.0146
counterfactFixed-rank consolidation, unpriced16.0 ± 0.011.3 ± 1.52.8277 ± 0.21570.0625 ± 0.0030
counterfactOur method15.0 ± 1.05.7 ± 0.60.9566 ± 0.05930.1894 ± 0.0076
mquakeRequested-rank account6.3 ± 0.60.0 ± 0.00.0 ± 0.00.0734 ± 0.0070
mquakeMeasured count, no reclamation13.3 ± 0.60.0 ± 0.00.0 ± 0.00.0658 ± 0.0057
mquakeFixed-rank consolidation, unpriced17.0 ± 0.06.0 ± 1.03.1159 ± 0.14280.0494 ± 0.0036
mquakeOur method14.7 ± 0.63.7 ± 0.60.9728 ± 0.04210.0743 ± 0.0115

Following your suggestions, we have replaced Tables 1 and 2 with these three-seed results (Lines 377–394 and 432–444), added them to Appendix A.20 (Lines 1641–1651), and pointed to them from Section 5.1 (Lines 349–352) and Appendix A.13 (Lines 1207–1218).


Summarized W3: Wall-clock time and memory against Allocation-only across width bands

A3: Following your suggestion, we further measure the per-arrival cost (including wall-clock time and peak memory) of our method against Allocation-only at the four width bands (i.e., 12, 16, 24 and 32) on zsre, on one CPU thread. Table R4 shows that the per-arrival cost of our method stays below 25 ms and 7 MB at the four width bands, where the widest band of 32 takes 21.83 ms and 6.52 MB per arrival against 2.89 ms and 3.35 MB for Allocation-only. These results indicate that our method is practical to deploy on long edit streams.

Table R4: Per-arrival cost across width bands.

Width bandAllocation-only (time)Our method (time)Allocation-only (memory)Our method (memory)
121.26 ms6.43 ms (×5.1)1.83 MB2.97 MB
161.57 ms9.01 ms (×5.7)2.08 MB3.68 MB
242.13 ms13.75 ms (×6.5)2.62 MB4.90 MB
322.89 ms21.83 ms (×7.6)3.35 MB6.52 MB

Following your suggestions, we have added these results to Appendix A.21 (Lines 1653–1667), and pointed to them from Appendix A.7 (Lines 938–946).


Summarized Q1: How many served corrections are still reproduced at stream end?

A4: We respectfully clarify that the edit success of 0.0 counts a correction as reproduced only when all eight of its probes are reproduced, and under this criterion the edit success of the eight arms in Table 1 stays at or below 0.2095 when each stream reaches three times the per-layer budget. As you suggested, we count the served corrections still reproduced at stream end across three streams (including zsre, counterfact and mquake) for two methods (including our method and Allocation-only), in terms of three counts (including served, fully reproduced on all eight probes, and partly reproduced on at least one probe). Table R5 shows that our method keeps more served corrections partly reproduced at stream end on the three streams, where 10, 12 and 8 of them are still partly reproduced against 9, 8 and 6 for Allocation-only.

Table R5: Served corrections still reproduced at stream end.

StreamMethodServedFully reproducedPartly reproduced
zsreAllocation-only919
zsreOur method13010
counterfactAllocation-only818
counterfactOur method15212
mquakeAllocation-only1306
mquakeOur method1508

Following your suggestions, we have added these results to Appendix A.24 (Lines 1709–1723).

(continued)

Summarized Q2: Extend to online estimate of the displacement tolerance

A5: Following your suggestion, we further replace the full-stream displacement tolerance with an online estimate (i.e., the median over the arrivals seen so far), and compare the two estimates (including full-stream and online) across three streams (including zsre, counterfact and mquake) in terms of four readings (including qual, comm, disp and served). Table R6 shows that the online estimate stays within one served correction and 0.0119 in qual of the full-stream estimate on the three streams. These results indicate that with the online estimate, our method does not require the stream in advance.

Table R6: Full-stream and online estimates of the displacement tolerance.

StreamEstimatequal ↑commdisp ↓served ↑
zsreFull-stream0.0833161.003113
zsreOnline0.0833161.095713
counterfactFull-stream0.1905150.992915
counterfactOnline0.2024140.953015
mquakeFull-stream0.0750160.981115
mquakeOnline0.0750150.892716

Following your suggestions, we have added these results to Appendix A.22 (Lines 1670–1687), and pointed to them from Section 5.1 (Lines 343–346) and Appendix A.9 (Lines 1040–1045).


Summarized Q3 (1): Why does quality drop to 0.0 at d = n?

A6 (1): We respectfully clarify that the quality drops to 0.0 when the per-layer budget dd equals the layer width nn, where the committed count settles at 96.0, all 96 input directions of the layer, and the reclamation still returns 4 directions at an accumulated displacement of 1.0245 (Appendix A.18, Lines 1555–1556; Table 17, Line 1596). For comparison, the quality reads 0.0833 at the default budget of 16.


Summarized Q3 (2): What are the two primary streams in Tables 3 and 9?

A6 (2): We respectfully clarify that the two primary streams are zsre and counterfact, and each cell of Tables 3 and 9 is the mean of the readings on these two streams.

Following your suggestions, we have revised Appendix A.13 (Lines 1207–1212) to make this clearer.


Best Wishes,

The Authors of Paper 52763

Dear Reviewer bvQH,

Thank you for your time in reviewing our work, and for recognizing that counting committed directions is a clean reframing.

We believe that our rebuttal has addressed your concerns, with the comparison on Llama-3-8B, the three-seed results, the runtime and memory measurements, and the online estimate of the displacement tolerance. As suggested, we will keep these experiments in the final version.

Thank you again for your constructive feedback.

Best Wishes,

The Authors of Paper 52763

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

We sincerely thank all reviewers for their time and constructive comments. Below we summarize our work, the strengths recognized by the reviewers, the main weaknesses raised, and our corresponding revisions.

Summary

Lifelong model editors decide when a layer is full by summing the ranks that edits request, which overstates the used width when edits share directions. Our method counts the width a layer has actually committed and prices each merge before executing it. Under the same budget, our method serves more than twice as many edits as the requested-rank account (Table 2, three stream seeds).

Strengths

  • Novel perspective. A clean reframing (bvQH) and a meaningful perspective on capacity consumption (kQeY).
  • Effective method. Admits more updates under the same budget (3kGG) and raises the corrections served (bvQH).
  • Sound design. A coherent framework (kQeY) that checks merge damage before execution (3kGG).
  • Clear presentation. Clear and easy to follow (bvQH).

Weaknesses

  1. Lack of experiments on a pretrained LLM. (bvQH, kQeY)
  2. The displacement tolerance needed the full stream. (bvQH, kQeY)
  3. Lack of runtime and memory measurements. (bvQH, kQeY)
  4. Single-seed results and no retention analysis. (bvQH, 3kGG)
  5. Unclear meaning of the capacity count and its dependence on the factorization. (3kGG)
  6. Unclear details of the displacement criterion and the evaluation streams. (bvQH, kQeY)

Revisions

  1. We added the main comparison on a layer of Llama-3-8B with LoRA edits, where our method serves 1842 corrections against 941 and 947 for the strongest baselines (Table R1 for bvQH, Table R1 for kQeY).
  2. We added an online estimate of the tolerance, within one served correction of the full-stream estimate (Table R6 for bvQH, Table R2 for kQeY).
  3. We added the per-arrival cost across four width bands, below 25 ms and 7 MB (Table R4 for bvQH, Table R3 for kQeY).
  4. We added three-seed results for Tables 1 and 2 and the retention at stream end (Tables R2, R3 and R5 for bvQH, Table R1 for 3kGG).
  5. We clarified the capacity count, showed it is unchanged under the canonical factorization, and separated installation error, consolidation loss and later interference (Tables R2 and R3 for 3kGG).
  6. We added the bound of the displacement criterion outside the stored probes and clarified the two primary streams.

After the rebuttal

  • Representation dependence of the capacity count: resolved, as confirmed by 3kGG.
  • Separation of the losses at stream end: confirmed as helpful by 3kGG.

All revisions are marked in the revised PDF (pages 33-64), and all experiments above will be kept in the final version.

Best Wishes,

The Authors of Paper 52763

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.