acceptodds
Under review as a conference paper at ICLR 2027

UAR-LoRA: Uncertainty-Aware Rank-Adaptive Low-Rank Adaptation for Lifelong Knowledge Editing

Abstract

Lifelong knowledge editing keeps a deployed large language model factually current one edit at a time, so thousands of corrections must accumulate without eroding earlier edits or unrelated knowledge. Existing low-rank adapter editors do not control, for each arriving edit, how much capacity its expert takes, which queries reach it and where it sits: a rank fixed before the stream starves hard corrections, a router without a calibrated threshold serves queries the pool should leave to the frozen base, and randomly initialised experts overlap earlier ones. We propose UAR-LoRA, an uncertainty-aware, rank-adaptive framework that treats capacity, admissibility and geometry as three conditions of one heuristic capacity-coherence criterion and meets them as each expert enters the pool. An Entropy Probe and Adaptive Rank Selector sizes each expert from the frozen base's predictive entropy, a Confidence-Gated Router returns queries below a calibrated threshold to the frozen base, and a Rank-Aware Orthogonal Initialiser places each new expert in directions no earlier expert occupies while free ones remain, with a truncated-SVD fallback once they run out. Averaged over three backbones on CounterFact at , UAR-LoRA reaches mean Reliability against for ELDER, and on LLaMA-2-7B it lowers Expected Calibration Error from to at a trainable-parameter ratio relative to fixed rank- LoRA. No single-module variant exceeds Reliability against with all three, and the probe matches a fixed rank of at rather than . Forgetting also accumulates more slowly: over ZsRE edits UAR-LoRA retains Reliability where ELDER falls to , losing points from the first edit against ELDER's .

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
lifelong knowledge editing, low-rank adaptation, entropy-guided rank allocation, calibrated confidence gating, orthogonal subspace initialization, mixture of LoRA experts

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_z2WPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UAR-LoRA, a framework for lifelong knowledge editing that combines three components: entropy-based rank selection, confidence-gated routing, and orthogonal initialization of LoRA experts. The authors evaluate UAR-LoRA on CounterFact, ZsRE, WikiBio, and MQuAKE-style tasks using three models. The reported results show improvements over existing methods in Reliability and Generality, with lower per-edit parameter costs. Ablation experiments are also provided to examine the contribution of each component.

Critical strengths

1.Clear Motivation and Useful Ablation Results: The three components address specific issues in adapter editing: rank allocation, routing, and interference between experts. The ablation results in Table 4 show that removing individual components leads to performance drops.

Critical weaknesses

1.Limited Novelty: The proposed method mainly combines three existing techniques: adaptive rank allocation (e.g., AdaLoRA), routing and deferral mechanisms (e.g., ELDER), and orthogonal low-rank constraints (e.g., O-LoRA). The overall novelty is limited.

2.Limited Experimental Coverage and Incomplete Baseline Comparisons: Table 1 provides a full comparison only on CounterFact with LLaMA-2-7B at T=1000. Table 2 includes additional backbones but still focuses on CounterFact and compares only against ELDER. The long-stream experiments on ZsRE are also limited to a single backbone and a few baselines. I suggest reporting more comprehensive results on CounterFact and ZsRE across multiple backbones and baselines at T=1000/5000, including stronger baselines such as LEMoE and WISE, as well as SoLA under more settings. Extending the experiments to T=10000 would also help assess scalability.

Questions for authors

1.Incomplete Reliability Evaluation: At T=5000, Reliability is still evaluated on a fixed subset of 1,000 historical edits rather than all 5,000 edits. This does not fully capture forgetting across the entire editing stream. The authors should report Reliability over all accumulated edits and provide results grouped by editing stage to examine whether older and newer edits are affected differently.

2.Insufficient Evidence for the Relationship Between Entropy and Edit Difficulty: Challenge 1 assumes that high-entropy edits require a larger rank. However, it is unclear whether higher predictive entropy actually indicates greater editing difficulty or a need for more parameters. The authors should analyze the relationship between entropy, required rank, and editing performance across edits of different difficulty levels. A random rank allocation baseline under the same parameter budget would also be useful.

3.Insufficient Experimental Analysis: (1) How does subspace occupancy evolve as more edits are added? Does it saturate quickly or gradually, and how does editing performance change before and after saturation? (2) The method introduces additional costs from routing and orthogonal projection. Training time and memory usage should be compared with those of the baselines. (3) Table 7 shows that performance is sensitive to the gating threshold. Does the same hyperparameter setting work consistently across different backbones and datasets, or is additional tuning required?

4.Presentation Issues: The text in Figures 1 and 2 is too small to read clearly. Also, (q(\cdot)) is introduced in Line 194 without an explicit definition. Its computation should be briefly explained in the main text rather than only in the appendix.

5.Code Availability: An anonymous code repository with the full experimental configurations would help reviewers verify and reproduce the reported results.

6 comments
AuthorsOpenReview ↗

Response to Reviewer z2WP

Dear Reviewer z2WP:

Thank you for recognizing the clear motivation, the method design, and the method effectiveness. We have responded point by point to your concerns (including novelty, more baselines and settings, evaluation over all edits, the difficulty signal, capacity and cost analysis, presentation, and code). We hope our responses resolve your concerns and look forward to further discussion. All revisions made in response to your comments are marked in blue in the revised manuscript.


Summarized W1: Limited novelty against AdaLoRA, ELDER and O-LoRA

A1: We respectfully clarify that what differs is when and on what each of the three decisions is taken, and that the three are coupled within one edit, while each technique exists on its own in AdaLoRA [1], ELDER [2] and O-LoRA [3]. Table R1 compares the three techniques with our method on each decision. The decisions are coupled because the rank chosen by the probe fixes the shape of the candidate that the initialiser projects, which requires the projection to hold for ranks that vary per edit, as Eq. (11) gives for any {r_s}, and because the confidence head is fitted on the edit that the new expert stores (Eq. (12)).

Table R1: Where each decision is taken, compiled from the method definitions of the paper.

Methodrankroutingorthogonality
AdaLoRA [1]allocated by importance (Section 1, A.1.2)not characterised in the papernot characterised in the paper
ELDER [2]fixed regardless of edit difficulty (A.12)a learned router within fixed mixtures (A.8), with a deferral that tracks no calibrated score (A.12)not characterised in the paper
O-LoRA [3]the same rank for every task (A.1.2)not characterised in the papera penalty held throughout training (A.1.2)
UAR-LoRA (Ours)per edit, before fitting, from frozen-model entropy (Eqs. (3), (4))separate confidence head, Platt-scaled, thresholded on held-out data (Eqs. (6) to (9))projection at placement, exact for the exact projector of Proposition 5 and up to the projector ridge of Table 9 as listed in Algorithm 1, while the guard has not fired; soft penalty afterwards (Eq. (12))

Following your suggestion, we have added this table to Appendix A.13 as Table 16 (Page 35, Lines 1865 to 1880).


Summarized W2: More backbones, datasets, baselines, and longer streams

A2: Following your suggestion, we further compare our method with four lifelong editors (including ELDER [2], WISE [4], LEMoE [5] and SoLA [6]) on two backbones (i.e., LLaMA-2-7B and Qwen2.5-7B) and two datasets (i.e., CounterFact and ZsRE) at T = 1000 and 5000, extend the ZsRE stream on LLaMA-2-7B to T = 10000, and report the gate threshold per setting. Table R2 shows that our method leads all four editors in Reliability and Generality in every setting of part 1, with Locality not ordered among the methods, and at T = 10000 its Reliability is 93.8 against 85.6 for ELDER and 83.9 for WISE, while every method is lower at T = 5000 than at T = 1000.

Table R2: Part 1. Coverage: two 7B backbones, two datasets, four lifelong editors and UAR-LoRA at T = 1000 and 5000.

Backbone / dataset / methodRel T=1000Gen T=1000Loc T=1000Rel T=5000Gen T=5000Loc T=5000
LLaMA-2-7B / CounterFact / ELDER [2]90.582.291.686.781.490.2
LLaMA-2-7B / CounterFact / WISE [4]91.981.893.889.280.087.2
LLaMA-2-7B / CounterFact / LEMoE [5]91.185.193.188.983.790.0
LLaMA-2-7B / CounterFact / SoLA [6]89.681.191.088.478.188.6
LLaMA-2-7B / CounterFact / UAR-LoRA (Ours)94.486.592.491.684.789.0
LLaMA-2-7B / ZsRE / ELDER [2]88.079.390.286.279.087.4
LLaMA-2-7B / ZsRE / WISE [4]87.881.490.984.077.086.9
LLaMA-2-7B / ZsRE / LEMoE [5]91.082.091.386.781.288.2
LLaMA-2-7B / ZsRE / SoLA [6]83.177.789.482.176.086.8
LLaMA-2-7B / ZsRE / UAR-LoRA (Ours)94.486.591.594.286.290.9
Qwen2.5-7B / CounterFact / ELDER [2]91.081.890.288.579.787.3
Qwen2.5-7B / CounterFact / WISE [4]90.583.091.489.881.388.6
Qwen2.5-7B / CounterFact / LEMoE [5]92.685.193.089.383.090.7
Qwen2.5-7B / CounterFact / SoLA [6]87.381.489.886.778.888.0
Qwen2.5-7B / CounterFact / UAR-LoRA (Ours)93.986.591.492.286.190.6
Qwen2.5-7B / ZsRE / ELDER [2]89.080.390.582.377.186.9
Qwen2.5-7B / ZsRE / WISE [4]88.181.191.984.079.487.8
Qwen2.5-7B / ZsRE / LEMoE [5]91.384.993.390.083.590.3
Qwen2.5-7B / ZsRE / SoLA [6]84.778.087.784.577.585.7
Qwen2.5-7B / ZsRE / UAR-LoRA (Ours)94.988.290.892.487.189.1

(Continued)

Table R2, part 2: ZsRE, LLaMA-2-7B, T = 10000.

MethodRelLoc
ELDER [2]85.686.0
WISE [4]83.985.8
UAR-LoRA (Ours)93.888.3

Table R2, part 3: ECE-minimising gate threshold of UAR-LoRA per cell, on the calibration stream of that cell's backbone and benchmark, grid {0.5, ..., 0.9}; the LLaMA-2-7B / CounterFact entry is the shared threshold of Section 5.1.

Backbone / datasettau
LLaMA-2-7B / CounterFact0.7
LLaMA-2-7B / ZsRE0.8
Qwen2.5-7B / CounterFact0.8
Qwen2.5-7B / ZsRE0.8

Following your suggestion, we have added the three parts of Table R2 to Appendix A.13 as Tables 17 to 19 (Pages 35 to 36, Lines 1890 to 1935).


Summarized Q1: Reliability over all accumulated edits and by stage

A3: Following your suggestion, we further report Reliability on all 5,000 edits of the T = 5000 ZsRE stream on LLaMA-2-7B, grouped by installation stage of 1,000 edits and scored after the last edit. Table R3 shows that Reliability over all 5,000 edits is 94.3 for our method against 87.4 for ELDER, and that later-installed edits hold Reliability at least as high as the first stage, from 94.2 to 94.4 for our method and from 86.2 to 88.0 for ELDER. The first stage is the fixed pool of 1,000 edits that every reported score uses, installed as the first 1,000 edits of the stream, and its score measures the retention of the earliest edits after all later ones. The nominal occupied fraction is counted from the mean rank and the cold start, and the last column counts the share of each stage past nominal saturation, which falls after edit 313 when the 2,496 directions left by the cold start are filled at the mean rank of 11.70.

Table R3: Reliability by installation stage.

StageRel UAR-LoRA (Ours)Rel ELDER [2]Nominal occupied fractionShare past nominal saturation
1-100094.286.21.0000.687
1001-200094.387.31.0001.000
2001-300094.387.41.0001.000
3001-400094.488.01.0001.000
4001-500094.488.01.0001.000
All 5,00094.387.41.0000.937

Following your suggestion, we have added Table R3 to Appendix A.13 as Table 20 (Page 36, Lines 1944 to 1956).


Summarized Q2: Relationship between entropy, required rank and edit difficulty

A4: Following your suggestion, we further relate entropy to the required rank (i.e., the capped sufficient rank) and to Reliability by entropy quartile, and compare the entropy policy with three controls (i.e., a random rank allocation, shuffled entropy scores and a fixed rank of 11), on CounterFact with LLaMA-2-7B at T = 1000. The fixed rank 11 is the largest integer rank whose Param does not exceed the entropy policy's, and the capped sufficient rank is the mean over a quartile of the smallest rank in R at which an edit installed alone (no cold-start rank) reaches its target, with an edit that reaches it at no rank of R scored 32, which mixes such edits with edits sufficient at rank 32 and the table does not count them. Table R4 shows that the capped sufficient rank rises from 5.7 to 13.9 across the entropy quartiles, and every control stays below the entropy policy in Reliability and Generality, where on the highest-entropy quartile the fixed rank falls to 84.0 while the entropy policy holds 94.0. The gap between this difficulty proxy and the true difficulty remains a boundary condition of our method, as stated in Appendix A.6.5.

Table R4: Part 1. Rank policies at the entropy policy's mean rank.

PolicyRelGenLocMean rankParam
Entropy (Ours)94.486.592.411.70.45
Random, same rank distribution91.684.291.111.70.45
Shuffled entropy90.884.592.111.70.45
Fixed rank 1189.884.292.1110.43

The random and shuffled policies use the rank distribution of the entropy policy.

Table R4, part 2: Entropy quartiles of the 1,000 edits (250 each).

QuartileEntropy (nats)Capped sufficient rankRel, entropy policyRel, fixed rank 11
Q10.885.794.496.4
Q21.178.895.290.4
Q31.989.494.088.4
Q42.6113.994.084.0

Following your suggestion, we have added both parts of Table R4 to Appendix A.13 as Tables 12 and 13 (Page 34, Lines 1817 to 1835).

Summarized Q3: Occupancy saturation, routing and projection cost, and threshold across settings

A5: We respectfully answer your three questions as follows.

(1) Under the conditions of Propositions 2 and 5 (each A_t of full row rank, pairwise orthogonality at placement, and the guard not fired), the idealised occupancy rises by the selected rank at every edit and saturates within a few hundred edits, with nominal saturation between edit 178 and edit 724 on a projection of width d = 4096, while the guard of Appendix A.6.1 fires earlier, at 0.95 d columns or on ill-conditioning. After placement, the soft penalty of Eq. (12) encourages orthogonality of the experts without guaranteeing it. In Table 3, T = 1 and 100 lie before saturation and T = 1000 and 5000 lie past it, under the truncated-SVD guard, and across this boundary the Reliability of our method goes from 95.9 and 94.6 to 94.4 and 94.2, and its Locality from 96.8 and 93.6 to 91.5 and 90.9. Table R3 above reports Reliability by stage past saturation.

(2) Following your suggestion, we further compare training time and memory with the six baselines of Table 1 other than Fine-Tune on CounterFact with LLaMA-2-7B at T = 1000 (batch size 1, bf16), under the same parameter accounting and with latencies derived from the frozen pass. The orthogonal projection runs once per edit at placement and takes 0.91 s of the 12.3 s per edit (Table 15 of the revised paper). Table R5 shows that our method takes 12.3 s per edit against 2.4 to 12.8 s for the baselines, and its pool grows with the stream to 33.9 GB, with an adapted pass of 170 ms over all stored experts, which is the cost of keeping one expert per edit that the per-edit parameter ratio does not show. ROME [7], MEMIT [8], GRACE [9], MELO [10] and SoLA [6] stay near the memory and latency of the frozen model, at a Reliability and Generality of at most 89.6 and 81.1 against 94.4 and 86.5 in Table 1. The time column is generated under the paper's protocol for every row, under which ELDER trains a mixture on each of the 96 adapted projections rather than on the six layers of its paper, and its 12.8 s is not comparable with the 2.12 s per edit that its paper reports on one 48GB NVIDIA A40.

Table R5: Time and memory of the baselines.

MethodParamPool GBPeak GB (lower bound)Time s/editLatency ms per pass
ROME [7]0.970.013.54.248
MEMIT [8]4.860.013.57.548
GRACE [9]1.000.0913.62.452
MELO [10]0.860.0813.64.949
SoLA [6]0.790.0713.64.357
ELDER [2]1.000.0913.612.849
UAR-LoRA (Ours)0.4533.947.412.3170 (admit), 48 (abstain)

(3) All reported settings share one gate threshold, namely tau = 0.7, selected once as the ECE minimiser of Table 7 on 500 held-out calibration paraphrases and held fixed across datasets and backbones, including every row of part 1 of Table R2. Part 3 of Table R2 shows that the ECE-minimising threshold of each setting moves within one grid step of it (0.7 to 0.8), and the shared 0.7 is the minimiser in one of the four settings.

Following your suggestion, we have added Table R5 to Appendix A.13 as Table 21 (Page 36, Lines 1958 to 1977).


Summarized Q4: Definition of q and figure text size

A6: We respectfully answer your two points as follows. (1) We have revised Section 4.2 to state in place that q min-max rescales the entropy H_t over the entropies of the W preceding edits, clipped to [0, 1], with a cold-start value d_0 for the first t_0 edits (Page 4, Lines 201 to 202). Written out as Eq. (14) in the appendix, q(H_t) = min(1, max(0, (H_t - min H) / (max H - min H))), with the whole prefix used until the window fills and d_t = d_0 if max H = min H, and W = 1000, t_0 = 100 and d_0 = 0.5. (2) We explain the two figures below, so that each element can be read without zooming in.

  • Figure 1 compares prior editors (left) with our method (right), one row per per-edit decision. In the first row, prior editors give every edit the same fixed rank, while our method sizes each expert from the frozen model's predictive entropy, so that easy edits receive a small rank and hard edits a large one. In the second row, prior editors route every query into the pool, while the confidence gate of our method returns queries below the threshold τ to the frozen model. In the third row, randomly initialised experts overlap the subspaces of earlier ones, while our method places each new expert orthogonally to them. The last row shows the outcome: interference compounds for prior editors, while our method keeps its Reliability over a stream of 5000 edits.
Reviewer_L8BoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenge of lifelong knowledge editing in large language models, where successive factual updates must preserve earlier edits and unrelated knowledge. The authors propose UAR-LoRA, an uncertainty-aware, rank-adaptive framework that coordinates expert capacity, query routing, and expert initialization. Specifically, it allocates expert ranks based on the frozen model’s predictive entropy, routes low-confidence queries back to the frozen model, and uses orthogonal initialization to reduce interference between experts. Experiments demonstrate improvements in editing reliability, calibration, and parameter efficiency. On CounterFact, UAR-LoRA achieves an average reliability of 92.9 across three backbones after 1,000 edits, compared with 90.4 for ELDER. On LLaMA-2-7B, it reduces expected calibration error from 0.19 to 0.11 while using only 44% of the trainable parameters of fixed rank-32 LoRA. It also exhibits less forgetting over longer editing sequences, retaining a reliability of 94.1 after 5,000 ZsRE edits, compared with 86.2 for ELDER.

Critical strengths

1)The problem studied in this paper is very meaningful. By simultaneously considering the success rate of editing, the generalization ability after rewriting, and the protection of irrelevant knowledge, it grasps the core contradiction in continuous knowledge updating. Discussing "how much capacity to allocate, which queries to use editing, and how to reduce expert interference" within the same framework has clear system design value.

2)This paper uses a gating mechanism for knowledge protection, and the independent confidence head separates "whether to use the expert pool" from "which experts to use". According to Equation (7), queries that are rejected from entering the expert pool do not accept adapter contributions at all, so the behavior of returning the base model has a clear implementation basis.

3)Experiments have demonstrated the effectiveness of the method, showing improved reliability and generalization compared to ELDER on GPT-2-XL, LLaMA-2-7B, and LLaMA-3-8B.

Critical weaknesses

1)The AI USE STATEMENT section is in the wrong place. Minor formatting issue: In line 257, the bolded words only have the first letter capitalized, but in line 289, the first letter of every word is capitalized.

2)It needs to be demonstrated whether predictive entropy can represent editing difficulty. The base model may be overly confident in erroneous facts, so low entropy does not necessarily mean easy modification; the uncertainty of the next token distribution may not necessarily represent the capacity required for long answers or multi-hop knowledge updates. The paper does not demonstrate the correlation between entropy and "minimum rank required to achieve the target effect," and lacks comparisons such as random assignment and shuffled entropy scores under the same average parameter budget.

3)The experiments in this paper have certain limitations. The evaluation is limited to GPT-2 and LLaMA model families. Evaluating additional model families, such as Qwen, would strengthen the evidence for cross-model generalization. Comparisons with closely related lifelong editing methods, such as WISE, are also missing.

4)The scalability of the method needs further proof. Based on an average rank of 11.3, each editor's adapter has approximately several megabytes of parameters, and 1,000 experts would accumulate to tens of gigabytes. If this average rank is maintained after 5,000 edits, the adapter alone would require hundreds of gigabytes. Furthermore, inference latency, memory usage, and other overhead also require further analysis.

Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer L8Bo

Dear Reviewer L8Bo:

Thank you for recognizing the meaningful problem, the independent confidence head, and the demonstrated effectiveness of our work. We have responded point by point to your concerns regarding entropy as a difficulty signal, model families and the WISE baseline, the scalability of storage and runtime, and formatting. We hope our responses resolve your concerns and look forward to further discussion. All revisions made in response to your comments are marked in blue in the revised manuscript.


Summarized W1: Placement of the AI Use Statement and heading capitalisation

A1: We sincerely thank the reviewer for pointing this out. We have revised the corresponding section accordingly.

Following your suggestion, we have revised the manuscript (Page 6, Line 293; Page 10, Line 486-500).


Summarized W2: Can predictive entropy represent editing difficulty?

A2: Following your suggestion, to check whether predictive entropy represents editing difficulty, we further compare the entropy policy with three controls (i.e., random ranks, shuffled entropy scores and a fixed rank of 11), and relate entropy to the required rank (i.e., the capped sufficient rank) and to Reliability by entropy quartile, on CounterFact with LLaMA-2-7B at T = 1000. The fixed rank 11 is the largest integer rank whose Param does not exceed the entropy policy's, and the capped sufficient rank is the mean over a quartile of the smallest rank in R at which an edit installed alone (no cold-start rank) reaches its target, with an edit that reaches it at no rank of R scored 32, which mixes such edits with edits sufficient at rank 32 and the table does not count them. Table R1 shows that every control stays below the entropy policy in Reliability and Generality, and the capped sufficient rank rises from 5.7 to 13.9 across the entropy quartiles, where on the highest-entropy quartile the fixed rank falls to 84.0 while the entropy policy holds 94.0. The cases you raise, a model confidently holding a wrong fact and long or multi-hop targets, are the gap between this difficulty proxy and the true difficulty that we name as a boundary condition of our method in Appendix A.6.5. On long-form and multi-hop targets, the margin over ELDER is 5.7 points in Generality on WikiBio and widens from 4.6 points at one hop to 11.1 at three, as shown in Tables 5 and 6.

Table R1: Part 1. Rank policies at the entropy policy's mean rank.

PolicyRelGenLocMean rankParam
Entropy (Ours)94.486.592.411.70.45
Random, same rank distribution91.684.291.111.70.45
Shuffled entropy90.884.592.111.70.45
Fixed rank 1189.884.292.1110.43

The random and shuffled policies use the rank distribution of the entropy policy.

Table R1, part 2: Entropy quartiles of the 1,000 edits (250 each).

QuartileEntropy (nats)Capped sufficient rankRel, entropy policyRel, fixed rank 11
Q10.885.794.496.4
Q21.178.895.290.4
Q31.989.494.088.4
Q42.6113.994.084.0

Following your suggestion, we have added both parts of Table R1 to Appendix A.13 as Tables 12 and 13 (Page 34, Lines 1811 to 1841).

(Continued)

Summarized W3: Evaluation on Qwen and comparison with WISE

A3: Following your suggestion, we further evaluate our method on the Qwen family (i.e., Qwen2.5-7B) beside LLaMA-2-7B and compare it with WISE [2], together with ELDER [1], LEMoE [3] and SoLA [4], on two datasets (i.e., CounterFact and ZsRE) at T = 1000 and 5000. Table R2 shows that our method leads WISE and the other lifelong editors in Reliability and Generality in every setting, with Locality not ordered among the methods, and on Qwen2.5-7B with CounterFact at T = 1000 its Reliability is 93.9 against 90.5 for WISE.

Table R2: Coverage: two 7B backbones, two datasets, four lifelong editors and UAR-LoRA at T = 1000 and 5000.

Backbone / dataset / methodRel T=1000Gen T=1000Loc T=1000Rel T=5000Gen T=5000Loc T=5000
LLaMA-2-7B / CounterFact / ELDER [1]90.582.291.686.781.490.2
LLaMA-2-7B / CounterFact / WISE [2]91.981.893.889.280.087.2
LLaMA-2-7B / CounterFact / LEMoE [3]91.185.193.188.983.790.0
LLaMA-2-7B / CounterFact / SoLA [4]89.681.191.088.478.188.6
LLaMA-2-7B / CounterFact / UAR-LoRA (Ours)94.486.592.491.684.789.0
LLaMA-2-7B / ZsRE / ELDER [1]88.079.390.286.279.087.4
LLaMA-2-7B / ZsRE / WISE [2]87.881.490.984.077.086.9
LLaMA-2-7B / ZsRE / LEMoE [3]91.082.091.386.781.288.2
LLaMA-2-7B / ZsRE / SoLA [4]83.177.789.482.176.086.8
LLaMA-2-7B / ZsRE / UAR-LoRA (Ours)94.486.591.594.286.290.9
Qwen2.5-7B / CounterFact / ELDER [1]91.081.890.288.579.787.3
Qwen2.5-7B / CounterFact / WISE [2]90.583.091.489.881.388.6
Qwen2.5-7B / CounterFact / LEMoE [3]92.685.193.089.383.090.7
Qwen2.5-7B / CounterFact / SoLA [4]87.381.489.886.778.888.0
Qwen2.5-7B / CounterFact / UAR-LoRA (Ours)93.986.591.492.286.190.6
Qwen2.5-7B / ZsRE / ELDER [1]89.080.390.582.377.186.9
Qwen2.5-7B / ZsRE / WISE [2]88.181.191.984.079.487.8
Qwen2.5-7B / ZsRE / LEMoE [3]91.384.993.390.083.590.3
Qwen2.5-7B / ZsRE / SoLA [4]84.778.087.784.577.585.7
Qwen2.5-7B / ZsRE / UAR-LoRA (Ours)94.988.290.892.487.189.1

Following your suggestion, we have added these results to Appendix A.13 as Table 17 (Page 35, Lines 1890 to 1914).


Summarized W4: Growth of storage, latency and memory with the stream

A4: We respectfully clarify that the stored pool grows linearly with the stream, to 33.9 GB at T = 1000 and 169.6 GB at T = 5000 at the revised mean rank of 11.70, while the parameter ratio of the paper is a per-edit ratio against a fixed rank-32 adapter, and Table R3 gives the pool at the two ends of the rank pool. For latency, every query runs the frozen pass (48 ms), and an admitted query then runs the adapted pass over all stored experts. Table R4 shows that our method holds less than half the pool memory of a fixed rank-32 pool with soft routing, and its adapted pass takes less than half the time, with 33.9 against 92.8 GB and 170 against 381 ms, on LLaMA-2-7B at batch size 1 in bf16, where the admit latency is the abstain latency scaled by (6.74B + pool) / 6.74B, without the frozen pass that the fixed pool also runs, and with no sequence length or hardware specified.

Table R3: Adapter storage, computed from the specified dimensions; router counts shown separately (the confidence head of d parameters excluded).

mean rankper expertpool at T=1000pool at T=5000
4 (min R)5,799,936 params, 11.6 MB5.8B params, 11.6 GB29.0B params, 58.0 GB
32 (max R)46,399,488 params, 92.8 MB46.4B params, 92.8 GB232.0B params, 464.0 GB
router Wsone row of 40964,096,000 params20,480,000 params

Table R4: Memory and time of our method and a fixed rank-32 pool.

MethodPool GBPeak GB (lower bound)Probe sProjection sFit sTotal s/editLatency ms (admit)Latency ms (abstain)
UAR-LoRA (Ours)33.947.40.0480.9111.312.317048
Fixed rank 32, soft routing92.8106.3-0.9212.713.6381-

Following your suggestion, we have added Tables R3 and R4 to Appendix A.13 as Tables 14 and 15 (Page 35, Lines 1836 to 1863), and Appendix A.11.8 states the linear growth of the pool (Page 33, Lines 1754 to 1756).

(Continued)

References

[1] Jiaang Li et al. ELDER: Enhancing lifelong model editing with mixture-of-LoRA. AAAI, 2025.

[2] Peng Wang et al. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. NeurIPS, 2024.

[3] Renzhi Wang and Piji Li. LEMoE: Advanced mixture of experts adaptor for lifelong model editing of large language models. EMNLP, 2024.

[4] Haihua Luo et al. Reversible lifelong model editing via semantic routing-based LoRA. arXiv, 2026.


Best Wishes,

The Authors

Dear Reviewer L8Bo,

Thank you for your time in reviewing our work, and for recognizing that it studies a meaningful problem with clear system design value, that the independent confidence head separates whether to use the pool from which expert to use, and that it improves over ELDER on three backbones.

We believe that our rebuttal has addressed your concerns, with the random-rank, shuffled-entropy and fixed-rank controls at no larger budget, the required rank by entropy quartile, the Qwen2.5-7B and WISE comparisons, and the storage, memory and latency measurements. As suggested, we will keep these experiments in the final version.

Thank you again for your constructive feedback.

Best Wishes,

The Authors of Paper 52780

Reviewer_7PveOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper propose UAR-Lora, an uncertainty awared rank adaptive framework. Experiment over baselines show some advantages.

Critical strengths

I don't find any strength of the paper.

Critical weaknesses

I am not really sure which problem the author aim to solve. The paper provide no background information for reader not familiar with the topic.

The "problem formulation" seems to be meaningless. It doesn't read like a constraint optimization problem. The presentation in Section 4 also looks very messy.

Moreover, this paper show a clear pattern of AI-generated paper. To list a few point:

  1. Fig.1 and Fig.2 show similar artifact as AI-generated image.

  2. There are many sentences in the paper read like AI-generated: e.g. Page 2 line 104: "Nothing in the score decides whether the pool should touch them at all." However, I don't have clear evidence that this paper is totally AI-generated.

Questions for authors

Could the author provide some background information on LIFELONG KNOWLEDGE EDITING?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7Pve

Dear Reviewer 7Pve:

Thank you for your careful review and constructive feedback on our work. We have responded point by point to your concerns regarding the problem the paper addressesand the structure of the method section. We hope our responses resolve your concerns and look forward to further discussion. All revisions made in response to your comments are marked in blue in the revised manuscript.


W1&Q1: Which problem the paper aims to solve, and background on lifelong knowledge editing

A1: Lifelong knowledge editing updates the facts of a deployed language model one edit at a time without retraining, and existing editors fall into three families: direct weight editors such as ROME (Meng et al., 2022) and MEMIT (Meng et al., 2023), memory-based editors such as GRACE (Hartvigsen et al., 2023), and adapter editors such as MELO (Yu et al., 2024), SoLA (Luo et al., 2026) and ELDER (Li et al., 2025). We understand that this comment concerns the task, the problem we address, the role of each component and the core contribution, which our writing did not make clear enough. Below we restate them; the revised manuscript adds the task and its three criteria to the opening of Appendix A.1.

  • Task. The input is a frozen LLM and a stream of edits arriving one at a time, each a query x_t with a new target y_t, and the output, after every edit, is the edited model, the frozen model together with a pool of LoRA experts. It is judged on three criteria: Reliability (the edited query returns y_t), Generality (paraphrases of the query also return it) and Locality (neighbouring facts keep the answers of the unedited model), under a parameter ceiling of one rank-32 adapter per edit on average (Eq. (2)).
  • Problem. Adapter editors add one low-rank expert per edit, but they do not decide, for each arriving edit, how much capacity its expert takes, which queries reach it and where it sits: a rank fixed before the stream starves hard corrections, a router without a calibrated threshold serves queries the pool should leave to the frozen model, and randomly initialised experts overlap earlier ones (Section 2, Challenges #1 to #3).
  • Method. UAR-LoRA makes these three decisions for every edit as its expert enters the pool, and this per-edit treatment is the core of our contribution. Each challenge is addressed by one module of our method. Table R1 shows that removing each module lowers the criterion it is designed for.

Table R1: Challenge, core design, and effect of removing each module (component ablation, CounterFact, LLaMA-2-7B, T = 1000).

ChallengeCore designWithout this module
A rank fixed before the stream starves hard correctionsEntropy Probe and Adaptive Rank Selector: sizes each expert from the frozen model's predictive entropyReliability ↓ 94.4 → 89.7 (rank 8); Param ↑ 0.45× → 1.09× (rank 32)
A router without a calibrated threshold serves queries the pool should leave to the frozen modelConfidence-Gated Router: returns queries below a calibrated confidence threshold to the frozen modelLocality ↓ 92.4 → 88.2
Randomly initialised experts overlap earlier onesRank-Aware Orthogonal Initialiser: places each new expert orthogonally to the earlier ones until a guard firesReliability ↓ 94.4 → 90.8
  • Experiments. We conduct experiments on 4 editing benchmarks (CounterFact, ZsRE, WikiBio and an MQuAKE-style multi-hop set) with 4 backbones (GPT-2-XL, LLaMA-2-7B, LLaMA-3-8B and Qwen2.5-7B), compare against 9 baselines (Fine-Tune, ROME, MEMIT, GRACE, MELO, SoLA, ELDER, WISE and LEMoE), and use both quality metrics (Reliability, Generality, Locality and ECE) and cost metrics (parameter ratio, time and memory). In total, we conduct 8 types of experiments, reported in 21 tables and 6 figures, where the first two support our core claim and the other six make the evaluation more thorough.
    • Experiment 1. Main comparison with the 7 baselines, showing that UAR-LoRA reaches 94.4 Reliability and 86.5 Generality against 90.5 and 82.2 for ELDER, with ECE 0.09 against 0.19, at a 0.45× parameter ratio.
    • Experiment 2. Ablation study, showing the necessity of each module (Table R1).
    • Experiment 3. Transfer to GPT-2-XL and LLaMA-3-8B, showing the generalization across backbones.
    • Experiment 4. Stream-length scaling on ZsRE up to T = 5000, and one stream of T = 10000, showing that forgetting accumulates more slowly than for ELDER.
    • Experiment 5. Long-form editing on WikiBio, showing a Generality margin of 5.7 points over ELDER.
    • Experiment 6. Multi-hop composition, showing that the margin over ELDER widens from 4.6 points at one hop to 11.1 at three.
    • Experiment 7. Sensitivity to the gate threshold, showing that calibration is tightest at τ = 0.7 (ECE 0.09).
    • Experiment 8. Cost analysis of storage, memory, time per edit and latency, showing the practicality of our method.

(Continued)

References

  • Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with GRACE: Lifelong model editing with discrete key-value adaptors. In Advances in Neural Information Processing Systems, volume 36, pp. 47934–47959. Curran Associates, Inc., 2023. doi: 10.52202/075280-2079.
  • Jiaang Li, Quan Wang, Zhongnan Wang, Yongdong Zhang, and Zhendong Mao. ELDER: Enhancing lifelong model editing with mixture-of-LoRA. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
  • Haihua Luo, Xuming Ran, Tommi Kärkkäinen, Zhonghua Chen, Jiangrong Shen, Qi Xu, and Fengyu Cong. Reversible lifelong model editing via semantic routing-based LoRA. arXiv preprint arXiv:2603.11239, 2026.
  • Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, 2022.
  • Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In International Conference on Learning Representations, 2023.
  • Lang Yu, Qin Chen, Jie Zhou, and Liang He. MELO: Enhancing model editing with neuron-indexed dynamic LoRA. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.

W2: The problem formulation as a constrained optimisation

A2: We respectfully clarify that the revised Section 3 states lifelong editing as a design objective under a parameter ceiling. Eq. (1) names the goal and Eq. (2) only restricts which allocations are admissible. Written out:

  • Decision variables: for every edit t, the expert factors A_t, B_t and its rank r_t, and the confidence head c_phi that decides whether a query is served by the pool.
  • Design objective, Eq. (1): the average over the stream of Rel_t + Gen_t + Loc_t, the three criteria evaluated after t edits.
  • Parameter ceiling, Eq. (2): the sum over t of r_t (d_in + d_out) is at most r_ref (d_in + d_out) T. Dividing both sides by (d_in + d_out) T gives (1/T) sum_t r_t <= r_ref: the mean rank of the stream may not exceed the reference rank r_ref = 32, i.e. on average an edit may not spend more than one fixed-rank adapter.

Following your suggestion, we have revised the manuscript (Page 3, Lines 130–152).


W3: Structure of Section 4

A3: We respectfully clarify that Section 4 introduces the three modules, one per subsection, and then their joint fitting.

  • Section 4.1 lists the three modules.

  • Section 4.2 (Challenge #1) has the entropy probe, rank bucketing and rank assignment, Eqs. (3) and (4).

  • Section 4.3 (Challenge #2) has the two heads, the gate and the confidence objective, Eqs. (5) to (9).

  • Section 4.4 (Challenge #3) has the orthogonal placement, Eqs. (10) and (11), and its saturation guard.

  • Section 4.5 fits the new expert and the two heads jointly under Eq. (12), with the rank already fixed by the probe. Algorithm 1 (Appendix A.4) gives the same procedure as one listing in five phases.


W4: AI-generated figures

A4: We respectfully clarify that Figure 2 and its enlarged panels, Figures 4 to 6, are plots drawn by a plotting script from the values of Table 7, Table 3 and the rank-pool sweep of Appendix A.2; the cumulative Locality drop at T = 500 and 2500 in Figure 2 (right) is printed only in the figure. Figures 1 and 3 are schematic diagrams of the method; their drawing was guided by a generative tool.


W5: AI-generated texts

A5: The sentence the reviewer quotes from line 104 makes a technical point, and the revised manuscript words it as "The score has no value that withholds the pool from such a query": a softmax router assigns unit probability mass over the stored experts (Eq. (5)), so its output has no value that means "no expert should serve this query". This is why Eq. (6) adds a separate scalar confidence head, and why the gate of Eq. (7) can return a query to the frozen model.

Following your suggestion, we have revised the manuscript (Page 2, Line 106).


Best Wishes,

The Authors

Dear Reviewer 7Pve,

Thank you for your time in reviewing our work.

We believe that our rebuttal has addressed your concerns, with the background on lifelong knowledge editing, the problem formulation stated as a design objective under a parameter ceiling, the structure of Section 4, and how each figure was produced. We will keep these clarifications in the final version.

Thank you again for your time.

Best Wishes,

The Authors of Paper 52780

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their time and comments. Line references refer to the revised PDF, where revisions are marked in blue.

Summary

Lifelong knowledge editing updates a deployed LLM one edit at a time with the backbone frozen, judged on Reliability, Generality and Locality, yet existing adapter editors fix the capacity of every expert before the stream and let a softmax router serve every query. UAR-LoRA decides three things for each arriving edit: the rank of its expert from frozen-model entropy, which queries reach the pool through a calibrated confidence head, and where the expert is placed through a rank-aware orthogonal projection. On CounterFact with LLaMA-2-7B at T = 1000 it reaches 94.4 Reliability and 86.5 Generality against 90.5 and 82.2 for ELDER, with ECE 0.09 against 0.19, at a per-edit parameter ratio of 0.45× relative to a fixed rank-32 adapter. The revision adds Qwen2.5-7B, WISE and LEMoE in a two-backbone, two-dataset comparison at T = 1000 and 5000, a T = 10000 stream, Reliability over all edits, entropy controls at no larger budget, and storage, latency and time per edit.

Strengths

  • Meaningful problem. Capacity, query admission and expert interference are treated in one framework, with clear system design value (L8Bo).
  • Well-founded gating. The independent confidence head separates whether to use the pool from which expert to use, and rejected queries receive no adapter contribution (L8Bo).
  • Motivation and ablations. The three components address specific issues, and removing each one lowers performance (z2WP); Reliability and Generality improve over ELDER on GPT-2-XL, LLaMA-2-7B and LLaMA-3-8B (L8Bo).

Weaknesses and Revisions

  1. Problem, background and formulation (7Pve). Added the task and its three criteria to the opening of Appendix A.1, and stated Section 3 as a design objective (Eq. (1)) under a parameter ceiling (Eq. (2)) (Lines 130–152). Section 4 gives one module per subsection, then joint fitting; Algorithm 1 (Appendix A.4) lists the procedure.
  2. Figures and wording (7Pve). Figure 2 and Figures 4 to 6 are drawn by a plotting script from Tables 7 and 3 and Appendix A.2; Figures 1 and 3 are schematic diagrams whose drawing was guided by a generative tool. The quoted sentence is reworded (Line 106).
  3. Entropy as a difficulty signal (L8Bo, z2WP). At no larger budget, random ranks, shuffled entropy and a fixed rank of 11 reach 91.6, 90.8 and 89.8 Reliability against 94.4, and are lower in Generality; the capped sufficient rank rises from 5.7 to 13.9 across entropy quartiles (Tables 12 and 13, Lines 1817–1835). Confidently held wrong facts are stated as a boundary condition (Appendix A.6.5).
  4. Backbones, datasets, baselines and stream length (L8Bo, z2WP). On Qwen2.5-7B and LLaMA-2-7B, CounterFact and ZsRE, T = 1000 and 5000, UAR-LoRA leads ELDER, WISE, LEMoE and SoLA in Reliability and Generality in all eight settings, and in Locality in two (Table 17, Lines 1890–1914). On ZsRE with LLaMA-2-7B at T = 10000, Reliability is 93.8 against 85.6 (ELDER) and 83.9 (WISE) (Table 18).
  5. Reliability over all edits (z2WP). Over all 5,000 ZsRE edits, Reliability is 94.3 against 87.4 for ELDER, and every stage of 1,000 edits lies between 94.2 and 94.4 (Table 20, Lines 1944–1956).
  6. Novelty (z2WP). Compared where AdaLoRA, ELDER, O-LoRA and UAR-LoRA take each decision: here all three are taken per edit and coupled within it, as the probed rank shapes the projected candidate and the confidence head is fitted on the stored edit (Table 16, Lines 1865–1880).
  7. Storage, latency and time (L8Bo, z2WP). The pool grows linearly, to 33.9 GB at T = 1000 and 169.6 GB at T = 5000, against at most 0.09 GB for the six baselines at T = 1000; an adapted pass takes 170 ms against 48 to 57 ms, and time is 12.3 s per edit against 2.4 to 12.8 s (Tables 14 and 15, Lines 1843–1863; Table 21, Lines 1958–1977; Appendix A.11.8, Lines 1768–1769).
  8. Saturation and threshold (z2WP). Past nominal saturation, ZsRE Reliability is 94.4 and 94.2 and Locality 91.5 and 90.9, against 95.9 and 94.6 and 96.8 and 93.6 before it (Table 3). One threshold, 0.7, is used throughout, and the per-setting optimum is 0.7 or 0.8 (Table 19).
  9. Presentation (L8Bo, z2WP). Revised the placement of the AI Use Statement and the heading capitalisation (Line 293, Lines 486–499); defined q in place in Section 4.2 (Lines 201–202), written out as Eq. (14) in the appendix.
  10. Code (z2WP). The paper specifies the full procedure (Algorithm 1, Tables 9 to 11, Appendices A.7 to A.10). We will release the full code with the camera-ready version.

All experiments above will be kept in the final version.

Best Wishes,

The Authors of Paper 52780

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.