- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes UAR-LoRA, a framework for lifelong knowledge editing that combines three components: entropy-based rank selection, confidence-gated routing, and orthogonal initialization of LoRA experts. The authors evaluate UAR-LoRA on CounterFact, ZsRE, WikiBio, and MQuAKE-style tasks using three models. The reported results show improvements over existing methods in Reliability and Generality, with lower per-edit parameter costs. Ablation experiments are also provided to examine the contribution of each component.
Critical strengths
1.Clear Motivation and Useful Ablation Results: The three components address specific issues in adapter editing: rank allocation, routing, and interference between experts. The ablation results in Table 4 show that removing individual components leads to performance drops.
Critical weaknesses
1.Limited Novelty: The proposed method mainly combines three existing techniques: adaptive rank allocation (e.g., AdaLoRA), routing and deferral mechanisms (e.g., ELDER), and orthogonal low-rank constraints (e.g., O-LoRA). The overall novelty is limited.
2.Limited Experimental Coverage and Incomplete Baseline Comparisons: Table 1 provides a full comparison only on CounterFact with LLaMA-2-7B at T=1000. Table 2 includes additional backbones but still focuses on CounterFact and compares only against ELDER. The long-stream experiments on ZsRE are also limited to a single backbone and a few baselines. I suggest reporting more comprehensive results on CounterFact and ZsRE across multiple backbones and baselines at T=1000/5000, including stronger baselines such as LEMoE and WISE, as well as SoLA under more settings. Extending the experiments to T=10000 would also help assess scalability.
Questions for authors
1.Incomplete Reliability Evaluation: At T=5000, Reliability is still evaluated on a fixed subset of 1,000 historical edits rather than all 5,000 edits. This does not fully capture forgetting across the entire editing stream. The authors should report Reliability over all accumulated edits and provide results grouped by editing stage to examine whether older and newer edits are affected differently.
2.Insufficient Evidence for the Relationship Between Entropy and Edit Difficulty: Challenge 1 assumes that high-entropy edits require a larger rank. However, it is unclear whether higher predictive entropy actually indicates greater editing difficulty or a need for more parameters. The authors should analyze the relationship between entropy, required rank, and editing performance across edits of different difficulty levels. A random rank allocation baseline under the same parameter budget would also be useful.
3.Insufficient Experimental Analysis: (1) How does subspace occupancy evolve as more edits are added? Does it saturate quickly or gradually, and how does editing performance change before and after saturation? (2) The method introduces additional costs from routing and orthogonal projection. Training time and memory usage should be compared with those of the baselines. (3) Table 7 shows that performance is sensitive to the gating threshold. Does the same hyperparameter setting work consistently across different backbones and datasets, or is additional tuning required?
4.Presentation Issues: The text in Figures 1 and 2 is too small to read clearly. Also, (q(\cdot)) is introduced in Line 194 without an explicit definition. Its computation should be briefly explained in the main text rather than only in the appendix.
5.Code Availability: An anonymous code repository with the full experimental configurations would help reviewers verify and reproduce the reported results.
6 comments
(Continued)
Table R2, part 2: ZsRE, LLaMA-2-7B, T = 10000.
| Method | Rel | Loc |
|---|---|---|
| ELDER [2] | 85.6 | 86.0 |
| WISE [4] | 83.9 | 85.8 |
| UAR-LoRA (Ours) | 93.8 | 88.3 |
Table R2, part 3: ECE-minimising gate threshold of UAR-LoRA per cell, on the calibration stream of that cell's backbone and benchmark, grid {0.5, ..., 0.9}; the LLaMA-2-7B / CounterFact entry is the shared threshold of Section 5.1.
| Backbone / dataset | tau |
|---|---|
| LLaMA-2-7B / CounterFact | 0.7 |
| LLaMA-2-7B / ZsRE | 0.8 |
| Qwen2.5-7B / CounterFact | 0.8 |
| Qwen2.5-7B / ZsRE | 0.8 |
Following your suggestion, we have added the three parts of Table R2 to Appendix A.13 as Tables 17 to 19 (Pages 35 to 36, Lines 1890 to 1935).
Summarized Q1: Reliability over all accumulated edits and by stage
A3: Following your suggestion, we further report Reliability on all 5,000 edits of the T = 5000 ZsRE stream on LLaMA-2-7B, grouped by installation stage of 1,000 edits and scored after the last edit. Table R3 shows that Reliability over all 5,000 edits is 94.3 for our method against 87.4 for ELDER, and that later-installed edits hold Reliability at least as high as the first stage, from 94.2 to 94.4 for our method and from 86.2 to 88.0 for ELDER. The first stage is the fixed pool of 1,000 edits that every reported score uses, installed as the first 1,000 edits of the stream, and its score measures the retention of the earliest edits after all later ones. The nominal occupied fraction is counted from the mean rank and the cold start, and the last column counts the share of each stage past nominal saturation, which falls after edit 313 when the 2,496 directions left by the cold start are filled at the mean rank of 11.70.
Table R3: Reliability by installation stage.
| Stage | Rel UAR-LoRA (Ours) | Rel ELDER [2] | Nominal occupied fraction | Share past nominal saturation |
|---|---|---|---|---|
| 1-1000 | 94.2 | 86.2 | 1.000 | 0.687 |
| 1001-2000 | 94.3 | 87.3 | 1.000 | 1.000 |
| 2001-3000 | 94.3 | 87.4 | 1.000 | 1.000 |
| 3001-4000 | 94.4 | 88.0 | 1.000 | 1.000 |
| 4001-5000 | 94.4 | 88.0 | 1.000 | 1.000 |
| All 5,000 | 94.3 | 87.4 | 1.000 | 0.937 |
Following your suggestion, we have added Table R3 to Appendix A.13 as Table 20 (Page 36, Lines 1944 to 1956).
Summarized Q2: Relationship between entropy, required rank and edit difficulty
A4: Following your suggestion, we further relate entropy to the required rank (i.e., the capped sufficient rank) and to Reliability by entropy quartile, and compare the entropy policy with three controls (i.e., a random rank allocation, shuffled entropy scores and a fixed rank of 11), on CounterFact with LLaMA-2-7B at T = 1000. The fixed rank 11 is the largest integer rank whose Param does not exceed the entropy policy's, and the capped sufficient rank is the mean over a quartile of the smallest rank in R at which an edit installed alone (no cold-start rank) reaches its target, with an edit that reaches it at no rank of R scored 32, which mixes such edits with edits sufficient at rank 32 and the table does not count them. Table R4 shows that the capped sufficient rank rises from 5.7 to 13.9 across the entropy quartiles, and every control stays below the entropy policy in Reliability and Generality, where on the highest-entropy quartile the fixed rank falls to 84.0 while the entropy policy holds 94.0. The gap between this difficulty proxy and the true difficulty remains a boundary condition of our method, as stated in Appendix A.6.5.
Table R4: Part 1. Rank policies at the entropy policy's mean rank.
| Policy | Rel | Gen | Loc | Mean rank | Param |
|---|---|---|---|---|---|
| Entropy (Ours) | 94.4 | 86.5 | 92.4 | 11.7 | 0.45 |
| Random, same rank distribution | 91.6 | 84.2 | 91.1 | 11.7 | 0.45 |
| Shuffled entropy | 90.8 | 84.5 | 92.1 | 11.7 | 0.45 |
| Fixed rank 11 | 89.8 | 84.2 | 92.1 | 11 | 0.43 |
The random and shuffled policies use the rank distribution of the entropy policy.
Table R4, part 2: Entropy quartiles of the 1,000 edits (250 each).
| Quartile | Entropy (nats) | Capped sufficient rank | Rel, entropy policy | Rel, fixed rank 11 |
|---|---|---|---|---|
| Q1 | 0.88 | 5.7 | 94.4 | 96.4 |
| Q2 | 1.17 | 8.8 | 95.2 | 90.4 |
| Q3 | 1.98 | 9.4 | 94.0 | 88.4 |
| Q4 | 2.61 | 13.9 | 94.0 | 84.0 |
Following your suggestion, we have added both parts of Table R4 to Appendix A.13 as Tables 12 and 13 (Page 34, Lines 1817 to 1835).
Summarized Q3: Occupancy saturation, routing and projection cost, and threshold across settings
A5: We respectfully answer your three questions as follows.
(1) Under the conditions of Propositions 2 and 5 (each A_t of full row rank, pairwise orthogonality at placement, and the guard not fired), the idealised occupancy rises by the selected rank at every edit and saturates within a few hundred edits, with nominal saturation between edit 178 and edit 724 on a projection of width d = 4096, while the guard of Appendix A.6.1 fires earlier, at 0.95 d columns or on ill-conditioning. After placement, the soft penalty of Eq. (12) encourages orthogonality of the experts without guaranteeing it. In Table 3, T = 1 and 100 lie before saturation and T = 1000 and 5000 lie past it, under the truncated-SVD guard, and across this boundary the Reliability of our method goes from 95.9 and 94.6 to 94.4 and 94.2, and its Locality from 96.8 and 93.6 to 91.5 and 90.9. Table R3 above reports Reliability by stage past saturation.
(2) Following your suggestion, we further compare training time and memory with the six baselines of Table 1 other than Fine-Tune on CounterFact with LLaMA-2-7B at T = 1000 (batch size 1, bf16), under the same parameter accounting and with latencies derived from the frozen pass. The orthogonal projection runs once per edit at placement and takes 0.91 s of the 12.3 s per edit (Table 15 of the revised paper). Table R5 shows that our method takes 12.3 s per edit against 2.4 to 12.8 s for the baselines, and its pool grows with the stream to 33.9 GB, with an adapted pass of 170 ms over all stored experts, which is the cost of keeping one expert per edit that the per-edit parameter ratio does not show. ROME [7], MEMIT [8], GRACE [9], MELO [10] and SoLA [6] stay near the memory and latency of the frozen model, at a Reliability and Generality of at most 89.6 and 81.1 against 94.4 and 86.5 in Table 1. The time column is generated under the paper's protocol for every row, under which ELDER trains a mixture on each of the 96 adapted projections rather than on the six layers of its paper, and its 12.8 s is not comparable with the 2.12 s per edit that its paper reports on one 48GB NVIDIA A40.
Table R5: Time and memory of the baselines.
| Method | Param | Pool GB | Peak GB (lower bound) | Time s/edit | Latency ms per pass |
|---|---|---|---|---|---|
| ROME [7] | 0.97 | 0.0 | 13.5 | 4.2 | 48 |
| MEMIT [8] | 4.86 | 0.0 | 13.5 | 7.5 | 48 |
| GRACE [9] | 1.00 | 0.09 | 13.6 | 2.4 | 52 |
| MELO [10] | 0.86 | 0.08 | 13.6 | 4.9 | 49 |
| SoLA [6] | 0.79 | 0.07 | 13.6 | 4.3 | 57 |
| ELDER [2] | 1.00 | 0.09 | 13.6 | 12.8 | 49 |
| UAR-LoRA (Ours) | 0.45 | 33.9 | 47.4 | 12.3 | 170 (admit), 48 (abstain) |
(3) All reported settings share one gate threshold, namely tau = 0.7, selected once as the ECE minimiser of Table 7 on 500 held-out calibration paraphrases and held fixed across datasets and backbones, including every row of part 1 of Table R2. Part 3 of Table R2 shows that the ECE-minimising threshold of each setting moves within one grid step of it (0.7 to 0.8), and the shared 0.7 is the minimiser in one of the four settings.
Following your suggestion, we have added Table R5 to Appendix A.13 as Table 21 (Page 36, Lines 1958 to 1977).
Summarized Q4: Definition of q and figure text size
A6: We respectfully answer your two points as follows. (1) We have revised Section 4.2 to state in place that q min-max rescales the entropy H_t over the entropies of the W preceding edits, clipped to [0, 1], with a cold-start value d_0 for the first t_0 edits (Page 4, Lines 201 to 202). Written out as Eq. (14) in the appendix, q(H_t) = min(1, max(0, (H_t - min H) / (max H - min H))), with the whole prefix used until the window fills and d_t = d_0 if max H = min H, and W = 1000, t_0 = 100 and d_0 = 0.5. (2) We explain the two figures below, so that each element can be read without zooming in.
- Figure 1 compares prior editors (left) with our method (right), one row per per-edit decision. In the first row, prior editors give every edit the same fixed rank, while our method sizes each expert from the frozen model's predictive entropy, so that easy edits receive a small rank and hard edits a large one. In the second row, prior editors route every query into the pool, while the confidence gate of our method returns queries below the threshold τ to the frozen model. In the third row, randomly initialised experts overlap the subspaces of earlier ones, while our method places each new expert orthogonally to them. The last row shows the outcome: interference compounds for prior editors, while our method keeps its Reliability over a stream of 5000 edits.
Response to Reviewer z2WP
Dear Reviewer z2WP:
Thank you for recognizing the clear motivation, the method design, and the method effectiveness. We have responded point by point to your concerns (including novelty, more baselines and settings, evaluation over all edits, the difficulty signal, capacity and cost analysis, presentation, and code). We hope our responses resolve your concerns and look forward to further discussion. All revisions made in response to your comments are marked in blue in the revised manuscript.
Summarized W1: Limited novelty against AdaLoRA, ELDER and O-LoRA
A1: We respectfully clarify that what differs is when and on what each of the three decisions is taken, and that the three are coupled within one edit, while each technique exists on its own in AdaLoRA [1], ELDER [2] and O-LoRA [3]. Table R1 compares the three techniques with our method on each decision. The decisions are coupled because the rank chosen by the probe fixes the shape of the candidate that the initialiser projects, which requires the projection to hold for ranks that vary per edit, as Eq. (11) gives for any {r_s}, and because the confidence head is fitted on the edit that the new expert stores (Eq. (12)).
Table R1: Where each decision is taken, compiled from the method definitions of the paper.
Following your suggestion, we have added this table to Appendix A.13 as Table 16 (Page 35, Lines 1865 to 1880).
Summarized W2: More backbones, datasets, baselines, and longer streams
A2: Following your suggestion, we further compare our method with four lifelong editors (including ELDER [2], WISE [4], LEMoE [5] and SoLA [6]) on two backbones (i.e., LLaMA-2-7B and Qwen2.5-7B) and two datasets (i.e., CounterFact and ZsRE) at T = 1000 and 5000, extend the ZsRE stream on LLaMA-2-7B to T = 10000, and report the gate threshold per setting. Table R2 shows that our method leads all four editors in Reliability and Generality in every setting of part 1, with Locality not ordered among the methods, and at T = 10000 its Reliability is 93.8 against 85.6 for ELDER and 83.9 for WISE, while every method is lower at T = 5000 than at T = 1000.
Table R2: Part 1. Coverage: two 7B backbones, two datasets, four lifelong editors and UAR-LoRA at T = 1000 and 5000.