acceptodds
Under review as a conference paper at ICLR 2027

How LoRA Remembers? A Parametric Memory Law for LLM Finetuning

Abstract

Large Language Models (LLMs) must continuously learn and update knowledge to remain effective in dynamic real-world environments. While Low-Rank Adaptation (LoRA) is widely used for such memory updates, existing studies mainly rely on qualitative downstream evaluations, leaving the quantitative capacity limits and underlying dynamics of exact parametric memory largely unexplored. To bridge this gap, we employ LoRA as a controlled memory capacity probe within the latent space to systematically quantify exact parametric memory. We introduce the LoRA Parametric Memory Law, a robust power law that characterizes incremental, latent-space parametric memory realized via LoRA, linking loss reduction to effective parameters and sequence length. At the token level, fine-grained analysis characterizes a transition from competition-dependent to probability-guaranteed recall, showing that a prediction probability of constitutes a sufficient condition for token-level recall under greedy decoding. Driven by these insights, we introduce MemFT, a threshold-guided optimization strategy that dynamically redistributes the training budget toward sub-threshold tokens. Empirical evaluations demonstrate that MemFT can enhance memory fidelity and efficiency.

Primary area:
interpretability and explainable AI
Keywords:
large language model, LLM memory, parametric memory, LoRA, LLM finetuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3yLWOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies exact parametric memorization in llm. The paper proposes an empirical power-law relationship between LoRA rank, memory length, and loss reduction. It examines token-level confidence and autoregressive recall failures, then introduces MemFT, which concentrates optimization on tokens below a confidence threshold. The empirical evaluation shows strong fits for the proposed loss-reduction formula and improvements over standard baselines.

Critical strengths

1.The distinction between teacher-forced loss and recall fidelity is useful. The token-level analysis exposes local failure points that average loss can conceal.

2.MemFT is simple and meaningful in this settings. Average gains across two models and two benchmarks suggest that the approach warrants further investigation.

Critical weaknesses
  1. Eq (6) and sec 3.3 demonstrate strong fits to observed loss reductions, but do not clearly establish prediction on unseen settings. And, Section 4.1 shows that low average loss can coexist with poor recall, weakening the interpretation of this as exact memory capacity law.

  2. eq (7) provide a sufficient condition for greedy correctness, but do not establish the optimality of the threshold used in eq(9). Table 2 mainly compares MemFT with SFT; comparisons against alternative thresholds and difficult-token weighting are needed to isolate the specific contribution of the proposed threshold-guided token optimization strategy.

Questions for authors
  1. Were the reported fitting metrics computed on the same configurations used to estimate (C,\alpha,\beta,b)?

  2. A threshold sweep determine whether 0.5 has a specific empirical advantage.

Reviewer_DYNTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

Based on my understanding; this paper is about studying how LoRA adapters store and recall exact information in language models and the paper has 3 main contributions; first one is authors claim to find a power law relating LoRA rank and sequence length to training loss reduction. Second; they study how low confidence tokens can lead to errors during greedy decoding and based on second contribution; they have their third contribution MemFT which focuses on penalizing target probability below 0.5. Authors compare MemFT with SFT on two large models (Qwen3-8B and Llama3.1-8B) and they claim their proposed method significantly surpassing (!) SFT in both fidelity and parameter efficiency.

Critical strengths
  1. I think the problem itself is very interesting and authors thought about in a good way because although the community uses low rank adaptation almost in all of fine-tuning settings and especially LoRA should be studied more to carefully understand the dynamics and the effect of using such techniques on large models' behaviours.
  2. Not in all cases; but in some cases MemFT results look promising and show improvements compared to SFT for those two large models authors tested which can show there might be an interesting connection between each token loss and the training.
Critical weaknesses
  1. I think the power law comes from the fact that the more parameters we have; the more information can be encoded because more parameters can be changed. Increasing the rank of LoRA will result in capturing the long tail of the input distribution because of having more parameters to change and fit on input data so the first contribution is not surprising to me; although this may sound like it is not a weakness; but because one of the authors' main claims (paper title) is this simple point and the only part done by authors was introducing the law itself which directly matches the rank and length; I found it as a weakness.
  2. The MemFT is reminding me of concepts like max loss optimization or min loss optimization or tilted empirical risk minimization (TERM) instead of optimizing empirical risk minimization which works with the average of losses. This problem is a good application for these kinds of losses but still too simple and the 0.5 threshold as a sufficient condition is not well studied and it comes from greedy decoding and has no relation with LoRA itself.
Questions for authors
  1. Can the memory law predict exact length of the memory that the network can store? This can change my opinion about the importance of this law.
  2. Can you do an ablation study on the 0.5 threshold for MemFT? It would be useful seeing how much MemFT is sensitive to this threshold.
  3. I could not find the experimental configuration the authors used to compare SFT and MemFT. One concern I have is the comparison of convergence speed and iteration numbers these methods need to converge.
Reviewer_LW2GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies exact memorization through LoRA. It fits an empirical law relating loss reduction to adapter rank and memory length (Eq. 6). It then shows that low average loss can coexist with failed recall, because a few tokens keep ptarget<0.5p_{\mathrm{target}} < 0.5 (Section 4). Finally, it proposes MemFT, which focuses training on tokens with Lt>ln⁡2L_t > \ln 2 (Eqs. 8–9). MemFT improves recall over standard SFT on Qwen3-8B and Llama3.1-8B.

Critical strengths

The paper shows how low average training loss can hide a few difficult token positions associated with generation failures (Section 4, Fig. 3). The paper translates its token-level observations into a simple intervention, MemFT-OT (Eqs. 8–9), which improves recall in the tested settings (Table 2). This connection between analysis and a practical method is a useful contribution, although the gains vary considerably across configurations.

Critical weaknesses
  1. The law does not measure exact-recall capacity and is not validated. Eq. 6, ΔL=Crαℓ−β+b\Delta L = C r^{\alpha} \ell^{-\beta} + b, models average loss reduction, and the fitting cutoff Lfinal≤0.69L_{\mathrm{final}} \leq 0.69 removes nearly every configuration that satisfies the paper's own recall guarantee. The fitted C,α,β,bC, \alpha, \beta, b are not reported, and there is no held-out prediction. Evaluation is limited to two 8B models and selected MLP modules (Appendix D).

  2. MemFT's distinctive contribution is insufficiently established. The guarantee ptarget>0.5p_{\mathrm{target}} > 0.5 and the threshold Lcrit=ln⁡2L_{\mathrm{crit}} = \ln 2 (Eq. 7) are elementary, as Section 4.3 acknowledges. The only baseline is uniform SFT, and comparisons with focal loss, alternative thresholds, or argmax/margin masking are missing. The active-token normalization in Eq. 8, which effectively raises the learning rate as the active set shrinks, is not ablated.

Questions for authors
  • What are the fitted parameters of Eq. 6, and does the law predict held-out (r,ℓ)(r, \ell) configurations?
  • With a tuned learning rate and longer training, does SFT still fail at the same position? Do MemFT's gains persist against alternative token-weighting objectives?
  • What exactly do Figs. 7–8 plot, and how many sequences and seeds does each point use?
1 comment

Thank you again for your thoughtful review, and we apologize for another brief follow-up. With only 30 minutes remaining before the discussion deadline, we are available to respond promptly to any further questions.

We hope our responses have addressed your concerns and would be grateful if you could consider updating your rating in light of the clarifications and additional results.

Thank you sincerely for your time and contribution to the review process.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Our Rebuttal (2/2)

Core contributions.

  1. Macroscopic exploration through the LoRA Parametric Memory Law. We use loss reduction to study incremental parametric memory, establishing an empirical power law relating loss reduction, LoRA rank, and sequence length, supported by held-out validation.
  2. Microscopic findings: loss–accuracy misalignment and stubborn tokens. Low average loss can coexist with poor recall. Across the examined configurations, stubborn token positions remain at low probabilities even after SFT convergence. These local bottlenecks are strongly associated with the first decoding failure and can trigger autoregressive error cascades.
  3. MemFT, motivated by the microscopic findings. We redirect optimization toward unresolved stubborn tokens rather than further sharpening confident predictions. Under the same LoRA parameterization, MemFT improves average recall over SFT across both models and benchmarks.

Thank you!

We sincerely hope our responses and revisions address all reviewers’ concerns. We sincerely believe that these updates may help us better deliver the benefits of the proposed work to the ICLR community. In conclusion, we extend our heartfelt thanks to all the reviewers for their invaluable suggestions, which have significantly enhanced the rigor and completeness of our work. We welcome further discussion and are more than willing to answer any additional questions you may have. Thank you very much!

AuthorsOpenReview ↗

Summary of Our Rebuttal (1/2)

Dear Reviewers, Area Chairs, Senior Area Chairs, and Program Chairs,

Thank you for your careful reading and constructive feedback. We summarize the reviewers' positive assessments, the manuscript revisions and new predictive validation, and the clarifications connecting incremental memory capacity, token-level bottlenecks, and MemFT.

Strengths recognized by the reviewers.

  • The importance of understanding LoRA memory. Reviewer DYNT found the problem very interesting and emphasized the need to understand how low-rank adaptation affects model behavior.
  • Token-level mechanisms beyond average loss. Reviewers LW2G and 3yLW highlighted the useful distinction between teacher-forced loss and recall fidelity, and the identification of local failure points that average loss can conceal.
  • A simple, mechanism-driven intervention. Reviewer LW2G recognized the connection between the analysis and the practical MemFT intervention. Reviewer 3yLW described MemFT as simple and meaningful, with average gains across two models and two benchmarks; DYNT also noted promising improvements in some configurations.

Manuscript revisions.

The additions listed below are highlighted in blue in the revised manuscript. The main text now includes a held-out prediction and memory-budgeting subsection, with complete experimental details and results in Appendix K. The new predictive validation reuses existing model runs.

Topic (reviewer)RevisionWhere to look
Holdout experiments (all three reviewers)Add a subsection and one summary table combining Random holdout, Whole-rank holdout, and Whole-length holdout for both models on Long-context (Combined) and PhoneBook.Sec. 3.4, Held-Out Prediction and Memory Budgeting
Complete validation details (all three reviewers)Add full-data fits and held-out results across 16 model–setting combinations, with training-only fitting, grouped splits, pooled whole-axis scoring and random-split SDs.Appendix K, both tables and protocol description
Rank and length estimates (DYNT, 3yLW)Add explicit inverse formulas for rank and memory length at a target loss reduction, plus the conversion from rank to added parameters.Sec. 3.4, Estimating rank and memory length
Convergence and figure guidance (LW2G, DYNT)Add navigation to the existing loss and recall figures; explain the rank–length panels, matched SFT/MemFT-OT comparison, final-checkpoint convergence, and recall-versus-length plots.Appendix D; Appendix H, Figures 4–6; Appendix I, Figures 7–9

How we addressed the concerns.

  • Held-out experiments validate loss-reduction prediction. Random holdout fits on 80% of rank–length groups and tests on 20%; Whole-rank and Whole-length holdout each exclude an entire rank or length in turn. Across both models on Long-context (Combined) and PhoneBook, held-out R² ranges from 0.958 to 0.985 within the fitted regime (Sec. 3.4; Appendix K).
  • The law quantifies incremental memory and supports budget estimation. ΔL measures additional compression through LoRA over a frozen backbone. With setting-specific coefficients, the law supports rank and length estimates at a target loss reduction, rather than equating low average loss with exact recall.
  • The 0.5 threshold has a mathematical basis. Given the correct prefix, target probability p > 0.5 is a mathematically derived sufficient condition for greedy correctness. This motivates targeting low-confidence tokens without setting-specific threshold tuning; empirical optimality is not claimed.
  • Training-loss convergence supports fair comparisons. We provide training-loss curves across rank–length configurations to verify that optimization steps are sufficient for loss to converge (Appendix H, Figures 4–6). All reported results use converged final checkpoints, ensuring fair comparisons between sufficiently trained methods.
  • Extensive experiments reveal hard-to-memorize stubborn tokens. Appendix J (Figures 10–13) presents Qwen3-8B probability grids across LoRA ranks, memory lengths, and datasets mixtures. Across this sweep, numerous token positions remain far below p = 0.5 after convergence, revealing persistent local memory bottlenecks that average loss can conceal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.