- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines when online LoRA updates can act as usable memory for a frozen LLM. The proposed pipeline uses an online-updated adapter to generate queries for an external text log, leaving the final question answering to the frozen base model. Across evaluations of recognition, direct recall, retrieval, and downstream use, the results show that while the approach works for isolated fact writes, it breaks down as new memories accumulate.
Critical strengths
- The paper is well-organized, with detailed explanations of the preliminary, intuition, and methodology.
- The paper separates likelihood preference, direct recall, retrieval, and downstream application, preventing proxy gains from being conflated with truly usable memory.
- The paper evaluates its findings across multiple benchmarks and experimental settings, providing useful insights into both the benefits and limitations of online in-weight memory.
Critical weaknesses
-
Giving the direct-QA supervision and ~8x more gradient updates than background turns essentially tells the adapter which fact will be tested. However, an empty query still achieves 0.496 target retrieval compared to 0.584 with the true query as reported in Table 2, suggesting the model mostly memorizes the most emphasized fact rather than learning query-dependently. The authors should test a balanced setting where multiple facts receive equal supervision and the model must selectively retrieve among them.
-
The LoRA adapter is trained on additional QA pairs and paraphrases that are absent from the frozen retrieval index. While HYP1 matches inference compute, it does not match this ingestion-time data advantage, making it unclear whether the gains come from parametric memory or simply better retrieval keys.
-
The accumulated-state evaluation compares valid and absent queries within trained adapters but omits a frozen baseline and downstream metrics; beating an empty query only shows basic query conditioning, not a benefit from weight updates. Moreover, comparing isolated and accumulated results shifts the underlying task population alongside the memory load, confounding the cause of the performance drop.
-
The apparent dissociation between recognition, recall, and application may be an artifact of noisy measurement rather than true retrieval failure. Specifically, likelihood gains can reflect surface template matching, recall relies on a brittle five-character stem-hit proxy, and the automated application judge has documented parsing failures without human validation. Semantically validated recall and a blinded human audit are needed to confirm whether knowledge is genuinely retained when access fails.
Questions for authors
- How does the pipeline compare with a frozen retriever that indexes the exact direct-QA supervision used for LoRA training?
- Can you add a frozen baseline and evaluate final answers in the new accumulation experiment?