- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
DocPatch is a parametric memory framework for using knowledge from continually updated documents with a frozen LLM. It converts document chunks into LoRAs using a pretrained Doc-to-LoRA hypernetwork, stores them in a knowledge bank, and retrieves and composes relevant LoRAs for each question. When a document is revised, new LoRAs are added. Given version and conflict metadata, the framework excludes superseded memories from the retrieved set when composing conflicting information. A shared reasoning adapter is trained to follow the activated knowledge and use information from multiple memories. The authors evaluate QA and routing performance on HotpotQA, as well as knowledge updating and preservation under sequential document revisions. They also report ablations on the reasoning adapter, LoRA selection and weighting, and counterfactual training, together with an inference efficiency comparison.
Critical strengths
I think the combination of reusable document-specific knowledge LoRAs and a shared adapter for using that knowledge is worth studying, and is included in a important problem statement. It fits the goal of incorporating new documents while reusing existing memories. I also found the separate evaluation of query-dependent weighting and reasoning training in Tables 3 and 5 useful. These results provide evidence about which components affect QA performance when building a system around document-derived parametric memories.
Critical weaknesses
1. I do not clearly understand what the paper resolves beyond existing work.
I agree with the motivation of managing knowledge from continually changing documents. What I found unconvincing was the next step: what specifically makes this difficult for existing document-derived LoRA memories, and why the proposed method addresses that difficulty well.
I could broadly follow the roles of routing, composition, conflict resolution, and reasoning training. The paper also includes useful ablations. However, I still had trouble identifying the central difficulty that DocPatch resolves. Showing that individual components improve performance is useful, but I also want to understand which limitation of existing approaches these improvements overcome.
Missing related work makes this harder to assess. PRAG and DyPRAG study the parameterization, retrieval, and use of document knowledge. Knowledge Modules/DCD separates knowledge modules from an adapter trained to use that knowledge. LAG studies selection from a library of knowledge and task LoRAs, while Understanding LoRA as Knowledge Memory analyzes memory capacity, routing, and composition. I think the paper needs to cite these paper and explain its relationship to prior studies so that readers can identify its starting point and additional contribution.
Other relevant studies are also missing, including Arrow on reusing and routing LoRA libraries, LoRA Soups on weighted LoRA composition and skill composition, and ELDER on lifelong model editing with multiple LoRAs and a router.
Overall, I did not come away with a clear understanding of what new result the paper establishes about continual LoRA memory under its stated version and conflict metadata assumptions.
2. I had difficulty understanding how the central experiments were conducted and what the results demonstrate.
Section 4.1 states that the authors use HotpotQA and report EM and token-level F1. However, the evaluation split and configuration, as well as the scope of the LoRA bank searched by the router, are unclear. I think this information is necessary to interpret the results. Retrieving from a question-specific candidate set and retrieving from the entire evaluation corpus require different retrieval capabilities.
Section 4.5 and Table 4 were particularly difficult to follow. The paper explains that documents are revised sequentially while previous LoRAs are retained. However, I could not determine which documents were used to construct these revision histories, how questions and answers were generated, or when evaluation took place during the update sequence. The brief descriptions of Unchanged, Updated, Old-Only, Override, and Unrelated were insufficient for me to reconstruct the actual evaluation. The precise metric reported in each column also needs clarification.
I also do not understand how ROME and MEMIT were applied. These methods modify model parameters to change specified factual associations. Applying them to document revisions requires an explanation of how document changes were converted into edit requests. Which facts were edited, how were edits applied across revisions, and what information was provided to each method?
The lack of explanatory figures and examples adds to this difficulty. A example connecting document revisions, metadata, activated memories, and expected answers would have helped me understand both the method and Table 4. Currently, the explanation relies on one overview diagram and aggregate performance tables.
Questions for authors
What is the most important difficulty you aim to address when applying existing document-derived LoRA memories to continually updated document collections? How does DocPatch address it, and which result demonstrates this most directly? Please explain this in relation to the prior work discussed above.
Could you provide a concrete example of a document revision history and its associated questions and answers in Table 4? When is evaluation performed after each revision, and what knowledge is considered correct, particularly for Old-Only and Override? What metric does each column report, and how are results aggregated across revision steps?
How were the version precedence and conflict-group annotations constructed? Please distinguish information supplied by the document update process from annotations constructed separately. I would also like to understand how an actual document revision is represented as conflicts between chunks, and which older memories remain active as a result.
How were document revisions provided to the editing baselines, including ROME and MEMIT? Please describe how edit requests were constructed from documents and whether they were applied sequentially or in batches. What information did DocPatch and each baseline receive at update time?
How were the counterfactual training pairs constructed from documents and questions? How were documents and revision histories separated between training and evaluation? Did QA baselines such as RAG also receive the QA supervision used to train the reasoning adapter? I think this matters for interpreting the roles of memory design and additional training.