- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LoRA Readout combines independently NTP-written document LoRAs with a shared reader trained across frozen memories using Reptile, answer NLL and contrastive ranking. New documents need raw-text writing but no target-document QA or reader adaptation. The paper reports gains over matched-backbone generator checkpoints, then uses memory controls, counterfactual facts and interventions to investigate when stored information supports answer generation. Its timing study measures a restricted repeated-query workload.
Critical strengths
-
The evidence goes beyond generic QA adaptation. Correct/wrong/random-memory controls, task-only readers, natural-generation branch ablations and 1,000 factual-counterfactual pairs are complementary. Both-variant factual success is 32.0% with the trained reader versus 12.5% with memory alone and 11.2% with the task-only reader (Table 8). The negative results and separation of candidate preference from generation make the analysis particularly informative.
-
Table 3 supplies a meaningful fixed-memory factorial rather than only an end-to-end comparison: both episodic updates and contrastive supervision improve the reported scores. Table 1 extends the recipe to three backbones and five datasets, including original-entity evaluations. The explicit training-exposure audit and discussion of writing, prompt and document-length sensitivity help establish what this relatively small-data recipe does and does not achieve.
Critical weaknesses
-
The novelty and baseline coverage remain limited. Table 1 compares three adapter generators, but only one per backbone, leaving the closest shared-reader precedent, DCD, without a matched comparison. Document-specific adapters and reusable knowledge extractors already exist in prior work; the methodological advance here is primarily the Reptile/contrastive training recipe. Table 3 supports this recipe under additive composition, but does not establish its advantage over a comparable reader with learned composition. This leaves the significance of the incremental method less convincing than its gains over generator checkpoints suggest.
-
The practical advantage is also insufficiently established. Table 2 shows a substantial quality gap to full-context answering, while the timing comparison in §5.5 and Figure 4 uses fresh prefill without prefix reuse. The reported crossover therefore applies to this restricted protocol, and 32k timing alone does not demonstrate reliable QA at that length. Evaluating answer quality and cost on the same workload, with a prefix-cached baseline and KV storage accounting, would make the deployment tradeoff more convincing.
Questions for authors
-
Which result most clearly distinguishes the contribution from DCD’s memory-conditioned extractor, beyond the internal additive-composition ablation? How stable are the recipe gains across independently trained readers? Please distinguish matched evidence from cross-paper comparisons and state the scope if these results are unavailable.
-
What changes in the repeated-query crossover when the full-context baseline reuses its document prefix, including the KV storage cost and adapter merging? If this measurement is unavailable, which concrete workload is the current timing claim intended to cover?
-
How well does the reader generalize from factual recall to questions requiring multiple facts to be combined? A targeted evaluation on such questions would help clarify whether it supports compositional use of the stored knowledge.
8 comments
(2/8) Matched evidence vs. cross-paper evidence (Q1)
As requested, we separate the two kinds of evidence.
Cross-paper comparison. The only cross-paper comparison is DCD's published table (App. A.1, Table 4). We use it to motivate the hypothesis above, not as evidence that our method is better: backbones, adapter capacities, documents, and budgets all differ.
Matched evidence already in the submission. All of the following use the same backbone, data, frozen memories, and budget:
- Table 3: Reptile and contrastive supervision each improve both update rules on all five datasets.
- App. A.4 and B.4: our reader vs. task-only readers that match its capacity, gold-QA exposure and order, and update count, but are trained without memory. Training with memories adds a memory benefit of 7.17 F1 points [1.61, 12.71].
- App. C: on natural counterfactual pairs, our reader follows both factual variants in 32.0% of cases, vs. 11.2% for the task-only reader.
New matched experiment. To test the principle in (1/8), we remove each element in turn. If the principle holds, memory specificity (correct − wrong memory) should drop, not just accuracy. All rows use Qwen3-8B, identical frozen evaluation memories, and 3 reader seeds (averages over the five benchmarks):
| Reader training | Correct mem. + reader | Wrong mem. + reader | Specificity |
|---|---|---|---|
| Full recipe | 0.441 | 0.281 | 0.160 |
| − entity replacement | 0.418 | 0.298 | 0.120 |
| Same-context → random cross-context negatives | 0.405 | 0.288 | 0.117 |
| Only 150 training memories (DCD's QuALITY scale) | 0.372 | 0.275 | 0.097 |
| Heavily over-written training memories (≈50–100 passes) | 0.392 | 0.283 | 0.109 |
The full-recipe row is the 3-seed mean; the single checkpoint in Table 3 scores 0.451 (see 3/8).
Every removal lowers specificity, from 0.160 to 0.097–0.120:
- Without entity replacement, the wrong-memory score rises (0.281 → 0.298): the reader falls back on prior knowledge.
- With random cross-context negatives, specificity falls by 0.043 and accuracy (0.405) approaches Reptile with NLL alone (0.381, 3-seed mean). Random negatives are too easy to tell apart and leave answer-type priors open.
- With only 150 training memories, specificity drops the most (0.097).
- With over-written training memories, specificity drops to 0.109, consistent with App. A.3 and the budget mismatch in Fig. 5b.
Each route listed in (1/8) therefore affects whether the reader learns to depend on the memory, as the principle predicts.
(3/8) Learned composition and stability across readers (W1, Q1)
Learned composition. The reviewer notes that Table 3 supports our recipe under additive composition, but does not compare it with a reader that uses learned composition. We train a DCD-style extractor on our frozen NTP memories: answer NLL, with learned per-layer weights combining the memory and reader updates. We cross objective and composition:
| Objective / update | Composition | Avg. (5 datasets) | Specificity |
|---|---|---|---|
| NLL / direct (≈ DCD extractor) | Learned per-layer weights | 0.365 | 0.086 |
| NLL / direct | Additive | 0.362 | 0.081 |
| NLL + contrastive / Reptile (ours) | Learned per-layer weights | 0.454 | 0.173 |
| NLL + contrastive / Reptile (ours) | Additive | 0.451 | 0.170 |
All four rows are single reader checkpoints, as in Table 3; the additive rows are the Table 3 averages over the five scores (QuAIL is accuracy). This design separates the contribution of the training recipe from that of the composition function.
Learned per-layer weights change the average by only +0.003 under either recipe, and specificity by +0.003 to +0.005. Under either composition, our recipe is about 0.09 higher on average and has roughly twice the specificity (0.170–0.173 vs. 0.081–0.086). The difference between DCD's setting and ours therefore comes from the training recipe, not from the composition function.
Stability across independently trained readers. The intervals in the submission condition on a single reader checkpoint (App. A.2), so they do not answer this question. We retrain the reader with three seeds for all four Table 3 configurations, with memories held fixed (averages over the five benchmarks):
| Configuration | Reported (single checkpoint) | Mean of 3 seeds | Std of avg. | Largest per-dataset std (ROPES / QuAIL) |
|---|---|---|---|---|
| Direct, NLL | 0.362 | 0.355 | 0.008 | 0.021 |
| Direct, NLL + contrastive | 0.388 | 0.383 | 0.007 | 0.017 |
| Reptile, NLL | 0.386 | 0.381 | 0.009 | 0.022 |
| Reptile, NLL + contrastive | 0.451 | 0.441 | 0.012 | 0.031 |
Seed variation is largest on ROPES and QuAIL, where the full recipe also gains the most (0.573 and 0.526 for the Table 1 checkpoint). The full recipe's 3-seed mean (0.441) is 0.010 below the reported checkpoint. The conclusion is unchanged: the full recipe stays 0.058 above the next configuration, several times the seed std. Table 3 and the Qwen3-8B rows of Table 1 will report mean ± std per dataset.
(1/8) What distinguishes the contribution from DCD (Q1)
We thank the reviewer for the precise review, and especially for asking us to separate matched evidence from cross-paper comparisons.
We do not claim the decomposition into document LoRAs and a shared extractor; DCD already has it. The result that distinguishes our work is that plain NTP memories become usable through read-side learning alone.
DCD found the opposite in its setting: NTP memory plus extractor performs about the same as the extractor alone. On Llama-3.1, DCD reports 40.9 vs. 39.3 on QuALITY and 22.3 vs. 20.9 on NarrativeQA. DCD concluded that the writer must change, and replaced NTP with distillation.
We believe the difference comes from the reader's training conditions, through one principle:
DCD's extractor setting leaves several cheaper routes open. Our recipe closes each of them:
Our own data agree with this account. QuAIL, a multiple-choice reading-comprehension set close to QuALITY, is the only source in our natural diagnostic with a negative correct-minus-wrong-memory estimate (−2.63 [−12.62, 6.63]; App. A.4).