acceptodds
Under review as a conference paper at ICLR 2027

LoRA Readout: Learning to Read Parametric Memories for Context-Free Inference

Abstract

Encoding context into model parameters can reduce the inference costs of in-context learning, but conventional fine-tuning can be costly and require context-specific supervision. We introduce LoRA Readout, a framework that separates context writing from memory reading to enable context-free inference. The writer encodes raw context into a memory LoRA through next-token prediction, without requiring context-specific question-answer pairs or other auxiliary supervision. A shared reader is trained across frozen memories to access the encoded knowledge and use it to answer queries. Incorporating a new context requires only an offline memory-writing step, while the reader is reused without further adaptation. At inference, the model combines the memory and reader to answer queries without including the original context tokens in the prompt. Experiments across multiple datasets show that LoRA Readout outperforms related approaches with a small training-token budget: only M training input tokens for source-memory writing and reader training on Qwen3-8B, less than of the smallest context-training corpus reported by these approaches.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Low-Rank Adaptation, Parametric Memory, Context Compression, Context-Free Inference, In-parameter learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GpZdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LoRA Readout combines independently NTP-written document LoRAs with a shared reader trained across frozen memories using Reptile, answer NLL and contrastive ranking. New documents need raw-text writing but no target-document QA or reader adaptation. The paper reports gains over matched-backbone generator checkpoints, then uses memory controls, counterfactual facts and interventions to investigate when stored information supports answer generation. Its timing study measures a restricted repeated-query workload.

Critical strengths
  1. The evidence goes beyond generic QA adaptation. Correct/wrong/random-memory controls, task-only readers, natural-generation branch ablations and 1,000 factual-counterfactual pairs are complementary. Both-variant factual success is 32.0% with the trained reader versus 12.5% with memory alone and 11.2% with the task-only reader (Table 8). The negative results and separation of candidate preference from generation make the analysis particularly informative.

  2. Table 3 supplies a meaningful fixed-memory factorial rather than only an end-to-end comparison: both episodic updates and contrastive supervision improve the reported scores. Table 1 extends the recipe to three backbones and five datasets, including original-entity evaluations. The explicit training-exposure audit and discussion of writing, prompt and document-length sensitivity help establish what this relatively small-data recipe does and does not achieve.

Critical weaknesses
  1. The novelty and baseline coverage remain limited. Table 1 compares three adapter generators, but only one per backbone, leaving the closest shared-reader precedent, DCD, without a matched comparison. Document-specific adapters and reusable knowledge extractors already exist in prior work; the methodological advance here is primarily the Reptile/contrastive training recipe. Table 3 supports this recipe under additive composition, but does not establish its advantage over a comparable reader with learned composition. This leaves the significance of the incremental method less convincing than its gains over generator checkpoints suggest.

  2. The practical advantage is also insufficiently established. Table 2 shows a substantial quality gap to full-context answering, while the timing comparison in §5.5 and Figure 4 uses fresh prefill without prefix reuse. The reported crossover therefore applies to this restricted protocol, and 32k timing alone does not demonstrate reliable QA at that length. Evaluating answer quality and cost on the same workload, with a prefix-cached baseline and KV storage accounting, would make the deployment tradeoff more convincing.

Questions for authors
  1. Which result most clearly distinguishes the contribution from DCD’s memory-conditioned extractor, beyond the internal additive-composition ablation? How stable are the recipe gains across independently trained readers? Please distinguish matched evidence from cross-paper comparisons and state the scope if these results are unavailable.

  2. What changes in the repeated-query crossover when the full-context baseline reuses its document prefix, including the KV storage cost and adapter merging? If this measurement is unavailable, which concrete workload is the current timing claim intended to cover?

  3. How well does the reader generalize from factual recall to questions requiring multiple facts to be combined? A targeted evaluation on such questions would help clarify whether it supports compositional use of the stored knowledge.

8 comments
AuthorsOpenReview ↗

(1/8) What distinguishes the contribution from DCD (Q1)

We thank the reviewer for the precise review, and especially for asking us to separate matched evidence from cross-paper comparisons.

We do not claim the decomposition into document LoRAs and a shared extractor; DCD already has it. The result that distinguishes our work is that plain NTP memories become usable through read-side learning alone.

DCD found the opposite in its setting: NTP memory plus extractor performs about the same as the extractor alone. On Llama-3.1, DCD reports 40.9 vs. 39.3 on QuALITY and 22.3 vs. 20.9 on NarrativeQA. DCD concluded that the writer must change, and replaced NTP with distillation.

We believe the difference comes from the reader's training conditions, through one principle:

A reader learns to use a memory only when reading it is the cheapest way to lower its training loss.

DCD's extractor setting leaves several cheaper routes open. Our recipe closes each of them:

FactorDCD extractor settingOursRoute it leaves open / closes
EntitiesOriginal (well-known books and scripts)ReplacedPretraining knowledge: the extractor alone lifts NarrativeQA from 12.5 to 20.9
NegativesNoneSame-context, same typeAnswer-type and format priors
Training memories150 (QuALITY), 1,102 (NarrativeQA)1,197Memorizing answers per memory
Writing budget1,500 steps × batch 86 passesOver-fitted memories: DCD's NTP module alone scores below the base model on QuALITY

Our own data agree with this account. QuAIL, a multiple-choice reading-comprehension set close to QuALITY, is the only source in our natural diagnostic with a negative correct-minus-wrong-memory estimate (−2.63 [−12.62, 6.63]; App. A.4).

AuthorsOpenReview ↗

(2/8) Matched evidence vs. cross-paper evidence (Q1)

As requested, we separate the two kinds of evidence.

Cross-paper comparison. The only cross-paper comparison is DCD's published table (App. A.1, Table 4). We use it to motivate the hypothesis above, not as evidence that our method is better: backbones, adapter capacities, documents, and budgets all differ.

Matched evidence already in the submission. All of the following use the same backbone, data, frozen memories, and budget:

  • Table 3: Reptile and contrastive supervision each improve both update rules on all five datasets.
  • App. A.4 and B.4: our reader vs. task-only readers that match its capacity, gold-QA exposure and order, and update count, but are trained without memory. Training with memories adds a memory benefit of 7.17 F1 points [1.61, 12.71].
  • App. C: on natural counterfactual pairs, our reader follows both factual variants in 32.0% of cases, vs. 11.2% for the task-only reader.

New matched experiment. To test the principle in (1/8), we remove each element in turn. If the principle holds, memory specificity (correct − wrong memory) should drop, not just accuracy. All rows use Qwen3-8B, identical frozen evaluation memories, and 3 reader seeds (averages over the five benchmarks):

Reader trainingCorrect mem. + readerWrong mem. + readerSpecificity
Full recipe0.4410.2810.160
− entity replacement0.4180.2980.120
Same-context → random cross-context negatives0.4050.2880.117
Only 150 training memories (DCD's QuALITY scale)0.3720.2750.097
Heavily over-written training memories (≈50–100 passes)0.3920.2830.109

The full-recipe row is the 3-seed mean; the single checkpoint in Table 3 scores 0.451 (see 3/8).

Every removal lowers specificity, from 0.160 to 0.097–0.120:

  • Without entity replacement, the wrong-memory score rises (0.281 → 0.298): the reader falls back on prior knowledge.
  • With random cross-context negatives, specificity falls by 0.043 and accuracy (0.405) approaches Reptile with NLL alone (0.381, 3-seed mean). Random negatives are too easy to tell apart and leave answer-type priors open.
  • With only 150 training memories, specificity drops the most (0.097).
  • With over-written training memories, specificity drops to 0.109, consistent with App. A.3 and the budget mismatch in Fig. 5b.

Each route listed in (1/8) therefore affects whether the reader learns to depend on the memory, as the principle predicts.

AuthorsOpenReview ↗

(3/8) Learned composition and stability across readers (W1, Q1)

Learned composition. The reviewer notes that Table 3 supports our recipe under additive composition, but does not compare it with a reader that uses learned composition. We train a DCD-style extractor on our frozen NTP memories: answer NLL, with learned per-layer weights combining the memory and reader updates. We cross objective and composition:

Objective / updateCompositionAvg. (5 datasets)Specificity
NLL / direct (≈ DCD extractor)Learned per-layer weights0.3650.086
NLL / directAdditive0.3620.081
NLL + contrastive / Reptile (ours)Learned per-layer weights0.4540.173
NLL + contrastive / Reptile (ours)Additive0.4510.170

All four rows are single reader checkpoints, as in Table 3; the additive rows are the Table 3 averages over the five scores (QuAIL is accuracy). This design separates the contribution of the training recipe from that of the composition function.

Learned per-layer weights change the average by only +0.003 under either recipe, and specificity by +0.003 to +0.005. Under either composition, our recipe is about 0.09 higher on average and has roughly twice the specificity (0.170–0.173 vs. 0.081–0.086). The difference between DCD's setting and ours therefore comes from the training recipe, not from the composition function.

Stability across independently trained readers. The intervals in the submission condition on a single reader checkpoint (App. A.2), so they do not answer this question. We retrain the reader with three seeds for all four Table 3 configurations, with memories held fixed (averages over the five benchmarks):

ConfigurationReported (single checkpoint)Mean of 3 seedsStd of avg.Largest per-dataset std (ROPES / QuAIL)
Direct, NLL0.3620.3550.0080.021
Direct, NLL + contrastive0.3880.3830.0070.017
Reptile, NLL0.3860.3810.0090.022
Reptile, NLL + contrastive0.4510.4410.0120.031

Seed variation is largest on ROPES and QuAIL, where the full recipe also gains the most (0.573 and 0.526 for the Table 1 checkpoint). The full recipe's 3-seed mean (0.441) is 0.010 below the reported checkpoint. The conclusion is unchanged: the full recipe stays 0.058 above the next configuration, several times the seed std. Table 3 and the Qwen3-8B rows of Table 1 will report mean ± std per dataset.

Reviewer_JWa6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LoRA Readout, a framework that independently encodes documents into context-specific LoRA memories using next-token prediction (NTP) and trains a shared reader to retrieve information without including the original documents during inference. The shared reader is optimized across frozen memories using Reptile, answer prediction, and contrastive learning, enabling reuse on unseen documents without context-specific QA supervision. Experiments across five datasets and three backbones demonstrate improvements over selected parametric context-compression baselines, accompanied by memory-dependence analyses and inference-efficiency measurements.

Critical strengths
  1. A modular implementation of parametric memory and knowledge retrieval.

The framework separates independently trained NTP memory adapters from a shared reader, allowing new documents to be incorporated without retraining the reader or requiring document-specific QA supervision. Although closely related to existing test-time learning approaches, this particular decoupling offers a potentially useful alternative to differentiating through memory-writing optimization.

  1. Extensive experimental analysis of memory utilization.

The paper provides multiple controlled experiments, including memory swapping, task-only reader comparisons, counterfactual interventions, and ablations of the reader-training procedure, to examine whether the model actually utilizes stored information. These experiments strengthen the empirical contribution by distinguishing improvements in general QA capabilities from the use of context-specific parametric memories.

Critical weaknesses
  1. Limited novelty relative to PERK and omission of essential prior work.

PERK (Chen et al., ICLR 2026) already proposes encoding contexts into LoRA adapters through self-supervised NTP updates and meta-learning the ability to answer questions using the resulting parametric memories without the original context, making the central motivation and overall framework of this submission substantially less novel than presented. Although LoRA Readout differs by independently training a reusable reader over frozen memories, the paper neither cites nor compares against PERK, leaving its primary methodological contribution and empirical advantages over this highly relevant precedent insufficiently established.

  1. Limited retrieval reliability and insufficiently demonstrated practical advantages.

Despite outperforming the selected baselines, LoRA Readout remains substantially below full-context inference, achieving only 32.0% successful retrieval of both factual variants compared with 89.8% for full-context inference in Appendix C, while Appendix D.3 shows that the reader can substantially degrade retrieval under certain writing budgets and document lengths. Moreover, the inference-efficiency evaluation excludes cross-query prefix-cache reuse and requires approximately 25–32 queries to amortize memory writing, making the practical advantage over conventional context-based inference less convincing.

Questions for authors
  1. Novelty and comparison with PERK.

PERK (Chen et al., ICLR 2026) already introduces NTP-based context encoding into LoRA memories and meta-learning for context-free question answering, which appears closely related to the central approach of this paper. Could the authors clarify the fundamental methodological novelty beyond PERK and provide a direct comparison under matched backbones, datasets, and computational budgets, particularly to demonstrate the benefits of independently training a shared reader rather than meta-learning the memory adapter's initialization?

  1. Reliability of the shared reader.

Appendix D.3 shows that adding the reader can substantially reduce factual retrieval accuracy, including a drop from 78.1% to 49.0% at 4,096 tokens with 20 writing updates, despite improvements on the main benchmarks. Could the authors explain the conditions under which the reader improves versus degrades memory retrieval and provide evidence that its benefits generalize reliably across document lengths, writing budgets, and question distributions?

6 comments
AuthorsOpenReview ↗

(1/6) LoRA Readout and PERK follow different ideas (Q1)

We thank the reviewer for pointing us to PERK and for the careful reading of Appendices C and D.3. We apologize for omitting PERK; the revision cites and discusses it in the main text.

PERK and LoRA Readout look alike on the surface. Both write a context into a LoRA with NTP and then answer without the context, and neither needs QA about a new context. Beyond this, the two approaches learn different things and aim at different kinds of generalization.

PERK learns a task-specific solver. PERK meta-learns the writing process (adapter initialization, per-step learning rates, and a token-weighting network) by backpropagating one task's QA loss through the writing steps. It trains one model per task (e.g., BabiLong QA1) and evaluates it on the same task at longer contexts. On BabiLong, its training target also includes the gold supporting facts. The learned initialization thus absorbs the task's answer format and reasoning pattern, so a new task needs new QA supervision and new meta-training.

We learn a task-agnostic reader. Our writer is plain NTP from a fixed initialization and contains nothing learned. A separate reader is trained once, on memories from five source datasets, and evaluated on five datasets it never saw, with no QA from the target tasks. It learns how to query the storage format of NTP memories, not how to solve a particular task.

  • PERK learns the writer: it minimizes E L(θ+Write(θ,K))\mathbb{E}\,\mathcal{L}\big(\theta + \mathrm{Write}(\theta, K)\big) over θ\theta, on one task's (K,q,a)(K, q, a).
  • Ours fixes the writer, ϕ(K)=Write(ϕ0,K)\phi(K) = \mathrm{Write}(\phi^{0}, K), and minimizes E L(ψ;ϕ(K))\mathbb{E}\,\mathcal{L}\big(\psi; \phi(K)\big) over the reader ψ\psi, on the source datasets' (K,q,a)(K, q, a).

We also use a meta-learning algorithm (Reptile), but only as the optimizer that trains the reader across frozen memories. It never touches the writer.

PERKLoRA Readout
What is learnedThe writerThe reader only
Writer at deploymentMeta-learned for the taskPlain NTP, fixed
Training scopeOne model per taskOne reader for all tasks
EvaluationSame task, longer contextsFive unseen datasets
Extra supervisionGold supporting facts (BabiLong)None beyond QA and same-document negatives
AuthorsOpenReview ↗

(2/6) Why this difference matters, and a matched comparison (Q1)

The two ideas lead to different properties.

Transfer without target-task supervision. This is what our method is designed for: one reader serves new documents from new tasks, with no QA from those tasks. PERK is designed for a different goal, strong performance on one task, including at lengths beyond training.

Memories outlive reader updates. A new reader can be deployed on all previously written memories. In PERK, stored memories depend on the meta-learned parameters, so updating them requires rewriting every memory.

Training never differentiates through writing. PERK backpropagates through truncated inner loops, so for 7B/8B backbones it restricts LoRA to the top 4 layers and trains on a 93 GB H100 (PERK, App. B). Our reader is trained first-order with full-depth adapters on Qwen3-8B, in 3.1 hours on one RTX 5090.

Matched comparison. We run two experiments:

  • (a) PERK on our setting (the key test): PERK's public code, trained on our five source datasets and evaluated on our five unseen targets. We use Qwen3-4B, one of our three backbones, where our result is already in Table 1. PERK's public code does not support Qwen3; the revision lists every change we made to run it.
  • (b) Our reader on a PERK task it was never trained on: BabiLong QA3 (three supporting facts, 8K contexts, the 100 public test samples, official BabiLong scorer). We use our existing Qwen3-8B reader, with no BabiLong training.
(a) Qwen3-4B, 5 unseen targetsCPCUADRepliQAROPESQuAIL (Acc.)Avg.
Base LM, question only (Table 1)0.1640.2090.2290.0760.2920.194
PERK0.1850.2970.3190.3530.2860.288
LoRA Readout (Table 1)0.3720.4170.4490.4900.5010.446

Metrics follow Table 1 (ROUGE-L F1; QuAIL is option accuracy).

(b) BabiLong QA3, 8KAccuracy
PERK (reported; Qwen2.5-7B, trained on QA3 with gold supporting facts)65.2
Ours: correct memory + reader21 / 100
Ours: wrong memory + reader12 / 100
Ours: memory only / reader only7 / 100, 3 / 100
Qwen3-8B with the full context32 / 100

(a) On our setting, PERK averages 0.288 vs. 0.446 for LoRA Readout, and is lower on all five targets. It improves over the question-only base model on four of them, but not on QuAIL (0.286 vs. 0.292). This is consistent with (1/6): a writer meta-learned on the source tasks absorbs their answer format and reasoning pattern, and this transfers poorly to unseen tasks. A reader trained on plain NTP memories transfers.

(b) PERK is much stronger on its own task. Our reader uses the memory without any BabiLong supervision (21 vs. 12 with a wrong memory and 7 with the memory alone), but stays below full context and far below PERK. The two are not matched: backbones and supervision differ, and PERK is trained on this exact task. We report (b) to mark the boundary, not as a head-to-head comparison.

Taken together, the two results match the different goals in (1/6). Per-task meta-training is stronger within the task it was trained on. Our reader transfers to unseen tasks without target supervision, but composes three facts poorly.

AuthorsOpenReview ↗

(3/6) When the reader helps and when it hurts (Q2)

The reviewer points out that in App. D.3 the reader can reduce retrieval (78.1% → 49.0% at 4,096 tokens with 20 updates), and asks under what conditions it helps or hurts. Collecting all our results gives a consistent picture.

ConditionEffect of adding the readerSource
Five main benchmarks (natural documents)Improves all fiveTable 2
Natural diagnostic, 20-update memoriesF1 15.85 → 41.45App. A.3
1,000 natural counterfactual pairsBoth variants followed: 12.5% → 32.0%App. C
Synthetic, 128 tokens, 20 updates+28.82 points [19.44, 38.89]App. D.3
Synthetic, 512–2,048 tokens, 20 updatesNo significant differenceApp. D.3
Synthetic, 4,096 tokens, 20 updates78.1% → 49.0%App. D.3
Synthetic, 6 updates, any lengthLower at every lengthApp. D.3

On natural documents in the regime the reader was trained for, it helps consistently. It hurts only in the synthetic grid, which departs from that regime in two ways:

  • It is a stress test built from repeated neutral filler and compact entity identifiers.
  • Its longer documents exceed the longest training context (2,547 tokens).

We therefore read the degradation as a coverage boundary of the current reader rather than a general failure. The next comment shows what fails at this boundary.

Reviewer_NH83OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LoRA Readout, which separates context writing from information access. Each document is encoded into a dedicated LoRA memory through next-token prediction, while a shared reader is trained using cross-document QA supervision, Reptile, and answer-level contrastive learning. For new documents, only raw-text memory writing is required; the reader is reused to answer questions without the original context. Experiments show improvements over the evaluated parametric context-compression methods across multiple QA datasets, while memory replacement and counterfactual interventions examine the effectiveness and limitations of memory access.

Critical strengths

Strength 1: Extensive controls and interventions examine memory use. The paper compares correct and mismatched memories, reader-only and memory-only models, and task-only adaptation. These controls help distinguish memory access from general QA improvements. On 1,000 natural-document counterfactual pairs, success on both factual variants increases from 12.5% with memory alone to 32.0% with the reader, providing evidence that answers respond to the encoded facts.

Strength 2: The diagnostics provide useful evidence about accessibility and failure boundaries. The paper distinguishes document fitting, candidate-answer preference, and free-generation performance. It also examines sensitivity to question wording, document length, and fact position. These analyses help characterize when NTP memories are useful and how output formatting contributes to apparent gains.

Critical weaknesses

Weakness 1: The motivation and theoretical plausibility is unclear. See Questions for authors.

Weakness 2: The experiements cannot show the generalization boundary of the reader. See Questions for authors.

Weakness 3: The incremental contribution over the closest prior work remains unclear. DCD already introduced document-specific LoRA memories combined with a shared knowledge extractor and evaluated NTP memories with that extractor. The core architecture and separation of writing from reading therefore have direct precedents. The main differences lie in Reptile training, answer-level contrastive learning, and data construction. Although the ablations in Table 3 support some of these choices within the proposed setup, the paper lacks a direct comparison with DCD under matched models, data, and training budgets. It therefore remains unclear which design changes provide substantive advances over the existing framework or explain the different gains from NTP memories reported by the two studies. The authors should clarify the specific novelty and validate these contributions through controlled comparisons.

Questions for authors

The key question about this paper is the motivation: Why should we expect a shared reader to generalize across newly encoded parametric memories? How does it differ from the capabilities acquired through the model’s original post-training, and what are its generalization boundaries?

A core assumption of this paper is that a shared reader can be trained to access parametric memories written from different passages through NTP. I remain uncertain about (1) why this assumption is theoretically plausible or (2) why such a reader requires additional training.

Q1: Specifically, let MM be a model that has completed pretraining and post-training, and let MaM_a be the model after encoding passage aa through NTP. The paper aims to learn a reader update rr that can also be reused with models MxM_x corresponding to other passages. Within the claimed scope of generalization, the passage xx changes while rr remains fixed. Why should independently written memories share a sufficiently stable structure for the same reader to access them?

Q2: This assumption raises a more fundamental question: if such a general reading capability exists, why is it not already part of the model’s existing capabilities? The model has undergone extensive pretraining and post-training, and an important role of post-training is precisely to enable it to answer questions using knowledge acquired during pretraining. The original post-training and the proposed reader training therefore appear to address the same problem: using knowledge encoded in parameters through NTP. Why is the extensive original training insufficient for accessing newly encoded NTP memories, while additional reader training can acquire a capability that transfers across passages? What learning signal does this additional training provide that the original training did not?

Explaining this merely by saying that the new knowledge is written after post-training seems insufficient. We can regard a fully pretrained and post-trained model as the initialization for another training process. From this perspective, NTP on a new passage is simply continued pretraining, while the initial parameters already contain the knowledge-use capabilities acquired through earlier post-training. Thus, I am not merely asking whether adding a reader improves empirical performance. Rather, if the original post-training has already taught the model to use knowledge acquired through NTP, why might further NTP produce knowledge that its existing capabilities cannot effectively use? What specific change does the additional reader repair?

If newly encoded knowledge shares sufficiently consistent representations and access mechanisms with existing knowledge, some transfer of the existing QA capability should be expected. If particular training conditions disrupt this transfer—for example, the scope of parameter updates, low-rank constraints, or the writing procedure—the authors should identify those factors. They should then explain why the effects of these factors are sufficiently consistent across passages for a fixed reader to address them. Otherwise, it is difficult to determine whether the method learns a general capability for reading parametric memories or an effective adaptation to a particular writing procedure and QA distribution. I would appreciate a theoretical or mechanistic explanation of the conditions under which such shared reading is possible.

Experimentally, I would also like a more systematic characterization of the scope and boundaries of this generality. Does the same reader remain effective when passages use different languages, concern substantially different domains such as literature, programming, and physics, or vary greatly in length? How far does the capability transfer when questions move from direct factual retrieval to reasoning and counterfactual questions? Do these changes affect memory writing, information access, or subsequent reasoning? I recognize that generalization to all passages and questions is a difficult goal. Precisely for this reason, clearly identifying the conditions for success and failure would help establish how general the proposed shared reader actually is and why its underlying assumption is plausible.

8 comments
AuthorsOpenReview ↗

(1/8) Why the post-trained model cannot use a new NTP memory (Q2)

We thank the reviewer for the thoughtful review. We agree that the central question is why a shared reader should work, not only whether it does.

Short answer: post-training learned to read knowledge in formats a new NTP memory does not have.

The reviewer is right that post-training teaches a model to use knowledge acquired through NTP. That knowledge, however, was acquired in a specific way. During pretraining a fact is seen many times, in many phrasings, through full-parameter updates. Allen-Zhu & Li (2023) show that this diversity is what makes knowledge extractable by questions; knowledge stored without it is present in the weights but poorly accessible.

A memory written by our writer has none of these properties. It is a single document in a single phrasing, written in 6 high-learning-rate passes into a rank-32 additive update. Post-training never had to read this format.

The mismatch also has a concrete mechanistic form. A memory branch adds ΔWh\Delta W h to a layer's output, where ΔW\Delta W is the memory's low-rank update and hh is the layer's input. Its NTP gradients align the input directions of ΔW\Delta W with hidden states produced while continuing the document (e.g., "X was born in"). At inference the model is in a question state (e.g., "Where was X born?"), and the memory responds only insofar as the two states align. This is the same conditioning mismatch that underlies the reversal and factorization curses (Berglund et al., 2023; Kitouni et al., 2024).

Three results in the submission support this account:

  1. Writing more makes the memory fit better but harder to use. From 6 to 20 writing updates, median document loss falls from 0.0975 to 0.0238 nats/token, while memory-only F1 drops from 24.86 to 15.85 (App. A.3). If post-trained QA ability transferred to NTP memories, better fitting should not hurt. It does.
  2. The reader works partly by changing what the frozen memory receives. On synthetic pairs, the reader raises correct-answer log-odds by 4.59 nats. The gain falls to 1.09 when memory-branch outputs are frozen at their reader-off values. Switching only those outputs to their reader-on values recovers 3.50 (App. A.5).
  3. The memory signal that matters is localized. Removing late-MLP memory branches cuts natural free-generation F1 by 10.74 points (95% CI [−17.97, −3.51]), while removing late attention branches has no measurable effect (App. B.3).

This account has a limit, which we state explicitly. On natural QA, prompt-position memory outputs explain only a small part of the reader's likelihood gain (4.60 vs. 4.51 nats/token with outputs frozen; App. A.5). General task adaptation and answer formatting also contribute, and the task-only controls in App. A.4 quantify part of this.

AuthorsOpenReview ↗

(2/8) What learning signal reader training adds (Q2)

The reviewer asks what signal reader training provides that post-training did not. Our answer is a specific contrast that post-training data never contains: the same question template has different answers depending only on which low-rank memory is attached.

Three elements of our training data create this contrast:

  • Consistent entity replacement makes every association new, so answers cannot come from pretraining knowledge.
  • Same-context negative answers have the same type and source as the reference, so only memory content separates them.
  • Episodes grouped by memory tie each update to one memory.

To show that this signal, and not generic QA training, drives the effect, the submission compares against task-only readers. These match ours in capacity, gold-QA exposure and order, and number of updates, but are trained without memory:

ComparisonMemory-trained readerTask-only readerSource
Benefit of attaching the correct memory (difference-in-differences)+7.17 F1 more [1.61, 12.71]—App. B.4
Natural counterfactual pairs: both factual variants followed32.0%11.2% (memory alone: 12.5%)App. C
Shift in preference toward the edited fact34.89 nats11.59 natsApp. C

General QA adaptation explains part of the improvement (App. A.4). Training with memories adds a measurable, memory-specific benefit on top of it.

What this suggests for training base models. The account in (1/8) and the contrast above point beyond our setting. A post-trained model cannot read a new NTP memory because no stage of its training ever asked it to answer or reason from a parameter update written on a single text. Our reader training adds this stage late and at small scale: 33.75M training input tokens for memory writing and reader training, with the backbone frozen. If the same objective were part of pretraining or post-training (write one text into the parameters with NTP, then answer or reason from that update), we expect the model to read such updates much better, without a separate reader. A model trained this way well would make the properties discussed here native: it could internalize a new document with a few NTP updates, keep it as a small reusable memory, and answer without the document in the prompt. We see our results as a first step in this direction and will discuss it in the revision.

Beyond factual knowledge. We evaluate knowledge-intensive QA because memory use can be measured exactly there. The approach itself is not specific to knowledge. The writer is plain NTP on raw text and makes no assumption about what the text contains, so any information expressed as text can be turned into parameters in the same way, for example instructions, or constraints on language style and format. A reader trained on such memories would apply them without the text in the prompt. This submission does not evaluate these uses; we will state them in the paper as a direction the framework supports, not as a result.

AuthorsOpenReview ↗

(3/8) Why one reader can serve independently written memories (Q1)

The reviewer asks why independently written memories should share a structure stable enough for a single reader. The answer is that they are all produced by the same writing operator: same backbone, initialization, writing prefix, optimizer, and step budget. Their content differs, but their storage format does not: which layers and modules carry the signal, and how input directions relate to document-continuation states. The reader learns a mapping from question states into this format, and that mapping does not depend on what a particular document says.

This also answers the reviewer's question of whether the method learns "a general capability for reading parametric memories or an effective adaptation to a particular writing procedure." It is the latter, and it is general across content. We now state this scope explicitly in the paper:

The reader generalizes across content, i.e., unseen contexts written by the same writing operator. It is not designed to generalize across writers.

We consider this the useful kind of generality. The writer is fixed, cheap, and needs no supervision, so any new document written by it can be read.

The submission's evidence is consistent with this scope:

  • Across content: gains on five unseen target datasets and three backbones without reader adaptation (Table 1).
  • Across writers, only partial transfer: a reader trained on 2-update memories reaches 36.23 F1 on 20-update memories, vs. 41.93 when budgets match (Fig. 5b).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.