acceptodds
Under review as a conference paper at ICLR 2027

Turning Documents into Weight Updates with Paired Memory Tokens

Abstract

Large language models use new documents through prompting or fine-tuning, requiring either the document in context or additional parameter optimization. We introduce Paired Memory Adaptation (PMA), which learns to convert documents into reusable low-rank weight updates. PMA constructs separate read and write memories inside the frozen language model. Each read memory guides the construction of its paired write memory, and learned projection maps turn their states into update factors. We train the mechanism through context distillation: the adapted model matches a teacher that receives the document in context. New documents then produce updates without per-document gradient optimization. We evaluate fact recall and application in new situations on our Document Acquisition and Retention (DocAR) dataset, with answers assessed by a separate language model. After the same short, 600-step training on Qwen3-4B, PMA outperforms retrained Doc-to-LoRA (D2L) and SHINE on these tests with 50.5% and 19.0% fewer trainable parameters. The share of answers rated fully correct exceeds the unadapted model's by 21.65 percentage points on SQuAD (token F1 and a second judge agree; exact match does not). On most tasks, PMA's performance drops when the update comes from an unrelated document. Removing the learned readout reduces PMA's trainable parameters by 54.4% while retaining useful acquisition, although some public QA scores fall below the unadapted model's. Gains vary across tasks, and updates can disrupt answers to unrelated questions. In separate PMA and D2L experiments, training to preserve those answers reduces this disruption but also reduces gains on document questions. D2L's released checkpoint, trained far longer, beats PMA on nearly all tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, document adaptation, low-rank adaptation, memory tokens, context distillation, knowledge acquisition, knowledge preservation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FgMvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Paired Memory Adaptation (PMA), which converts documents into reusable low-rank weight updates using separate read/key and write/value memories, with key-conditioned value construction and layer-specific readout maps. The mechanism is trained through context distillation; new documents then require no per-document gradient optimization. Under a common 600-step training budget, PMA improves document acquisition and selected QA results relative to the configured D2L and SHINE baselines.

Critical strengths
  1. Promising performance under a short training budget. It also uses approximately 19% fewer trainable parameters than the configured SHINE baseline (208.30M versus 257.31M).

  2. Reusable document adaptation. Generated updates support multiple questions without retaining document text in the answering context or performing document-specific optimization.

Critical weaknesses
  1. The baseline comparison is not fair. SHINE is retrained without its reconstruction/completion pretraining and QA instruction-tuning stages; its projection coverage, Meta-LoRA rank, and regularization are also changed. Although this provides a controlled comparison under a shared recipe, it does not establish superiority over SHINE trained with its original procedure. Longer-budget learning curves and a baseline following SHINE’s original training procedure would strengthen the evidence.

  2. Limited evidence of generality. Experiments use one backbone. Additional model families and scales are needed to determine whether the observed advantages generalize.

  3. The acquisition–retention tradeoff remains unresolved. PMA does not uniformly outperform SHINE: SHINE performs better on DROP, HotpotQA shows no clear difference, and PMA preserves fewer out-of-scope answers under acquisition-only training. Preservation supervision improves retention but reduces acquisition. Comparing methods at matched retention levels would better establish whether PMA offers a preferable overall tradeoff.

Questions for authors

none

2 comments

Thank you for the review. It examines the comparison with SHINE more closely than any other, and this is where the new evidence says the most: PMA's advantage over SHINE is training cost and speed.

1. The comparison with SHINE.

Fairness of the comparison. Comparing architectures requires training all three the same way, so we unified the training recipe, and we did not write that recipe ourselves. It is D2L's: its released code, data, distillation loss and default optimizer settings, used for D2L, SHINE and PMA alike. We did not tune it for PMA. Its learning rate (4e-5) lies within the range that SHINE's own training stages use (1e-5 to 5e-5). SHINE's original procedure adds pretraining on a 6B-token corpus and instruction tuning on question-answering data that include MS MARCO, SQuAD, DROP and HotpotQA, four of our six evaluation sets. Giving those stages to SHINE alone would not be a fair comparison either: the same pretraining and tuning would have to be given to PMA and D2L. That is a different and far larger experiment. The paper's claim is about the three architectures under one recipe.

Longer-budget curves. We trained all three systems for 5,212 updates, 8.7 times the paper's budget, under the shared recipe: the same batches in the same order, one pass over a set with four times as many documents. SHINE learns under this recipe, later than PMA. To reach the DocAR direct score that PMA has at the 600-, 1,303- and 2,606-update checkpoints of its run (32.2, 42.9 and 49.6), SHINE needs about 1,560, 2,720 and 3,890 updates (interpolating between its checkpoints): 2.6, 2.1 and 1.5 times as many. PMA's lead on DocAR direct at the three checkpoints is 21.2 [16.8, 25.6], 13.1 [8.0, 18.0] and 7.4 [2.4, 12.5] points. At 5,212 updates SHINE has drawn level: 47.6 against 47.7 on SQuAD and 52.1 against 54.7 on DocAR direct (difference +2.6 [−2.0, +7.2]).

Cost of an update. PMA has no hypernetwork to generate the low-rank update: the factors are read from the backbone's own activations at the key and value tokens. SHINE generates them with a four-block Transformer that every update must run and backpropagate through. In addition, PMA never backpropagates through the document; SHINE does, because its Meta-LoRA acts on the document tokens. On one four-GPU node, with the same student pass and loss code, a PMA update takes 4.0 s. A SHINE update takes 13.3 s in the fastest setting we could give it (no layer checkpointing, compiled student pass), 3.3 times PMA's; 82% of it is the generator and the document pass with its backward. The gap grows with document length: with the documents lengthened to about 2,300 tokens, SHINE no longer fits on an 80 GB card without checkpointing and its update costs 4.5 times PMA's.

PMA's training path was re-implemented after submission, so these timings are faster than those in the paper's appendix (the paper's 600-update rows reproduce on it). To compare efficiency, D2L and SHINE are timed on the same fast path: the same optimized student pass and loss code for all three.

Accuracy against compute. PMA after 2,606 updates (12 GPU-hours) is level with SHINE after 5,212 (77 GPU-hours): SQuAD +0.4 [−4.4, +5.4], DocAR direct −2.5 [−7.5, +2.1], indirect −1.4 [−5.2, +2.3]. With SHINE's compute, PMA does not stop at 5,212 updates: a longer PMA run, on a larger set from the same data, reaches 15,636 updates in 70 GPU-hours and is then ahead of SHINE's endpoint by 12.6 [7.7, 17.6], 12.1 [7.7, 16.5] and 12.3 [9.0, 15.7] points on SQuAD, DocAR direct and DocAR indirect. By then it has read three times as many documents, which is what a cheaper update buys. We did not train SHINE at that scale, where the same updates would cost it about 230 GPU-hours.

Meta-LoRA rank and dropout. We reran SHINE at 600 updates with the Meta-LoRA rank of its paper (128) and without our dropout. Its scores change by +1.0 on SQuAD [−1.9, +4.1] and +0.5 on DocAR direct [−0.8, +1.9], so these two settings do not explain its distance to PMA at 600 updates (14 and 16 points).

2. Seeds. The comparison with D2L at 600 updates now covers three training seeds (check the first comment to all reviewers , point 3). For SHINE, the 600-update checkpoint of its 5,212-update run agrees with the paper's row (30.4 and 11.0 against 29.3 and 8.8).

3. Acquisition and retention. Against D2L, the paper's preservation-supervision table is the closest to a comparison at matched retention, and there retention is trained for. With the preservation loss, PMA has 31.7 on direct and 18.6 on indirect acquisition at retention 44.8 (near-miss) and 46.0 (unrelated); D2L has 17.3 and 16.5 at 47.5 and 48.4. PMA acquires 14 points more on direct at 2 to 3 points lower retention. Against SHINE we do not claim a better trade-off: at 5,212 updates the two acquire the same and SHINE keeps more of the base model's answers.

The author's response mostly solves my concern of weakness 1: I can agree the comparison is fair. However, weakness 2 and 3 are still not solved. In my view, PMA's advantage is not too limited to training efficiency. So I will keep my score.

Reviewer_4GqAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper builds on the doc-to-LoRA literature and proposes a modification to the hypernetwork architecture that maps a document to a LoRA adapter so that a student model with the LoRA gains knowledge from the document. The key difference is that PMA appears to be slightly simpler: there is no separate readout network as in prior work; rather, the LoRA factors are taken directly from the hypernetwork’s activations. Additionally, when constructing a value for a LoRA dim, the corresponding key’s final hidden state is passed in as a soft input token. The authors provide some evidence suggesting their method may be more sample efficient: it has fewer learnable params, and in a short training budget (600 steps), PMA outperforms prior methods for adapting Qwen3-4B-Instruct.

Critical strengths
  • Improving the sample/meta-training efficiency of document-to-LoRA hypernetworks is a valuable direction, since the meta-training cost of prior methods are quite high.
  • Although the motivation is somewhat unclear in places, I appreciate the general flavor of reasoning from first principles about architecture and expressivity when designing the hypernetwork, rather than adding a generic readout network.
Critical weaknesses
  • The experiments are limited to a single model and seed, and most concerningly to a very small training budget (600 steps, compared to 100K for Doc-to-LoRA), where I worry all the results are comparisons between undertrained models. If the result were that PMA at the lower training budget outperforms D2L at its full step budget, that would be compelling. But the authors only show that, in this really undertrained regime, PMA does better than the baselines, while it still substantially underperforms D2L trained with the full budget, or simply placing the document in context. In almost all benchmarks, PMA and the other methods evaluated at 600 steps are much closer to the frozen base model’s performance than to released D2L (and the in-context document is higher still).
  • Poor clarity: Much of the prose reads as though it was primarily language-model-generated, and is hard to follow due to strange terminology and un-specific statements. I acknowledge that some of this is a matter of taste, but I think it is worth raising as a separate critique from the weakness of the results themselves. Notably, many of the figure captions are hard to follow, as is the core method section itself.
Questions for authors

please see my concerns about the small training step count

1 comment

Thank you for the review, you name the result that would be compelling: PMA at a lower budget outperforming D2L at its full budget. This is where the evidence now stands (numbers in our response to all reviewers).

PMA at 600 updates is above D2L at 5,212. Under one recipe, with 8.7 times the budget, D2L is still below the paper's PMA row: by 11.5 points on SQuAD, 15.1 on DocAR direct and 5.7 on DocAR indirect, all with intervals that exclude zero. Giving D2L 8.7 times the budget does not change the comparison.

Against the released checkpoint. Against D2L our claim is both accuracy and efficiency. With equal training PMA is the more accurate system on SQuAD and DocAR, at 600 updates and at 5,212 (response to all reviewers, point 1). Against the fully trained checkpoint, PMA reaches most of its accuracy with a small fraction of its training. A longer PMA run, after 15,636 updates against its 100,000, is level with it on DocAR (direct 64.2 against 66.6, indirect 55.7 against 53.0; both intervals include zero) and has 60.2 against 74.8 on SQuAD, 72% of the released checkpoint's gain over the base model (21.9). On SQuAD and DocAR, PMA is then closer to the released checkpoint than to the base model.

And 16% of the updates overstates PMA's share of the training, because its updates are smaller and faster. The run covers 6% of the released checkpoint's context tokens (2.8 B against 48.5 B), and on the same hardware a PMA update takes 4.0 s against 9.1 s for a D2L update of the same size (timings in our reply to Reviewer FgMv). By these timings, PMA's 15,636 updates take about 7% of the time of 100,000 D2L updates.

Training cost is the problem the paper addresses. The D2L paper names it as its open problem (a training run on a 2B model "takes around 5 days on 8 H200 GPUs", and "significantly improving the efficiency of the meta-training phase is a fruitful research direction for future work"), and you name it as the valuable direction. PMA has no generator network and never backpropagates through the document. Against SHINE, the other recent method, this gives the same accuracy for a sixth of the compute, and twelve points more for the same compute (check the first comment to all reviewers , point 2).

Seeds. The comparison with D2L now covers three training seeds (response to all reviewers, point 3).

Clarity. Section 3 and several captions are hard to follow, and we will rewrite them. In LoRA terms, the method is:

  • For a document, PMA produces a rank-8 update ΔWl=γBlAl\Delta W_l = \gamma B_l A_l to the MLP output projection of every layer.
  • It appends 8 key tokens and 8 value tokens to the document and runs the frozen backbone over them. Low-rank attention adapters act only at these 16 positions; the document itself is read by the unmodified backbone.
  • Row ii of AlA_l is the activation entering that projection at key token ii. Column ii of BlB_l is the residual-stream state after layer ll at value token ii. Each passes through a learned linear map, one per layer. No separate network generates the factors.
  • Value token ii takes the final state of key token ii as its input embedding, through a learned map. This is the pairing.
  • Answering uses Wl+ΔWlW_l + \Delta W_l alone: no document, no extra tokens, no adapters.

In the final version we will rewrite Section 3 in this form, add two short algorithms (encoding a document; answering with its update), use one name for each object (key and value tokens; rows of AA and columns of BB), and rewrite the figure captions.

Reviewer_NfCYOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Converts a document into a low-rank weight update in one forward pass. The method adds two sets of memory tokens. “Key” tokens read the document. Each key then shapes a paired “value” token. Their activations become the two halves of a low-rank update. The system learns this by copying a teacher model that has the document in context.

Critical strengths
  1. A clean idea: Matching memory tokens to the read and write roles of a low-rank update is well motivated.
  2. Good controls for document use (correct vs. unrelated document), a readout-free ablation, and a second judge.
  3. Candid reporting, including the released D2L’s superiority and the retention cost.
Critical weaknesses
  1. Unfair comparison: All three methods train for only 600 steps. Doc-to-LoRA normally trains for about 100,000, and its document signal starts switched off. SHINE runs without its pretraining. The released Doc-to-LoRA still beats PMA on almost every task.

  2. The results rest on one LLM judge. The judge was chosen after looking at the other metrics. On SQuAD, exact match puts PMA below the base model. On ROPES and HotpotQA, F1 does too.

  3. The pairing idea is not tested. Pairing is the paper’s main contribution but PMA still works well without the learned readout. Replacing the paired value inputs changes little. The gains may come from other parts of the design.

Questions for authors
  1. Is the comparison fair? All three methods train for only 600 steps. Doc-to-LoRA normally trains for about 100,000, and its document signal starts switched off. SHINE is run without its pretraining. How do the methods compare when each is trained to convergence? Can you show learning curves over a longer run?

  2. Can the results be trusted without the LLM judge? Most results come from one LLM judge, chosen after looking at the other metrics. On SQuAD, exact match puts PMA below the base model. On ROPES and HotpotQA, F1 does too. Can you add a human evaluation, even on a small sample? Can you report results over several training seeds?

  3. Does the pairing actually matter? Pairing is the main idea. But PMA still works well without the learned readout. Replacing the paired value inputs also changes little. Can you run an ablation that removes only the pairing and keeps everything else?

  4. DocAR documents average 55 tokens. How does PMA do on long documents?

1 comment

Thank you for the review. We follow your four questions.

1. Is the comparison fair? It is a controlled comparison: one recipe, one data stream and one student pass for three architectures. The recipe is not ours: it is D2L's released one, and we did not tune it for PMA. Your question is what happens with more training (check the first comment to all reviewers). Against D2L the ranking holds: with 8.7 times the budget PMA's lead is intact (SQuAD +15.7, DocAR direct +44.7), and D2L at 5,212 updates is still below PMA at 600.

D2L's document signal starts switched off. The paper's appendix ("D2L's gate and a longer training check") already trains D2L with that scale initialised at one, and for 1,300 updates. Its SQuAD accuracy changes by +0.8 to +1.9 points and its DocAR direct accuracy stays within 1.0 point of the main D2L run. The initialisation is not what holds D2L back.

SHINE. It draws level with PMA at 5,212 updates, at three to six times the training compute (check the first comment to all reviewers , point 2). Its pretraining was not added because, to be fair, the same pretraining would then have to be given to PMA and D2L (reply to Reviewer FgMv).

2. Can the results be trusted without the judge? Yes. We add a score that needs no judge and no choice from us: whether a gold answer appears in the output, on all 497 SQuAD questions and three training seeds.

Gold answer contained (%)
Base model, no document16.9
D2L, three seeds17.1 / 16.7 / 16.9
SHINE, one seed20.7
PMA, three seeds32.2 / 29.2 / 34.2
PMA with an unrelated document, three seeds7.0 / 8.7 / 6.8

PMA's lead on SQuAD is there without a judge and on every seed, and it disappears when the update comes from another document.

Exact match. It also puts the released D2L checkpoint (1.4) below the base model (8.9) on SQuAD, though that checkpoint is the strongest of the adapted systems on SQuAD by every other measure. On this backbone exact match follows answer length: over PMA's three seeds it is 5.2, 10.1 and 1.2 while containment stays between 29 and 34. The D2L paper reports ROUGE-L recall in place of F1 for this backbone for the same reason (its Appendix E).

One judge. The paper's appendix has a second judge from another model family (Opus 5.5); the two agree on 96.3% of SQuAD decisions (κ = 0.92).

ROPES and HotpotQA. The paper claims no gain over the base model on either set: it reports that the ROPES interval includes zero and that HotpotQA "shows no clear difference", these datasets are a minority in the training data.

Human evaluation. In its place we offer the containment score above, which involves no judge at all, and the second judge.

3. Does the pairing matter? The mechanism has two parts, and the paper's ablations separate them.

First, both factors of the update are read from the backbone's own activations at dedicated tokens, paired by index: key ii and value ii form the ii-th rank-one term. This is what replaces the generator network, and on DocAR it carries most of the gain. As you note, PMA works without the learned readout: at 600 updates that variant reaches 21.0 on DocAR direct, against 8.8 for SHINE and 6.6 for D2L.

Second, value token ii starts from the final state of key ii. The ablation you ask for is in the paper ("trained value embeddings"): it replaces that input with a learned constant and keeps everything else. The paper compared two runs with different sample orders; we reran full PMA with the ablation's order, so the two now differ only in that input. Removing it costs 4.0 points on SQuAD [−0.9, 8.4] and 2.3 on DocAR indirect [0.3, 4.2]; the other measures show no difference. The key-to-value input is the smaller part of the mechanism, and a measurable one.

4. Long documents. The 55-token average describes DocAR only, which is short by design. The other evaluation documents are longer: on average 164 tokens for SQuAD, 202 for ROPES, 269 for DROP, 795 for MS MARCO and 1,318 for HotpotQA (up to 3,004), all encoded whole. The paper's Table 2 covers the last two. On MS MARCO, PMA gains 5.3 points over the base model across all turns at 600 updates. On HotpotQA no system trained for 600 updates is above the base model, and the released D2L checkpoint, with its full training, gains 4.7 points where the model with the document in context reaches 81.2. Multi-passage documents of this length remain hard for this family of methods.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Longer training, more seeds, and training cost

We thank the three reviewers. D2L and SHINE both use a hypernetwork to generate the low-rank update. PMA removes the hypernetwork and reads the low-rank update from the backbone's own activations at the key and value tokens. The reviews share one concern about its evidence: at 600 updates every system is undertrained, so the comparison might not survive more training. We extended the training to 8.7 times as many steps. PMA's advantage is training efficiency, and it holds at every budget we measured.

SystemUpdatesGPU-hoursSQuADDocAR directDocAR indirect
Base model, no document21.93.912.9
D2L (paper)60024.76.615.3
SHINE (paper)60029.38.816.7
PMA (paper)60043.525.026.8
D2L5,2125332.010.021.1
SHINE5,2127747.652.143.4
PMA5,2122347.754.744.1
PMA, longer run15,6367060.264.255.7
D2L, released checkpoint100,00074.866.653.0

The 5,212-update runs read the same batches in the same order, one pass over a set with four times as many documents. The 15,636 row is from a longer PMA run, three quarters of one pass over a set with 16x as many documents. GPU-hours are updates times the measured seconds per update on four A100 GPUs, with the same student pass for all and each system in its fastest setting (PMA 4.0 s, D2L 9.1 s, SHINE 13.3 s). One seed per system in these runs.

1. Against D2L, PMA stays ahead with 8.7 times the budget. At 5,212 updates PMA leads by 15.7 points on SQuAD [10.7, 20.2], 44.7 on DocAR direct [39.6, 50.1] and 23.0 on DocAR indirect [19.1, 26.9]; at 600 updates the leads were 18.8, 18.5 and 11.5. D2L does learn from the additional documents, but after 5,212 updates it is still below the paper's 600-update PMA row: by 11.5 points on SQuAD [6.4, 16.4], 15.1 on DocAR direct [11.2, 19.2] and 5.7 on DocAR indirect [3.1, 8.2].

2. Against SHINE, PMA's edge is training cost and speed, and it is large. Under the shared recipe SHINE does learn, later than PMA, and draws level with it at 5,212 updates. But a SHINE update costs 3.3 times a PMA update, for a reason in the design: PMA has no hypernetwork to generate the low-rank update, and it never backpropagates through the document.

  • The same accuracy for a sixth of the compute. PMA after 2,606 updates (12 GPU-hours) is level with SHINE after 5,212 (77 GPU-hours) on every judged measure.
  • Twelve points more for the same compute. In less compute than SHINE's run (70 against 77 GPU-hours) a longer PMA run reaches 15,636 updates over three times as many documents, and is ahead by 12.6 points on SQuAD [7.7, 17.6], 12.1 on DocAR direct [7.7, 16.5] and 12.3 on DocAR indirect [9.0, 15.7].

The curves and the details of these timings are in our reply to Reviewer FgMv.

3. The lead over D2L holds across training seeds. We repeated the 600-update runs of PMA and D2L with two more seeds. PMA minus D2L on SQuAD is +18.8, +15.5 and +18.0. On DocAR, for the two seeds whose answers were stored, it is +18.5 and +17.8 on direct and +11.5 and +12.5 on indirect. Every interval excludes zero. On DROP and ROPES, which the paper already reports as uncertain, the lead is present at two seeds and absent at the third; we make no claim on these two sets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.