- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Paired Memory Adaptation (PMA), which converts documents into reusable low-rank weight updates using separate read/key and write/value memories, with key-conditioned value construction and layer-specific readout maps. The mechanism is trained through context distillation; new documents then require no per-document gradient optimization. Under a common 600-step training budget, PMA improves document acquisition and selected QA results relative to the configured D2L and SHINE baselines.
Critical strengths
-
Promising performance under a short training budget. It also uses approximately 19% fewer trainable parameters than the configured SHINE baseline (208.30M versus 257.31M).
-
Reusable document adaptation. Generated updates support multiple questions without retaining document text in the answering context or performing document-specific optimization.
Critical weaknesses
-
The baseline comparison is not fair. SHINE is retrained without its reconstruction/completion pretraining and QA instruction-tuning stages; its projection coverage, Meta-LoRA rank, and regularization are also changed. Although this provides a controlled comparison under a shared recipe, it does not establish superiority over SHINE trained with its original procedure. Longer-budget learning curves and a baseline following SHINE’s original training procedure would strengthen the evidence.
-
Limited evidence of generality. Experiments use one backbone. Additional model families and scales are needed to determine whether the observed advantages generalize.
-
The acquisition–retention tradeoff remains unresolved. PMA does not uniformly outperform SHINE: SHINE performs better on DROP, HotpotQA shows no clear difference, and PMA preserves fewer out-of-scope answers under acquisition-only training. Preservation supervision improves retention but reduces acquisition. Comparing methods at matched retention levels would better establish whether PMA offers a preferable overall tradeoff.
Questions for authors
none
2 comments
The author's response mostly solves my concern of weakness 1: I can agree the comparison is fair. However, weakness 2 and 3 are still not solved. In my view, PMA's advantage is not too limited to training efficiency. So I will keep my score.
Thank you for the review. It examines the comparison with SHINE more closely than any other, and this is where the new evidence says the most: PMA's advantage over SHINE is training cost and speed.
1. The comparison with SHINE.
Fairness of the comparison. Comparing architectures requires training all three the same way, so we unified the training recipe, and we did not write that recipe ourselves. It is D2L's: its released code, data, distillation loss and default optimizer settings, used for D2L, SHINE and PMA alike. We did not tune it for PMA. Its learning rate (4e-5) lies within the range that SHINE's own training stages use (1e-5 to 5e-5). SHINE's original procedure adds pretraining on a 6B-token corpus and instruction tuning on question-answering data that include MS MARCO, SQuAD, DROP and HotpotQA, four of our six evaluation sets. Giving those stages to SHINE alone would not be a fair comparison either: the same pretraining and tuning would have to be given to PMA and D2L. That is a different and far larger experiment. The paper's claim is about the three architectures under one recipe.
Longer-budget curves. We trained all three systems for 5,212 updates, 8.7 times the paper's budget, under the shared recipe: the same batches in the same order, one pass over a set with four times as many documents. SHINE learns under this recipe, later than PMA. To reach the DocAR direct score that PMA has at the 600-, 1,303- and 2,606-update checkpoints of its run (32.2, 42.9 and 49.6), SHINE needs about 1,560, 2,720 and 3,890 updates (interpolating between its checkpoints): 2.6, 2.1 and 1.5 times as many. PMA's lead on DocAR direct at the three checkpoints is 21.2 [16.8, 25.6], 13.1 [8.0, 18.0] and 7.4 [2.4, 12.5] points. At 5,212 updates SHINE has drawn level: 47.6 against 47.7 on SQuAD and 52.1 against 54.7 on DocAR direct (difference +2.6 [−2.0, +7.2]).
Cost of an update. PMA has no hypernetwork to generate the low-rank update: the factors are read from the backbone's own activations at the key and value tokens. SHINE generates them with a four-block Transformer that every update must run and backpropagate through. In addition, PMA never backpropagates through the document; SHINE does, because its Meta-LoRA acts on the document tokens. On one four-GPU node, with the same student pass and loss code, a PMA update takes 4.0 s. A SHINE update takes 13.3 s in the fastest setting we could give it (no layer checkpointing, compiled student pass), 3.3 times PMA's; 82% of it is the generator and the document pass with its backward. The gap grows with document length: with the documents lengthened to about 2,300 tokens, SHINE no longer fits on an 80 GB card without checkpointing and its update costs 4.5 times PMA's.
PMA's training path was re-implemented after submission, so these timings are faster than those in the paper's appendix (the paper's 600-update rows reproduce on it). To compare efficiency, D2L and SHINE are timed on the same fast path: the same optimized student pass and loss code for all three.
Accuracy against compute. PMA after 2,606 updates (12 GPU-hours) is level with SHINE after 5,212 (77 GPU-hours): SQuAD +0.4 [−4.4, +5.4], DocAR direct −2.5 [−7.5, +2.1], indirect −1.4 [−5.2, +2.3]. With SHINE's compute, PMA does not stop at 5,212 updates: a longer PMA run, on a larger set from the same data, reaches 15,636 updates in 70 GPU-hours and is then ahead of SHINE's endpoint by 12.6 [7.7, 17.6], 12.1 [7.7, 16.5] and 12.3 [9.0, 15.7] points on SQuAD, DocAR direct and DocAR indirect. By then it has read three times as many documents, which is what a cheaper update buys. We did not train SHINE at that scale, where the same updates would cost it about 230 GPU-hours.
Meta-LoRA rank and dropout. We reran SHINE at 600 updates with the Meta-LoRA rank of its paper (128) and without our dropout. Its scores change by +1.0 on SQuAD [−1.9, +4.1] and +0.5 on DocAR direct [−0.8, +1.9], so these two settings do not explain its distance to PMA at 600 updates (14 and 16 points).
2. Seeds. The comparison with D2L at 600 updates now covers three training seeds (check the first comment to all reviewers , point 3). For SHINE, the 600-update checkpoint of its 5,212-update run agrees with the paper's row (30.4 and 11.0 against 29.3 and 8.8).
3. Acquisition and retention. Against D2L, the paper's preservation-supervision table is the closest to a comparison at matched retention, and there retention is trained for. With the preservation loss, PMA has 31.7 on direct and 18.6 on indirect acquisition at retention 44.8 (near-miss) and 46.0 (unrelated); D2L has 17.3 and 16.5 at 47.5 and 48.4. PMA acquires 14 points more on direct at 2 to 3 points lower retention. Against SHINE we do not claim a better trade-off: at 5,212 updates the two acquire the same and SHINE keeps more of the base model's answers.