- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper revisits an explicit copy distribution for decoder language models. A Copy-Paste head attends to previous token states and aggregates probability over positions containing the same vocabulary item. Its output is mixed with the LM distribution using a gate from a dedicated LM-head token and a sink-based fallback. The model is trained with ordinary next-token cross-entropy, without copy labels. Experiments cover pretraining from 135M to 1.7B parameters, full and local attention, hybrid attention, and smaller Gated DeltaNet models. CPLMs improve perplexity, yield modest downstream gains, and provide useful cache-memory tradeoffs for local backbones and document retrieval. The paper attributes much of the benefit to an additional gradient pathway around the vocabulary head, supported by gradient measurements, a synthetic task, a stop-gradient intervention, and byte-level training.
Critical strengths
-
The empirical investigation is substantial and well controlled. Matched training recipes, several scales and backbones, a window-size sweep, and three seeds at 135M establish that the perplexity gains are not confined to a single architecture or a small random fluctuation (Sections 3.1, A, and C.1–C.5). Full-attention comparisons are particularly valuable because they distinguish the effect of the output mechanism from merely adding global access to a local model.
-
The paper positions its contribution honestly. Section 2.2 acknowledges pointer-sentinel mixtures, and Appendix C.6 compares several pointer designs under a common setup. Table 8 shows that the proposed gate is only slightly better than, or tied with, conventional pointer alternatives. The authors explicitly frame the contribution as a modern pretraining study and analysis, rather than inventing copying. The comparison with a tuned inference-time neural cache is also useful.
-
Several results have practical and scientific value. The 1.7B SW model's cache/perplexity tradeoff, the document-keys-only HotpotQA setup, and the reduction in induction heads provide different views of explicit memory. The paper reports failures as well as successes: full models still fail on sufficiently long retrieval inputs, and post-SFT LongBench gains are uneven. These qualifications make the results easier to interpret.
-
The proposed explanation is tested, not merely asserted. The stop-gradient and byte-level studies in Appendix H are useful interventions. They provide stronger evidence about training dynamics than the gradient-projection plots alone, even though they do not fully identify the proposed mechanism.
Critical weaknesses
-
The evidence supports useful backbone–copy-head co-adaptation more directly than it establishes gradient-bottleneck relief as the dominant cause. Appendix H.1 removes the direct gradients from the copy queries and keys into the backbone, while retaining the LM-head gate path and training the copy projections. The reduction in gains establishes the importance of this direct pathway in this setup, but is also compatible with learning representations specialized for position-level matching; it does not by itself identify reduced rank compression as the reason that pathway helps. Appendix H.2 changes the vocabulary rank, but simultaneously changes tokenization, the semantic extent of a 2,048-unit context, the amount of text processed, and the total parameter count: approximately 20B bytes, or 5B tokens of text, and 107M total parameters. The byte-level vanilla/CPLM comparison remains matched within that setting. However, the change in gains across token-level and byte-level settings does not isolate rank compression from the other changes. The statement that this mechanism accounts for most of the gains is therefore stronger than the interventions establish. A narrower conclusion would preserve the main empirical contribution. A more selective intervention or a matched vocabulary/gradient manipulation would be valuable future evidence.
-
The measured runtime overhead does not cover the configurations central to the efficiency claim. Appendix B explicitly benchmarks a full-attention backbone at a 4k training length and 4k/16k inference prompts. The largest memory advantages, however, concern SW/GDN backbones and retrieval with many documents. The global copy head adds a scan over all stored keys during a computed copy step and pairwise query/key scoring during training. Its relative cost need not stay negligible when the backbone itself has local or recurrent context processing. Cache-memory reductions are supported, but end-to-end latency and throughput in those configurations remain unmeasured in the PDF. The 9B/30B timing results use untrained architectures, and the largest runs exclude the optimizer step; the paper discloses this and should keep those measurements separate from demonstrated large-scale learning results.
-
Some downstream claims are harder to assess than the pretraining results. Table 2 does not label that MATH uses an 861-problem level-1/2 subset and pass@4, GSM8K uses pass@4, coding uses pass@10, and IFEval averages four accuracy criteria. These details are available in Appendix D, so the issue is presentation rather than an undisclosed protocol. The average combines different operating points, and the small full/hybrid gains lack corresponding paired uncertainty. The document-memory experiment uses 200 HotpotQA validation questions and a last-layer-only access control. It supports the value of copy-key memory in this controlled setting. Its broader applicability remains open because it does not evaluate a complete retrieval pipeline or isolate multi-hop reasoning from answer extraction; the paper does not explicitly claim equivalence to a general retrieval system. The paper already cautions that differences within about 3.5 percentage points may reflect sample noise, and Appendix F reports GSM8K improvements across pass@1–8. Existing per-question results, support-document information, and clearer metric labels would improve interpretability without requiring a large new benchmark.
Questions for authors
- Which existing result distinguishes mitigation of the LM-head rank bottleneck from learning representations specialized for a second output head? Would you narrow the causal wording if that distinction is not identified by the current experiments?
- Do existing timing logs include SW/GDN models, the chunked 1.7B training-loss implementation, or document-memory retrieval at high distractor counts? Please separate cache footprint, measured runtime, and asymptotic cost in the claims.
- Can the Table 2 headings state the exact subsets and pass@k metrics, and can existing predictions provide paired uncertainty for its small gains? For HotpotQA, beyond the stated two gold paragraphs and uniformly sampled 5–50 training distractors, what were the rules for selecting the 200 validation questions and constructing the shared distractor pool, and how often can the gold answer be copied from the supplied documents?
2 comments
How were the 200 validation samples chosen?
The 200 QA samples were chosen as the first 200 samples in the HotpotQA validation set. We do not use the full validation set to reduce evaluation cost, especially in setups with hundreds of distractors.
The evidence supports useful backbone–copy-head co-adaptation more directly than it establishes gradient-bottleneck relief as the dominant cause.
We address this in GR-2. In short, the effect of the LM head bottleneck cannot be strictly isolated in a completely unaltered pretraining setup. We therefore control for the factors that affect this bottleneck (vocabulary size, hidden dimension, LM head rank, data) and verify that the CPLM gains follow them. We clarified the description of the SpamLang experiment and added two new controlled setups (Appendices H.3 and H.4).
The measured runtime overhead does not cover the configurations central to the efficiency claim
Sliding-Window Decoding Speed
We report inference timings for sliding-window (SW-256) models. The table below reports throughput relative to the vanilla full-attention model on an RTX A6000 (batch size 16), together with FinePDFs perplexity:
CPLM SW reaches lower perplexity than the vanilla full-attention model while decoding 1.7–3.4x faster. Compared with vanilla SW, the CP head leaves prefill unchanged and costs 3–6% in decoding, since it attends over all previous tokens while the backbone only reads its 256-token window. This overhead grows slowly with context length, as it amounts to a single dot product per previous token. Note: the lower SW prefill throughput comes from the masked dense attention kernel used in this benchmark, which does not skip out-of-window blocks.
We expect GDN to perform roughly similarly, as the CP head time overhead is orthogonal to the specific backbone choice.
Document-memory retrieval at high distractor counts.
We use the HotpotQA inference setup with full-attention 1.7B models on a B200 GPU (batch size 32), adding real paragraphs from other HotpotQA questions as extra distractors. The table below reports decode time per token (ms), with speed relative to vanilla in parentheses:
With documents in context, decoding slows down linearly with the number of distractors. With document in copy memory only, decode time stays nearly constant, because each step reads a single copy key per document token instead of the full KV cache. The speedup over standard inference therefore grows with the number of distractors, reaching 9.4x at 50 paragraphs.
Some downstream claims are harder to assess than the pretraining results
We updated the Table and corresponding paragraph to clarify our methodology in the core text. Note that our results hold across different pass@k regimes. We restrict the MATH evaluation to easier levels to avoid near-random noise, as neither model handles more difficult questions consistently.
We extend our MATH evaluation to 32 samples per problem, and run a one-sided paired permutation test over problems to assess whether the gaps are meaningful. Table R1 reports the significance of the results in the paper, and Table R2 reports performance across all levels.
Table R1: MATH pass@4, Levels 1–2 (861 problems)
Table R2: MATH pass@4 by level
Table R1 confirms that all our results are significant (p<0.05).
Table R2 shows that scores beyond L2 are small (<10%), which is why we only considered easier levels in the paper. Our models are too small and/or insufficiently trained to perform strongly on more difficult math questions.