acceptodds
Under review as a conference paper at ICLR 2027

Copy-Paste Language Models

11 views

Abstract

Transformers are intuitively thought to be efficient at copying from their inputs. In this work, we show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), adding a lightweight Copy-Paste head similar to pointer networks on top of language models. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.

Primary area:
foundation or frontier models, including LLMs
Keywords:
pointer networks, language modeling, pretraining, in-context retrieval, softmax bottleneck, gradient bottleneck, efficiency

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_APSbOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper revisits an explicit copy distribution for decoder language models. A Copy-Paste head attends to previous token states and aggregates probability over positions containing the same vocabulary item. Its output is mixed with the LM distribution using a gate from a dedicated LM-head token and a sink-based fallback. The model is trained with ordinary next-token cross-entropy, without copy labels. Experiments cover pretraining from 135M to 1.7B parameters, full and local attention, hybrid attention, and smaller Gated DeltaNet models. CPLMs improve perplexity, yield modest downstream gains, and provide useful cache-memory tradeoffs for local backbones and document retrieval. The paper attributes much of the benefit to an additional gradient pathway around the vocabulary head, supported by gradient measurements, a synthetic task, a stop-gradient intervention, and byte-level training.

Critical strengths
  1. The empirical investigation is substantial and well controlled. Matched training recipes, several scales and backbones, a window-size sweep, and three seeds at 135M establish that the perplexity gains are not confined to a single architecture or a small random fluctuation (Sections 3.1, A, and C.1–C.5). Full-attention comparisons are particularly valuable because they distinguish the effect of the output mechanism from merely adding global access to a local model.

  2. The paper positions its contribution honestly. Section 2.2 acknowledges pointer-sentinel mixtures, and Appendix C.6 compares several pointer designs under a common setup. Table 8 shows that the proposed gate is only slightly better than, or tied with, conventional pointer alternatives. The authors explicitly frame the contribution as a modern pretraining study and analysis, rather than inventing copying. The comparison with a tuned inference-time neural cache is also useful.

  3. Several results have practical and scientific value. The 1.7B SW model's cache/perplexity tradeoff, the document-keys-only HotpotQA setup, and the reduction in induction heads provide different views of explicit memory. The paper reports failures as well as successes: full models still fail on sufficiently long retrieval inputs, and post-SFT LongBench gains are uneven. These qualifications make the results easier to interpret.

  4. The proposed explanation is tested, not merely asserted. The stop-gradient and byte-level studies in Appendix H are useful interventions. They provide stronger evidence about training dynamics than the gradient-projection plots alone, even though they do not fully identify the proposed mechanism.

Critical weaknesses
  1. The evidence supports useful backbone–copy-head co-adaptation more directly than it establishes gradient-bottleneck relief as the dominant cause. Appendix H.1 removes the direct gradients from the copy queries and keys into the backbone, while retaining the LM-head gate path and training the copy projections. The reduction in gains establishes the importance of this direct pathway in this setup, but is also compatible with learning representations specialized for position-level matching; it does not by itself identify reduced rank compression as the reason that pathway helps. Appendix H.2 changes the vocabulary rank, but simultaneously changes tokenization, the semantic extent of a 2,048-unit context, the amount of text processed, and the total parameter count: approximately 20B bytes, or 5B tokens of text, and 107M total parameters. The byte-level vanilla/CPLM comparison remains matched within that setting. However, the change in gains across token-level and byte-level settings does not isolate rank compression from the other changes. The statement that this mechanism accounts for most of the gains is therefore stronger than the interventions establish. A narrower conclusion would preserve the main empirical contribution. A more selective intervention or a matched vocabulary/gradient manipulation would be valuable future evidence.

  2. The measured runtime overhead does not cover the configurations central to the efficiency claim. Appendix B explicitly benchmarks a full-attention backbone at a 4k training length and 4k/16k inference prompts. The largest memory advantages, however, concern SW/GDN backbones and retrieval with many documents. The global copy head adds a scan over all stored keys during a computed copy step and pairwise query/key scoring during training. Its relative cost need not stay negligible when the backbone itself has local or recurrent context processing. Cache-memory reductions are supported, but end-to-end latency and throughput in those configurations remain unmeasured in the PDF. The 9B/30B timing results use untrained architectures, and the largest runs exclude the optimizer step; the paper discloses this and should keep those measurements separate from demonstrated large-scale learning results.

  3. Some downstream claims are harder to assess than the pretraining results. Table 2 does not label that MATH uses an 861-problem level-1/2 subset and pass@4, GSM8K uses pass@4, coding uses pass@10, and IFEval averages four accuracy criteria. These details are available in Appendix D, so the issue is presentation rather than an undisclosed protocol. The average combines different operating points, and the small full/hybrid gains lack corresponding paired uncertainty. The document-memory experiment uses 200 HotpotQA validation questions and a last-layer-only access control. It supports the value of copy-key memory in this controlled setting. Its broader applicability remains open because it does not evaluate a complete retrieval pipeline or isolate multi-hop reasoning from answer extraction; the paper does not explicitly claim equivalence to a general retrieval system. The paper already cautions that differences within about 3.5 percentage points may reflect sample noise, and Appendix F reports GSM8K improvements across pass@1–8. Existing per-question results, support-document information, and clearer metric labels would improve interpretability without requiring a large new benchmark.

Questions for authors
  1. Which existing result distinguishes mitigation of the LM-head rank bottleneck from learning representations specialized for a second output head? Would you narrow the causal wording if that distinction is not identified by the current experiments?
  2. Do existing timing logs include SW/GDN models, the chunked 1.7B training-loss implementation, or document-memory retrieval at high distractor counts? Please separate cache footprint, measured runtime, and asymptotic cost in the claims.
  3. Can the Table 2 headings state the exact subsets and pass@k metrics, and can existing predictions provide paired uncertainty for its small gains? For HotpotQA, beyond the stated two gold paragraphs and uniformly sampled 5–50 training distractors, what were the rules for selecting the 200 validation questions and constructing the shared distractor pool, and how often can the gold answer be copied from the supplied documents?
2 comments

The evidence supports useful backbone–copy-head co-adaptation more directly than it establishes gradient-bottleneck relief as the dominant cause.

We address this in GR-2. In short, the effect of the LM head bottleneck cannot be strictly isolated in a completely unaltered pretraining setup. We therefore control for the factors that affect this bottleneck (vocabulary size, hidden dimension, LM head rank, data) and verify that the CPLM gains follow them. We clarified the description of the SpamLang experiment and added two new controlled setups (Appendices H.3 and H.4).

The measured runtime overhead does not cover the configurations central to the efficiency claim

Sliding-Window Decoding Speed

We report inference timings for sliding-window (SW-256) models. The table below reports throughput relative to the vanilla full-attention model on an RTX A6000 (batch size 16), together with FinePDFs perplexity:

ModelPrefill (1K)Prefill (3.8K)Decode (1K)Decode (3.8K)PPL (1K)PPL (3.8K)
Vanilla fullx1.00x1.00x1.00x1.009.428.76
Vanilla SWx0.94x0.77x1.72x3.5910.029.69
CPLM SWx0.94x0.77x1.67x3.389.268.26

CPLM SW reaches lower perplexity than the vanilla full-attention model while decoding 1.7–3.4x faster. Compared with vanilla SW, the CP head leaves prefill unchanged and costs 3–6% in decoding, since it attends over all previous tokens while the backbone only reads its 256-token window. This overhead grows slowly with context length, as it amounts to a single dot product per previous token. Note: the lower SW prefill throughput comes from the masked dense attention kernel used in this benchmark, which does not skip out-of-window blocks.

We expect GDN to perform roughly similarly, as the CP head time overhead is orthogonal to the specific backbone choice.

Document-memory retrieval at high distractor counts.

We use the HotpotQA inference setup with full-attention 1.7B models on a B200 GPU (batch size 32), adding real paragraphs from other HotpotQA questions as extra distractors. The table below reports decode time per token (ms), with speed relative to vanilla in parentheses:

Paragraphs (avg. context)Vanilla, docs in contextCPLM, docs in contextCPLM, document in copy only
10 (1.6K tokens)10.610.8 (x0.98)2.4 (x4.37)
20 (3.1K tokens)14.414.8 (x0.98)2.5 (x5.84)
50 (7.4K tokens)26.226.8 (x0.98)2.8 (x9.39)

With documents in context, decoding slows down linearly with the number of distractors. With document in copy memory only, decode time stays nearly constant, because each step reads a single copy key per document token instead of the full KV cache. The speedup over standard inference therefore grows with the number of distractors, reaching 9.4x at 50 paragraphs.

Some downstream claims are harder to assess than the pretraining results

We updated the Table and corresponding paragraph to clarify our methodology in the core text. Note that our results hold across different pass@k regimes. We restrict the MATH evaluation to easier levels to avoid near-random noise, as neither model handles more difficult questions consistently.

We extend our MATH evaluation to 32 samples per problem, and run a one-sided paired permutation test over problems to assess whether the gaps are meaningful. Table R1 reports the significance of the results in the paper, and Table R2 reports performance across all levels.

Table R1: MATH pass@4, Levels 1–2 (861 problems)

BackboneVanillaCPLMΔp (superiority)
Full14.716.8+2.20.0002
Hybrid16.117.1+1.00.046
SW-2566.516.7+10.1<1e-4

Table R2: MATH pass@4 by level

BackboneLevelnVanillaCPLMΔp (superiority)
FullL129019.722.1+2.50.012
L257112.114.2+2.00.003
L37078.910.3+1.40.005
L47275.66.2+0.50.11
L57773.43.8+0.40.11
HybridL129021.122.0+1.00.22
L257113.514.6+1.10.060
L37078.59.0+0.50.17
L47275.84.9−0.90.98
L57773.22.8−0.30.85
SW-256L12908.622.8+14.2<1e-4
L25715.513.6+8.1<1e-4
L37073.78.2+4.5<1e-4
L47272.56.4+3.8<1e-4
L57771.73.7+2.0<1e-4

Table R1 confirms that all our results are significant (p<0.05).

Table R2 shows that scores beyond L2 are small (<10%), which is why we only considered easier levels in the paper. Our models are too small and/or insufficiently trained to perform strongly on more difficult math questions.

How were the 200 validation samples chosen?

The 200 QA samples were chosen as the first 200 samples in the HotpotQA validation set. We do not use the full validation set to reduce evaluation cost, especially in setups with hundreds of distractors.

Reviewer_bb1YOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Copy-Paste Language models (CPLMs). The key point of the paper is that current language models are not as good at copying information from their inputs as the can be and this is due to the low rank nature of the LM head. CPLMs use a lightweight copy paste head similar to pointer networks to improve the models copying ability. The approach is evaluated on training efficiency (pretrain on fineweb frmo 135M to 1.7B params and up to 200B tokens), Donwstream evals (Hellaswag, LAMBADA, PIQA, etc), reasoning tasks, long context and retrieval abilities. CPLM shows consistent improvements across model scales, backbones and different tasks.

Critical strengths

Thorough Results and Experiments: The breadth of experiments in the paper is really great! From language model pretraining to downstream evals. The authors evaluate and different parameter scales and scale the token budget up to 1.7B, makes it easier to trust that the results are not due to some specific dataset or condition. There are also eval results on long context and reasoning tasks as well (math, code instruction). Even on these tasks, e.g lm pretraining, the paper uses different backbones e.g SWA, full attention, hybrid. This is good to know that this is not constrained to a specific setting. A lot of the studies in this paper are controlled.

Thorough Analysis: I really like how the analysis part of the paper is setup. The paper goes into great depth on which components matter, when copying might help and why the approach seems to work. All very good and insightful analysis.

Critical weaknesses
  • The mechamism of operation seems so much similar to pointer networks (This work contributes a lot more than that but just to mention the architecture design might not be necessarily new).
  • On the zero shot downstreams, the improvements are quite modest on all the tasks with the exception of LAMBADA which then bumps up the average, so it is still a bit unclear how this helps. Thanks for highlighting the lambda gains.
Questions for authors
  • in Figure 2, there is a blue dot meant for full attention, but I can't see the results for those on the plot. I know you meniton they are superposed but would be nice to maybe visualize it a bit so that can see immediately.

  • For the downstreams, what is the uncertainty for the measurements. i.e are your results there just within noise error ?

  • What happens if you have tied embeddings ? does this still work ?

  • In table 6, are the inference numbers computed with the two stage ?

2 comments

We are grateful to the reviewer for their thoughtful comments and for their nice suggestions.

The mechamism of operation seems so much similar to pointer networks

We thank the reviewer for acknowledging our novelty, and we agree that the CP head design itself is not a strong novelty. We clarify our positioning in GR-1.

Consistency of the zero-shot downstream results

Some zero-shot benchmarks are noisy at small pretraining scales, where models are poorly calibrated for question answering. We add MMLU-RC, a cloze-format rewriting of MMLU that removes the difficulty small models have with letter-based options, and we report 95% confidence intervals for the CPLM − Vanilla gap below. The weighted-average gap is significant for all three backbones, and the unweighted average gap is significant for Full and SW-256. Overall, CPLMs outperform their vanilla counterparts on 18/24 (75%) backbone/task pairs, with the largest gains on LAMBADA.

We hypothesize that benchmarks such as PIQA or ARC do not discriminate between Vanilla LMs and CPLMs because they mostly test knowledge memorization, which CPLMs improve less than in-context retrieval, chain-of-thought reasoning, or ambiguous token prediction (as in LAMBADA).

TasknWeightFullHybrid 3:1SW-256
HellaSwag1004227.2%+0.6 [+0.0, +1.2]*+0.6 [+0.1, +1.2]*+0.8 [+0.2, +1.4]*
LAMBADA515314.0%+2.5 [+1.3, +3.6]*+3.2 [+2.1, +4.3]*+4.0 [+2.8, +5.1]*
PIQA18385.0%−0.4 [−1.7, +0.9]+0.0 [−1.4, +1.4]+0.2 [−1.3, +1.6]
SciQ10002.7%+1.0 [−0.5, +2.5]−0.4 [−1.8, +0.9]+0.3 [−1.3, +1.8]
ARC-e23766.4%−0.3 [−1.7, +1.2]−2.9 [−4.3, −1.4]*+0.4 [−1.1, +1.9]
ARC-c11723.2%+0.3 [−1.9, +2.5]+0.6 [−1.5, +2.7]−0.2 [−2.4, +2.0]
WinoGrande12673.4%+0.6 [−2.1, +3.4]+1.5 [−1.6, +4.3]+0.6 [−2.5, +3.6]
MMLU-RC1404238.1%+0.4 [−0.1, +1.0]+0.1 [−0.4, +0.7]+0.8 [+0.2, +1.3]*
Avg36890+0.6 [+0.0, +1.2]*+0.4 [−0.2, +0.9]+0.9 [+0.3, +1.4]*
Weighted avg36890+0.7 [+0.3, +1.0]*+0.6 [+0.2, +0.9]*+1.1 [+0.8, +1.5]*

Each cell gives the CPLM − Vanilla accuracy gap (in points) with its 95% confidence interval; * marks intervals that exclude zero. The weighted average weights each task by its number of examples.

Improving the visualization of Figure 2

Thank you for this suggestion. We adjusted the line thickness in the revised version.

What happens if you have tied embeddings ? does this still work ?

We use tied embeddings in all our experiments, and the gradient bottleneck reported in [1] applies to both tied and untied setups. To study the impact of embedding tying, we train 135M models with 256-token sliding-window attention on 10B FineWeb-Edu tokens, with input/output embeddings either shared (tied) or separate (untied). We report PPL on held-out FineWeb-Edu (in-domain) and FinePDFs documents (out-of-domain).

VariantEmbeddingsParamsFineWeb-Edu PPLFinePDFs PPL
Vanillatied134.5M18.1918.17
CPLMtied135.2M17.1914.84
Vanillauntied162.8M18.2118.32
CPLMuntied163.5M17.1514.85

The gaps between CPLM and Vanilla remain stable across tied and untied setups.

In table 6, are the inference numbers computed with the two stage ?

No. Table 6 reports standard inference measurements, where logits are materialized to allow for any sampling mechanism. The two-stage scheme is a side remark, which we clarified in the revised version.

[1] Lost in Backpropagation: The LM Head is a Gradient Bottleneck, Godey et al., 2026

Very great work! All my questions are answered. Congrats to the authors on a valuable contribution

Reviewer_ZN7DOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds a pointer-style copy head alongside the standard vocabulary head, trained jointly with the next-token objective. Experiments cover pretraining up to 1.7B parameters, downstream evaluation, instruction tuning, and long-context retrieval. The authors report faster convergence and improved performance, and attribute much of the benefit to an additional gradient pathway around the LM head.

Critical strengths
  1. The pretraining study covers several model sizes and backbones. Improvements on full-attention models are particularly useful evidence, since they cannot be explained simply by adding global access to a local backbone.

  2. The sliding-window and document-memory results provide a practical reason to revisit copy mechanisms. Retaining access to distant tokens through a small copy cache is useful, and the reported overhead is low.

Critical weaknesses
  1. The methodological novelty is limited. The formulation closely follows existing pointer mixtures, and the shared-softmax baseline in Table 8 performs almost identically. These comparisons are restricted to 135M models, leaving unclear whether the proposed design offers a meaningful advantage over established copy mechanisms at the main experimental scale.

  2. The gradient-bottleneck explanation is not sufficiently isolated. Stopping the copy gradient also prevents the backbone from learning representations suited to copying. Switching to bytes changes the prediction task and effective context alongside the output rank. These experiments support the importance of jointly training the copy branch, but do not establish reduced gradient compression as the main cause. Given the limited architectural novelty, this uncertainty substantially weakens the paper’s contribution.

Questions for authors

Please refer to Weaknesses.

2 comments

We are grateful for the positive comments and for the valuable feedback.

The methodological novelty is limited.

We address this in depth in GR-1. In short, our main novelty is the use of a pointer mixture for LM pretraining, not the invention of pointer mixtures themselves. In Table 8, we compare pointer variants developed in other contexts (e.g. summarization or single-turn dialogue). These baselines are not reproducing results from the corresponding papers; they are adapting the pointer mechanisms developed at the time in a decoder-only pretraining setup. We agree that our CP head design is not deeply innovative in the context of pointer networks, but it marginally outperforms these variants, which justifies our choice.

The gradient-bottleneck explanation is not sufficiently isolated.

We also reply to this comment more extensively in the General Response. Because controlling for the gradient bottleneck implies modifying the setup to some extent (e.g. by reducing the vocabulary size or increasing the hidden dimension), it appears impossible to design an experiment that perfectly isolates the gradient bottleneck effect. We instead design specific experiments where different components are modified to control for the LM head bottleneck, and we verify how the training dynamics of CPLMs reflect better optimization in these setups. To address this concern, we propose 2 additional controlled experiments: one where we control for vocabulary size and show that the CPLM advantage appears only when it crosses roughly 2−6×2-6\times the hidden dimension; and another where we control for the LM head rank independently of the backbone and show that CPLMs provide better training efficiency when the rank is smaller.

We hope to have addressed all the reviewer's concerns, and we are happy to answer any additional questions within the discussion period.

Thanks for the clarification. That addressed some of my concerns, and I’ll raise my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their positive comments, and for acknowledging the breadth of our experiments, the practical relevance of our method, and the success of CPLMs across architectures and sizes. In the following, we address general concerns.


GR-1: Lack of novelty compared to pointer networks (ZN7D, bb1Y)

Our main novelty does not lie in the specific design of the pointer network, but in the use of pointer networks to pretrain autoregressive LMs from scratch. We selected our copy/sink design among other pointer candidates through smaller-scale ablations, where it outperforms existing designs in at least one setup (Table 8). In seminal pointer network works [1,2], and in CopyNet, which introduced the shared-softmax design [3], pointers were used on top of RNNs that did not have an attention mechanism internally, for tasks such as summarization. Chang et al. [4] study pointer networks in a closer autoregressive setup, but mostly as a post-hoc addition to existing models, and with a less effective pointer mechanism (Table 8).

To the best of our knowledge, we are the first to use pointer networks in a modern LM pretraining setup and to report noticeable gains. Our pointer design is new and marginally better than past designs in our setup, but it is not where our main novelty lies.


GR-2: Insufficient isolation of the gradient bottleneck effect (ZN7D, APSb)

Figure 6 (right) shows that the CP head directly addresses the LM head bottleneck: it retains up to 85% of the logit gradient norm, against <1% for the LM head. We improved the clarity of Section 4.1 to make this analysis easier to read. The open question is therefore whether this mitigation causally improves performance. It is not clear how to design a single experiment that perfectly isolates the reduced gradient bottleneck causal effect on performance. We instead run complementary experiments with distinct controlled interventions, which all show that CPLMs help most where the LM head bottleneck is strongest:

  • SpamLang (Section 4.1): We control the data and the vocabulary size so that the vocabulary size only changes how hard the task is to optimize, not how hard it is to model. Vanilla LMs fail to learn at large vocabulary sizes, while all CPLMs reach perfect accuracy.
  • Byte-level modeling: When we reduce the vocabulary to bytes on natural-language data, the gap between CPLMs and vanilla models disappears. The different effective byte count alone cannot explain this, since the NLL gaps are stable throughout training (Figure 2).
  • Stop-gradient (Appendix H): When we block the gradient from the CP head into the backbone, 75% of the CPLM gain disappears. This is close to a causal intervention, although it also prevents the backbone representations from adapting to the CP mechanism.
  • Non-copied tokens (Figure 14): CPLMs achieve lower loss even on tokens where the copy path is not used, which shows that backpropagation through the CP head also improves the backbone for the LM head pathway.

Furthermore, we add two new experiments to consolidate these findings:

  • Vocabulary size sweep (new Appendix H.3): We trim BPE merges from the SmolLM2 tokenizer to obtain intermediate vocabulary sizes, and train Vanilla and CPLM 135M models. The BPB gap stays at noise level when the vocabulary size VV is smaller than or comparable to the hidden dimension DD, and becomes a clear CPLM gain for V∈[2D,6D]V \in [2D, 6D] (Figure 19). The CPLM improvement thus tracks the V/DV/D ratio, as predicted by the gradient bottleneck interpretation.
  • Factorized LM head (new Appendix H.4): We factorize the LM head as a matrix product with inner dimension dd, which caps its rank independently of the backbone and is known to slow down vanilla pretraining [5]. CPLMs are much less affected by this constraint: their token-efficiency advantage over vanilla LMs grows from ×1.4 to ×2.2 as the rank shrinks.

[1] Pointer Networks, Vinyals et al., 2015

[2] Pointer Sentinel Mixture Models, Merity et al., 2016

[3] Incorporating Copying Mechanism in Sequence-to-Sequence Learning, Gu et al., 2016

[4] Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyond, Chang et al., 2023

[5] Lost in Backpropagation: The LM Head is a Gradient Bottleneck, Godey et al., 2026

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.