acceptodds
Under review as a conference paper at ICLR 2027

Learning to Copy, Learning to Depend: Language Models Offload In-Context Copying to a Pointer Head

43 views

Abstract

Many next words are copies: a character's name, a number, a term defined a paragraph earlier. We train language models jointly with a pointer-copy head that copies such tokens from context, then remove it at inference: accuracy on LAMBADA, where most targets appear earlier in the passage, collapses below 7%, far below the 18–23% of baselines trained without it, while eight other benchmarks barely move. We find this output-pathway offloading, in which the backbone's vocabulary readout delegates in-context copying to the head rather than duplicating it, in all five architecture families: two Transformers, Mamba-1, a Jamba-style hybrid, and Gated DeltaNet, each trained from scratch as a controlled baseline/+copy pair at 130M–198M, with further pairs from 45M to 350M. In both Transformers, a linear probe decodes copy targets from the +copy backbone's final hidden state as well as from the baseline's, while its native vocabulary readout becomes less accurate on them. With the head in place, WikiText-103 perplexity and LAMBADA accuracy improve in every pair, and held-out perplexity in all five 5B-token pairs. Copyability stratification confines every pair's LAMBADA gain to targets present in the preceding context, and randomizing the retrieved identities drops every model below its baseline. Under one fixed head configuration, Mamba-3 SISO - a backbone redesigned for retrieval - still benefits on every headline metric. A three-seed replication at 60M reproduces every headline gain's sign and the closed-gate collapse. Because the head configuration differs between architecture classes, we do not rank architectures; all 130M–350M results are single-seed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
copy mechanism, pointer network, state space model, Mamba, language model, retrieval, cross-architecture, SSM, efficient architecture, pointer-copy, gate warmup, bfloat16

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_LXVyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a modular attention-based component that supports copying by calculating the output token prediction by aggregating attention weights over prior occurrences of tokens, then interpolating with the distribution determined by the main model’s backbone with a learned gate. The authors demonstrate that models with this learnable modular attachment improve on overall perplexity and accuracy on WikiText-103 and LAMBADA over baselines; however, the gains are concentrated to when the target token has previously occurred in the context window. Experiments primarily between the 125M and 350M scale, trained from 2B to 5B tokens, across various model families show that this “copying head” improves performance on measured downstream tasks within these experimental configurations but can degrade tasks such as MQAR.

Critical strengths

The paper demonstrates that the copying head improves performance on the small scale under their experimental setup. The authors test variations of the copy head, explore the limitations of the methodology at two scales, sweep cap c values, and replicate findings with multiple seeds.

Critical weaknesses

The experimental setup raises concerns about whether the improvements observed in this paper would persist under stronger training configurations. For instance, the Mamba series’ improved training recipe trains the 125M and 350M models with a peak lr of 3e-3 and 1.5e-3 respectively. Although the baseline and copy-head models utilize the same lr, it is unclear whether the gains would hold under better hyperparameters. In addition, upon looking at some of the scripts in the supplemental material, it seems like there are potential issues with weight initialization where the layers are initialized with the same parameters (see GPT2TF initialization in train_gpt2_tf_130m_copy.py). Furthermore, the context length of 1K is smaller than the more standard 2K/4K context lengths in architecture research literature. It remains to be seen whether the improvements hold at longer training context, as it may be possible that a shorter context length may bias certain patterns in the data.

While the authors provide several ablations and comparisons, I believe that there is a lack of important experiments which isolate the functionality of the copy head and establish what the copy head contributes beyond a context-dependent repetition prior. For instance, there should be additional ablations that try to determine out whether the “copy head” is just upweighting previously seen tokens: a simple baseline would be to replace the attention weights with a uniform weight based on the distribution of previous tokens. Furthermore, a standard baseline which replaces the copy head with a SWA head (or a slightly adjusted variant with an identity value projection) would additionally help determine whenever the copy-head provides benefits beyond an additional attention-based retrieval mechanism.

Questions for authors

See weaknesses. How were the experimental hyperparameters selected, and do the gains persist with stronger baselines? Furthermore, do the gains hold across scale? Further fleshing out the role the copy-head plans would be helpful.

For Appendix M, is there a reference to what these tasks are based on or where they specifically measure? Based on my understanding, these types of tasks are usually toy-tasks that require a model to be trained from scratch instead of testing an already trained model on them (see Mechanistic Design and Scaling of Hybrid Architectures).

6 comments

(Part 1/2)

Thank you for the review. Numbering follows the revised PDF.

Learning rate. The peak learning rate is the same for every family at 130M-198M and was not tuned per family; higher peak rates are untested (Sections 4 and 7).

How the hyperparameters were selected. The paper reports the values used and describes no search, and we do not present the recipe as tuned. At 130M-198M every family uses peak learning rate 6 x 10^-4, not tuned per family (Section 4, Appendix A); weight decay, warmup, learning-rate floor and micro-batching are set per architecture (Table 5). The 350M runs use an effective batch of 32 sequences and peak 3 x 10^-4 (Appendix A); the 800M-class runs use peak 2.5 x 10^-4 (Appendix AB). The paper gives no selection procedure for either. The unified recipe was fixed before any run (Appendix AA). All these rates are below the ones you cite; higher peak rates are untested (Section 7), so whether the gains hold under a stronger recipe is open.

Weight initialization. You were right. The GPT-2-style stack is built with nn.TransformerEncoder, which clones one layer, so all layers start with identical weights in both arms at 60M, 130M and 350M; those runs also keep N(0,1) embeddings. This was not disclosed before; it is now in the caption of Table 6, Section 4 and Section 7. The other four families build layers independently and all show the collapse. A pre-registered control (GPT-2-style 60M, 1B tokens, three seeds; layers redrawn independently, every other draw kept; published checkpoints rescored in the same environment; baselines head-free on both sides, unlike Table 17) moved none of the three pre-registered quantities (closed-gate LAMBADA accuracy, LAMBADA and WikiText-103 copy gains) past its threshold. The closed-gate threshold (about 15 items, one LAMBADA pass's binomial resolution) was fixed before any run; under our first rule, three literal seed s.d., the closed-gate move (26 -> 33 of 5,153) would have been flagged, so we report it. At either value the closed gate stays about 10 points below the head-free baseline (11.0%). Two cells outside the decision moved by several published seed s.d.; we attribute no cause. This excludes large moves, not small ones, and was not run at 130M or 350M, at other token budgets, or with N(0,0.02) embeddings. The new runs used a different GPU type and possibly a different PyTorch build, which three seeds cannot separate from the initialization. We therefore keep the caveat (Table 6, Section 7) that GPT-2-style magnitudes are not comparable across families; files: supplement code/init60m/ and data/init60m/.

Context length. Training context is 1,024 tokens throughout; at inference, for one Mamba-1 pair, the head holds at 2,048 tokens and reverses on WikiText-103 at 4,096 (Appendix AB), and training at longer context has not been run.

Controls not run. We have no trained uniform-attention head and no sliding-window attention head. The causal and binding shuffles (Table 4) and context partition (Table 40) are inference-time and set-based controls; they do not replace the training-time experiment you ask for. Truncating the Variant B window at inference removes 5-10% of the measured WikiText-103 and holdout gains and none of the LAMBADA gain (Appendix Q), but shorter-window training was not isolated.

Seeds and scale. Three seeds at 130M/2B are in Table 44: removal costs 19.8 +/- 0.7 (GPT-2-style) and 21.1 +/- 0.6 LAMBADA points (Mamba-1) against gains of 5.9 +/- 1.4 and 6.1 +/- 0.7, so removal exceeds the gain 2.9- to 4.3-fold in every seed (Section 5.4). Table 45 adds 10B-token rows at 130M and about 800M parameters, one seed each: the closed/baseline LAMBADA ratio is unchanged from 5B to 10B at 130M and higher at 800M; the holdout gain shrinks, and the other gains show no consistent trend.

(Part 2/2)

The copy head's role, as far as the paper establishes it. (1) Reliance is not benefit: removal costs 20.9-25.6 LAMBADA points in the five 5B pairs; 2.7-7.9 are gain over a head-free baseline, 13.9-18.2 a backbone deficit (Section 5.4). (2) The gain sits on copyable positions: +4 to +11 LAMBADA points where the target occurs in the prior 512 tokens; elsewhere accuracy is 1-4% with or without the head (Section 5.3, Table 19; the split is approximate). (3) It needs position-specific identities: permuting identities within each query's exact candidate set, at inference, drops each of the five main 5B models below its baseline (Table 4). (4) Reliance persists when the head's gradients are kept out of the backbone: with the backbone detached, closed-gate LAMBADA is 0.7% and 1.5% against baselines of 10.7% and 13.3% (60M, three seeds; GPT-2-style, initialization caveat of Section 7, and Mamba-1; Table 41). (5) Beyond the head's window (1.8% of WikiText-103 tokens) four 130M closed-gate backbones are 0.10-0.23 nats per token worse than their baselines; stratum chosen post hoc, one seed each, cause not identified (Appendix AE).

Appendix M. Appendix M now defines each task and states that these are zero-shot probes we designed, not train-from-scratch tasks.

Thank you for your response and additional experiments.

Unfortunately, this response raises additional concerns regarding the experimental methodology, reproducibility, and presentation.

The new runs used a different GPU type and possibly a different PyTorch build, which three seeds cannot separate from the initialization.

The authors should confirm whether there were any changes to the GPU type and PyTorch build. Combined with the weight initialization issue from before, this uncertainty about experimental setup is frankly concerning. I would expect authors to be able to verify their own experimental setup.

In addition, my original concerns remain unanswered, particularly those that ask for a simple context-dependent baseline.

Finally, the new rewrite has further decreased readability. For instance, this is the first sentence of the abstract:

Inference-time removal measures reliance within a jointly trained system and can overstate or understate what including a component during training does.

It has come to my attention that another reviewer (atSF) felt like the submission was heavily AI generated. Pangram also assigns the joint response, my specific response, and abstract as 100% AI generated.

Notwithstanding the empirical findings and soundness, the current submission's presentation needs to be rethought.

Reviewer_rqDYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the impact of jointly training language models with an output-side pointer-copy head, focusing on both predictive performance and how copying behavior is distributed between the backbone's vocabulary output and the additional copy mechanism. The authors conduct experiments across five architecture families, comparing baseline models with copy augmented variants at the 130M--198M parameter scale. The main finding is a phenomenon referred to as output-pathway offloading: while adding the copy head improves performance, disabling it after joint training leads to a substantial degradation on copying-related tasks, even compared to models trained without the head. This suggests that the models learn to rely on the copy mechanism rather than independently maintaining the same behavior through their vocabulary readout. The paper's main contribution is an empirical investigation of this phenomenon rather than a novel copying architecture. The authors support their findings through experiments on copyable and non-copyable tokens, source identity corruption, hidden-state probing, multi-seed replication at smaller scales, and additional experiments with larger models. The results provide evidence of specialization between the two output pathways, although differences in copy-head configurations across architecture families limit direct cross-architecture comparisons.

Critical strengths
  1. A substantial and replicated distinction between performance benefit and learned dependence.

Tables 1 and 2 (pages 7–8) show that adding the copy head improves LAMBADA accuracy by 2.6–7.9 percentage points across the five architectures. However, when the copy head is disabled at inference after joint training, accuracy drops to just 3.6–6.5%, substantially below the 18–23% achieved by models trained without it. This result highlights an important distinction: removing a component after training does not necessarily reveal how much that component contributes to performance, but rather how much the model has learned to depend on it. Furthermore, the same degradation is observed across all 14 available checkpoints in the 60M multi-seed experiments, providing additional evidence that this behavior is consistent across different training runs.

  1. The performance benefit extends beyond the most copy-favorable benchmark.

Beyond LAMBADA, the copy head consistently improves perplexity on the temporally separated FineWeb-Edu holdout, with reductions ranging from 3.3–9.9% across the five model pairs. The improvements on WikiText-103 also remain after accounting for the identified tokenization artifact, suggesting that the benefits are not limited to a single benchmark. Another strength is the copyability analysis, which reveals an important trade-off: while the copy head improves predictions for tokens that appear in the context, it leads to higher LAMBADA perplexity on non-copyable targets. This analysis provides a more complete understanding of when the copy mechanism is beneficial and where it introduces limitations, rather than relying only on overall performance improvements.

Critical weaknesses
  1. Some quantitative comparisons remain affected by training or evaluation inconsistencies.

Although the authors have corrected the evaluation issues affecting the main 5B benchmark results, some inconsistencies remain. In particular, some of the 2B baselines were evaluated using an earlier version of the evaluation backend, in which an untrained copy gate was still active. This makes the corresponding comparisons less reliable. Moreover, the GPT-2-style baseline and its Variant B copy-augmented counterpart use different learning-rate warmup durations (500 versus 125 optimizer steps), introducing an additional difference that makes it difficult to attribute the observed improvements solely to the copy head. While the Variant A retraining helps address this concern, it does not fully clarify the contribution of the original Variant B configuration. The authors also report a discrepancy in the Llama-style WikiText baseline results between the main and stratified evaluation procedures, which remains unexplained.

Although these inconsistencies are unlikely to account for the substantial performance degradation observed when disabling the copy head, they weaken the reliability of some quantitative comparisons. Since the paper's main contribution relies on controlled empirical experiments, ensuring consistent training and evaluation settings is particularly important. Resolving these discrepancies and re-evaluating the affected models would strengthen the validity of the reported results and the conclusions drawn from them.

  1. The explanatory contribution remains narrower than the strength of the behavioral evidence.

As the authors themselves point out, some degree of specialization between the vocabulary and copy pathways is expected from the training objective. When the copy head successfully predicts a target token, the vocabulary pathway receives a weaker training signal for that prediction. Therefore, the main contribution is not the observation that specialization occurs, but rather the analysis of how strongly it affects the model and its performance. While the hidden-state probing experiments provide useful insights, they do not fully demonstrate that copying-related information is preserved in the backbone's representations. In particular, the probe achieves approximately 40% accuracy, compared to around 50% for the baseline model's original vocabulary readout. As a result, similar probing performance between the baseline and copy-augmented models does not necessarily rule out a loss of information that the original readout could exploit. The authors also acknowledge this limitation.

Overall, the experiments provide convincing evidence that joint training causes the model to rely more heavily on the copy head for predicting tokens that appear in the context. However, it remains unclear whether the observed degradation in the vocabulary pathway is entirely due to changes in the output readout or whether the underlying hidden representations are also affected. A more detailed investigation of this distinction would strengthen the explanation of the observed behavior and provide a deeper understanding of the mechanisms behind output-pathway offloading.

Questions for authors
  1. Consistency of experimental comparisons.

Could the 2B results, particularly the Mamba-1/Mamba-3 comparison, be re-evaluated using the corrected evaluation backend and consistent inference-time settings? Additionally, could the authors clarify the discrepancy between the main and stratified WikiText-103 evaluation results? It would be useful to understand whether resolving these inconsistencies would affect any of the reported improvements or conclusions.

  1. Further investigation of output-pathway offloading.

The hidden-state probing results suggest that copying-related information may still be present in the backbone representations, even though the original vocabulary readout performs worse. Would it be possible to investigate this further by freezing the jointly trained backbone and training a new vocabulary readout? In particular, could such a readout recover the copying performance of the baseline model? This additional experiment would help clarify whether the observed degradation is mainly due to specialization of the output layer or whether joint training also changes the information encoded in the hidden representations.

  1. Understanding the performance degradation on MQAR-update.

The GDN model shows a substantial decrease in MQAR-update accuracy when the copy head is introduced, dropping from 82.5% to 28.0%. Could the authors investigate whether disabling the copy head at inference restores the original performance? It would also be interesting to examine how the vocabulary and copy pathways individually score the corrected and outdated values. Such an analysis could help determine whether the degradation is caused primarily by the copy mechanism favoring outdated information or by changes in the vocabulary pathway during joint training.

3 comments

(Part 1/3)

Thank you for the review. Numbering follows the revised PDF.

Weakness 1: 2B baselines scored with an untrained gate. You were right. Every 2B cell is now rescored in one environment, baselines head-free (Table 7, Appendix P.1). The GDN 2B WikiText-103 reduction is -12.7%, not the -18.3% we reported; the Llama-style baseline and two copy WikiText-103 cells also change. The 60M tables keep their published baselines; Appendix P.1 bounds the untrained head's effect on them, and every 60M baseline/+copy and closed-gate comparison keeps its sign, the four for the one +copy checkpoint not retained against its published GPU values.

Weakness 1: learning-rate warmup in the GPT-2-style pair. We now call the pairs runs on one corpus and nominal recipe, not controlled pairs. The main GPT-2-style Variant B pair differs in learning-rate warmup (caption of Table 1), and in every pair the head's initialization advances the random stream, so the arms draw different training windows (Appendix A). Exact attribution to the head in that pair needs a matched-warmup rerun, which we have not run; the matched-warmup Variant A pair of Table 43 gives the same signs.

Weakness 1 and Q1: Llama-style WikiText-103 discrepancy, 56.45 vs. 55.68. The submitted version left this untraced. It was neither the scorer nor bfloat16 attention: both numbers came from the freshly initialized, unseeded copy head the backend attached to baselines. Head-free, the baseline agrees across bfloat16, fp32 and two machines, and without the \n\n tokens of Table 21 it scores the same with and without the head, so the effect sits on those tokens. The corrected reduction is -24.4% (submitted -21.7%); see Appendix P, item (8), Appendix P.1 and Appendix Z.

Q1: Mamba-1 vs. Mamba-3 at 2B. Table 13 is rescored in one environment, baselines head-free; the Mamba-1 2B baseline reproduces. One asymmetry remains: the Mamba-1 pair was scored uncapped and Mamba-3 at five caps. At c=0.5 for both, the unified-recipe rows give Mamba-1 at 2B over three seeds (Table 44) and Mamba-3 SISO at 2B, one seed (Table 43). These are different backbones under one head configuration: their widths differ, so the head's projections differ too.

Q1: do the corrections change any conclusion? No copy-vs-baseline comparison changes sign; two statements weaken, some wording changes, and one claim is no longer made. Unchanged: with baselines rescored head-free, no copy-vs-baseline difference changes sign in any remeasured table (expectation E4, written down before rescoring; Appendix P.1); every Table 1 WikiText-103 reduction is unchanged or larger (14.6-24.4%, previously 14.6-21.7%) and the LAMBADA gains move by at most 0.1 points; closed-gate LAMBADA accuracy and WikiText-103 perplexity stay worse than the rescored baselines (Table 3). Weakened: the 2B GDN WikiText-103 reduction is -12.7%, not -18.3% (Table 28), and Mamba-1's WikiText-103 gain at the training cap no longer grows from 130M to 350M (Appendix P.1, Section 5.5). Wording changed: the pairs are "runs on one corpus and nominal recipe", not controlled pairs (Table 1); gains are "concentrated in" copyable targets, not confined to them (Section 5.3). No longer claimed, for a reason separate from the rescoring: that copy targets stay about as decodable from the hidden states (Appendix V, Table 36; see the hidden-state paragraph below).

One head on all backbones. Appendix AA trains one Variant A head configuration on six backbones under one recipe (Table 43). Every sign holds in every row, with closed-gate LAMBADA accuracy below a third of baseline; rows are single-seed (five backbones at 5B, Mamba-3 SISO at 2B).

(Part 2/3)

Weakness 2: the explanatory contribution. You noted that some specialization is expected from the training objective. The objection is fair against the submitted framing, and we no longer present the existence of offloading as the finding. The revision leads with this: inference-time removal measures reliance within a jointly trained system and can overstate or understate what including a component during training does. On LAMBADA it overstates the benefit: for Mamba-1 at 5B tokens, removing the head costs 20.9 points, while the full model is 2.7 points above a baseline trained without a head; the other 18.2 points are a backbone deficit (Section 5.4; single seed at 5B, three seeds at 130M/2B in Table 44). On WikiText-103 non-copyable tokens removal instead overstates the head's cost (2.1- to 2.8-fold, three Variant A pairs), and over all tokens it overstates the log-perplexity benefit 1.7- to 2.6-fold (Appendix AC; one seed, not pre-registered). On MQAR-update, a retraction task, the larger share is the head's: it carries most of the regression in both GDN pairs scored, and removal understates the harm by the backbone's remaining deficit, resolved in one of them; in one of two Mamba-1 pairs the backbone shows no resolved deficit and the head carries the regression (Section 6, Figure 1(c)). The mixture argument predicts a backbone deficit on copyable positions, which covers the LAMBADA deficit and could cover GDN's remaining MQAR-update backbone deficit; it does not predict the head's own vote for the superseded value, or the absence of a resolved Mamba-1 backbone deficit. We now compute a reference for it: with the trained head fixed and the baseline standing in for the data, the objective's optimum reproduces 0.70 of the LAMBADA deficit in one 60M GPT-2-style pair (uncapped head, one seed, the initialization caveat applies; Appendix AD) and 0.47-0.63 in four 130M pairs with the capped head (pre-registered; unified-recipe checkpoints, one seed each; Appendix AE). We do not claim the LAMBADA deficit exceeds what that objective requires. The same reference asks for no deficit on targets whose most recent earlier occurrence lies beyond the windowed head's reach (1.8% of scored WikiText-103 tokens), yet on those targets the closed-gate backbones of those four pairs are 0.10-0.23 nats per token worse than their baselines, every 95% interval above zero (Appendix AE; the stratum was chosen post hoc on the main-table pairs, and these are other checkpoints). On targets with no earlier occurrence they stay better than their baselines (Table 49). We have not identified the cause of the beyond-window deficit.

Weakness 2 and Q2: hidden-state probes and a new readout. The submitted claim that copyable identities stay about as decodable from the hidden states was too strong, and you are right that a probe near 40% against a native readout near 50% cannot rule out a loss. A full-vocabulary linear readout refit on all five families narrows but does not close the baseline-copy gap on copyable targets, which stays significant in every family (Table 36). The refit lowers both arms' accuracy, so it does not localize the gap's cause or answer your Q2, and we no longer claim a share of the deficit for the readout. We have since run the pre-registered native-initialized readout test on the Llama-style pair; it was underpowered (at the largest Adam learning rate that left the baseline's own readout undamaged on held-out text, the optimizer recovered only 13% of a known perturbation of that readout in a power control), so its result, R = 0.004 [95% CI -0.004, 0.013], is a lower bound at that setting, does not localize the deficit, and the question remains open.

(Part 3/3)

Q3: MQAR-update. We scored MQAR-update on the same 5B checkpoints, gate closed and per pathway, under a pre-registered protocol (Appendix N, Table 23; 200 items, one seed; published cells reproduced within tolerance after two exact-gate failures, all disclosed; repeat scoring moves some cells by a few items, cause not attributed). In GDN the copy pathway's vote and the gate closure both point to the head; the same label holds in one of two Mamba-1 pairs. Llama-style is unresolved at this sample size, and the GPT-2-style head improves the task. For Table 22, the models answer the same items and only per-task counts were stored, so we removed the independent standard errors and give no paired intervals. We do not present a lower cap as a mitigation: at c = 0.1 the same checkpoints lose 16.9 (GDN) and 15.6 (Mamba-1) LAMBADA points (Table 11).

In those four pairs head variant and backbone class coincide, so we scored five further pairs that all carry the Variant A head (the Appendix R retrain and four Appendix AA pairs; Table 23, lower rows), with vote and class rules fixed in advance. Neither single-factor outcome occurred: both SSM-family copy pathways vote stale, but so does one of the three Transformer pathways, on a regression carried mainly by the backbone, while the other two favour the corrected value. The pre-registered class verdict is therefore mixed and we draw no architecture conclusion; the GDN label recurs, and the Mamba-1 regression does not recur as resolved under the unified recipe, although its copy pathway still votes stale.

Q3: does closing the gate restore the baseline score? In GDN it does not; in Mamba-1 it does within resolution. Same-pass accuracy, baseline / gate closed / published state (Table 23): GDN 82.0 / 69.0 / 28.0%, so closing recovers 41.0 of 54.0 lost points and a backbone deficit of 13.0 remains (95% CI 6.5-19.5); Mamba-1 69.5 / 72.0 / 59.5%, backbone part -2.5 (CI -10.5 to 5.0), not resolved. Under the unified recipe GDN is 82.0 / 75.0 / 28.5, backbone part 7.0 (CI 0.0-14.5), not resolved. Per pathway, GDN's copy pathway alone ranks the superseded value first on 94% of items; its vocabulary pathway ranks the corrected value first on 69%, against 82% for the baseline (Appendix N). The remaining rows are in Table 23: the upper Llama-style and GPT-2-style rows have no resolved regression, and the unified GPT-2-style regression noted above is carried mainly by the backbone, so closing the gate does not restore it either. 200 items, one seed.

Reviewer_Lw9xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The problem of concern is the notion of copying , which indeed goes back to about a decade ago in neural language modeling. Attention heads learned to do in-context retrieval implicitly, and as such, dedicated copying heads have all but disappeared; as such, the authors examine if training models with an explicit pointer-copy head would improve their ability to copy.

With respect to experimentation, the authors have five architecture families (Mamba-1, Jamba style, Gated DeltaNet, GPT-2 and Llama) which they evaluate on WikiText-103, LAMBADA, downstream benchmarks, and held-out FineWeb-Edu dataset.

Main finding is that this joint training improves WikiText-103 perplexity and LAMBADA accuracy across all of the architecture variants. Further, they find that removing the copy head at inference causes LAMBADA accuracy to decrease to 3.6-6.5% (which was below baselines without said copy head). There is some discussion about how this demonstrates an output-pathway offloading effect, "in which the backbone’s vocabulary readout delegates in-context copying to the head rather than duplicating it".

Critical strengths
  • I think the empirics are fairly thorough. In particular, they had many ablations and inference-time interventions to support their conclusions.
  • Also, the experiments involving removing the copy head at inference and probing the hidden representations were well-done. Their finding that the backbone's vocabulary readout becomes less accurate at copying, while the relevant information remains approximately as decodable from the hidden states, provides evidence for their proposed offloading effect.
Critical weaknesses
  • I am not entirely convinced by novelty & significance of the finding. The copying mechanism is, of course, not new, and the observation that this jointly trained model leads to additional output pathway seems natural. Now, the authors do indeed demonstrate the phenomenon convincingly, but beyond this expected behavior it is unclear to me what the main scientific insight is.
  • The experiments are on small models (130M-350M) and are conducted primarily on only a single seed. Further, the copy-head configurations differ across the architectures, which evidently limits the conclusions that can be drawn about how particular architectures benefit from copying.
Questions for authors
  • (Related to weakness 1:) What is the main scientific contribution beyond the established behavior of pointer-generator models? Why should the offloading effect be considered surprising or fascinating?
  • (Related to weakness 2) Would the same degree of dependence on copy head arise in larger and more trained models (I know this may be hard to test in rebuttal period but any sort of preliminary results even could add a lot)?
  • Did you investigate/could you investigate ways to mitigate negative effects of copying outdated information (for example, looking at the significant degradation on MQAR-update for the Gated DeltaNet from 82.5% to 28%)?
4 comments

(Part 1/2)

Thank you for the review. Numbering follows the revised PDF.

Weakness 1 and Q1: the main scientific contribution. You asked what the insight is beyond mixture-likelihood specialization. The objection is fair against the submitted framing, and we no longer present the existence of offloading as the finding. The revision leads with this: inference-time removal measures reliance within a jointly trained system and can overstate or understate what including a component during training does. On LAMBADA it overstates the benefit: for Mamba-1 at 5B tokens, removing the head costs 20.9 points, while the full model is 2.7 points above a baseline trained without a head; the other 18.2 points are a backbone deficit (Section 5.4; single seed at 5B, three seeds at 130M/2B in Table 44). On WikiText-103 non-copyable tokens removal instead overstates the head's cost (2.1- to 2.8-fold, three Variant A pairs), and over all tokens it overstates the log-perplexity benefit 1.7- to 2.6-fold (Appendix AC; one seed, not pre-registered). On MQAR-update, a retraction task, the larger share is the head's: it carries most of the regression in both GDN pairs scored, and removal understates the harm by the backbone's remaining deficit, resolved in one of them; in one of two Mamba-1 pairs the backbone shows no resolved deficit and the head carries the regression (Section 6, Figure 1(c)). The mixture argument predicts a backbone deficit on copyable positions, which covers the LAMBADA deficit and could cover GDN's remaining MQAR-update backbone deficit; it does not predict the head's own vote for the superseded value, or the absence of a resolved Mamba-1 backbone deficit. We now compute a reference for it: with the trained head fixed and the baseline standing in for the data, the objective's optimum reproduces 0.70 of the LAMBADA deficit in one 60M GPT-2-style pair (uncapped head, one seed, the initialization caveat applies; Appendix AD) and 0.47-0.63 in four 130M pairs with the capped head (pre-registered; unified-recipe checkpoints, one seed each; Appendix AE). We do not claim the LAMBADA deficit exceeds what that objective requires. The same reference asks for no deficit on targets whose most recent earlier occurrence lies beyond the windowed head's reach (1.8% of scored WikiText-103 tokens), yet on those targets the closed-gate backbones of those four pairs are 0.10-0.23 nats per token worse than their baselines, every 95% interval above zero (Appendix AE; the stratum was chosen post hoc on the main-table pairs, and these are other checkpoints). On targets with no earlier occurrence they stay better than their baselines (Table 49). We have not identified the cause of the beyond-window deficit.

Concurrent work (also relevant to Q2). Concurrent work (Godey & Artzi, 2026) reports gains from a jointly trained pointer head at up to 1.7B parameters and fewer induction heads in those backbones, but no scores with the head removed. Our closed-gate scores supply them: LAMBADA 3.6-6.5%, against 18-23% without a head. Whether reliance persists to 1.7B as their gains do is open. The heads differ, so the results are complementary, not a replication.

Weakness 2 and Q2: seeds and scale. Three seeds at 130M/2B are in Table 44: removal costs 19.8 +/- 0.7 (GPT-2-style) and 21.1 +/- 0.6 LAMBADA points (Mamba-1) against gains of 5.9 +/- 1.4 and 6.1 +/- 0.7, so removal exceeds the gain 2.9- to 4.3-fold in every seed (Section 5.4). Table 45 adds 10B-token rows at 130M and about 800M parameters, one seed each: the closed/baseline LAMBADA ratio is unchanged from 5B to 10B at 130M and higher at 800M; the holdout gain shrinks, and the other gains show no consistent trend.

Weakness 2: copy-head configurations differ across architectures. Appendix AA trains one Variant A head configuration on six backbones under one recipe (Table 43). Every sign holds in every row, with closed-gate LAMBADA accuracy below a third of baseline; rows are single-seed (five backbones at 5B, Mamba-3 SISO at 2B).

(Part 2/2)

Q3: MQAR-update. We scored MQAR-update on the same 5B checkpoints, gate closed and per pathway, under a pre-registered protocol (Appendix N, Table 23; 200 items, one seed; published cells reproduced within tolerance after two exact-gate failures, all disclosed; repeat scoring moves some cells by a few items, cause not attributed). In GDN the copy pathway's vote and the gate closure both point to the head; the same label holds in one of two Mamba-1 pairs. Llama-style is unresolved at this sample size, and the GPT-2-style head improves the task. For Table 22, the models answer the same items and only per-task counts were stored, so we removed the independent standard errors and give no paired intervals. We do not present a lower cap as a mitigation; its LAMBADA cost is given below.

In those four pairs head variant and backbone class coincide, so we scored five further pairs that all carry the Variant A head (the Appendix R retrain and four Appendix AA pairs; Table 23, lower rows), with vote and class rules fixed in advance. Neither single-factor outcome occurred: both SSM-family copy pathways vote stale, but so does one of the three Transformer pathways, on a regression carried mainly by the backbone, while the other two favour the corrected value. The pre-registered class verdict is therefore mixed and we draw no architecture conclusion; the GDN label recurs, and the Mamba-1 regression does not recur as resolved under the unified recipe, although its copy pathway still votes stale.

Q3, mitigation. The paper offers no mitigation for copying a superseded value; it measures the harm and attributes it. The one setting we scored is the inference-time cap. Lowering it raises MQAR-update accuracy on the same checkpoints (in %, at c = 0.5, 0.25 and 0.1: GDN 28.0, 45.5, 63.0; Mamba-1 59.5, 66.0, 70.0), but relative to c = 0.5 those checkpoints lose 16.9 (GDN) and 15.6 (Mamba-1) LAMBADA points at c = 0.1 (Table 11), and with the gate fully closed GDN is still 13.0 points below its baseline on MQAR-update (Table 23). The cap therefore moves a trade-off, and we do not call it a mitigation (Appendix N). The training interventions of Appendix Y target the backbone's LAMBADA deficit, and the paper reports no MQAR-update score for them. The paper reports no variant trained against this failure: its claim is about what removal measures (Sections 5.4 and 6), not a better head, and it proposes no remedy. Scope of the numbers: 200 synthetic items, one seed, Jamba-style not scored.

Hidden-state probes, which your review cites. The submitted claim that copyable identities stay about as decodable from the hidden states was too strong, and rqDY is right that a probe near 40% against a native readout near 50% cannot rule out a loss. A full-vocabulary linear readout refit on all five families narrows but does not close the baseline-copy gap on copyable targets, which stays significant in every family (Table 36). The refit lowers both arms' accuracy, so it does not localize the gap's cause or answer rqDY's Q2, and we no longer claim a share of the deficit for the readout. We have since run the pre-registered native-initialized readout test on the Llama-style pair; it was underpowered (at the largest Adam learning rate that left the baseline's own readout undamaged on held-out text, the optimizer recovered only 13% of a known perturbation of that readout in a power control), so its result, R = 0.004 [95% CI -0.004, 0.013], is a lower bound at that setting, does not localize the deficit, and the question remains open.

Firstly, I want to thank the authors for their response.

  1. I believe the paper is written in a very unusual manner now. It is almost as if it is written to be a summary of the initial submission, and rather than cleaning up the presentation, the paper reads as if every paragraph is a summary of what was once there. I would spend significant effort improving writing quality.

  2. Also, as Reviewer atSF also mentioned, what exactly does the following mean: "The revision leads with this: inference-time removal measures reliance within a jointly trained system and can overstate or understate what including a component during training does." I do not believe that the response provided to them was sufficiently clarifying.

  3. I am also extremely confused on your response to my question regarding mitigation. What do you mean by "The cap therefore moves a trade-off, and we do not call it a mitigation"?

I will maintain my score given these outstanding concerns.

Thank you for the follow-up, and for being direct about the writing. We agree with you. The earlier revision tried to keep almost every result in the main text, so each paragraph turned into a list of findings. In this version the main text keeps the core results and explains them more slowly. The secondary results, such as the larger scales, Mamba-3 and the cap sweep, are now short pointers to the appendix. We also define the three quantities we compare once, in the introduction, so the same arithmetic is not repeated in every section. Section 5.4 shows the change best. The results themselves have not changed.

You asked what we meant by "inference-time removal measures reliance within a jointly trained system and can overstate or understate what including a component during training does". By "the revision leads with this" we meant that the revised paper now starts with this claim. The claim means that removing a part from a trained model can greatly overstate, or understate, what that part adds. Take Mamba-1 on LAMBADA. Trained with the copy head, it scores 25.9%. When we remove the head at inference, it scores 5.0%. A Mamba-1 trained without the head scores 23.2%. So removal costs 20.9 points, but the head adds only 2.7. The other 18.2 points are lost by the rest of the model. Trained next to the head, it scores worse on its own than a model trained without the head, mostly on words that appeared earlier in the text. Removing a part from a trained model therefore measures how much the model depends on that part (here 20.9 points), not what the part added (2.7). On MQAR-update, removal understates the harm instead. The head lowers GDN's accuracy from 82.5% to 28.0% (Section 6). The decomposition in Table 23, from a separate scoring pass, shows that removing the head brings GDN back only to 69.0%, 13 points below the baseline's 82.0% in that pass, because the rest of the model is also worse.

You also asked what we meant by "The cap therefore moves a trade-off, and we do not call it a mitigation". The Variant A head has a gate cap c, which limits how much it can copy, and we can lower it at inference without retraining. On MQAR-update this helps. GDN rises from 28.0% at c = 0.5 to 63.0% at c = 0.1. But the same change costs 16.9 LAMBADA points, because the head is also the source of the LAMBADA gain. Lowering the cap improves MQAR-update but worsens LAMBADA, so at best it is a partial fix; that is what we meant by "moves a trade-off". Our auxiliary loss works on a different problem, the backbone's LAMBADA shortfall: an extra training loss on the backbone's own predictions recovers 48-65% of the shortfall and keeps 88-100% of the head's gain on the replication seeds (77% on the Mamba-1 selection seed; 60M, two families). We have not tested whether it also helps on MQAR-update.

Reviewer_atSFOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1(−1)
Read review
Paper summary

This paper studying whether an output-side copying module improves performance and how joint training changes the behavior of the underlying model. The authors evaluate five architecture families:

  1. Transformers: GPT-2-style and Llama-style Transformers
  2. SSM Mamba-1, a Jamba-style hybrid, and Gated DeltaNet

with a controlled no pointer-copy baseline and copy-augmented training run, at a 130M–198M parameters and 5B training tokens.

The copy head constructs a distribution over previously encountered token identities and interpolates it with the backbone's vocabulary distribution using a learned gate. This has two effects:

  1. copying is offloaded to the module meant for copying
  2. the vocabulary pathway (in the pointer-copy architecture) leads to improved perplexity on non-copyable tokens

The authors show improvements on WikiText-103 perplexity, LAMBADA accuracy, and held-out language modeling perplexity across all five families.

However, disabling the copy head at inference leads to a substantial degradation in copying-sensitive performance, with LAMBADA accuracy falling below independently trained baselines. The authors interpret this as output-pathway offloading: the backbone's vocabulary readout learns to rely on the copy head rather than independently modeling in-context repetition. However, Linear-probe experiments on two Transformer backbones further suggest that copyable token identities remain comparably decodable from their final hidden representations, despite reduced accuracy of their native vocabulary readouts.

Critical strengths

1. Broad experimental coverage across architecture families.

The paper evaluates the same general intervention across five distinct language model architectures, including models with conventional attention, state-space models, and recurrent-style architectures. The consistent improvements within baseline/+copy pairs provide useful empirical evidence that explicit output-side copying remains beneficial beyond traditional Transformers.

2. Careful inference-time controls.

The closed-gate and source-identity randomization experiments are useful for characterizing the contribution of the copy pathway. In particular, corrupting the retrieved token identities while preserving the attention weights and gate behavior provides evidence that the observed gains depend on the identities of retrieved tokens.

3. Useful distinction between representations and output readouts.

The linear-probe experiments provide evidence that the degradation in native copying performance is not accompanied by an equivalent reduction in the linear decodability of copy targets from hidden states. Although limited to two Transformer backbones, this is a useful observation that helps localize the measured dependency.

Critical weaknesses
  1. Unclear problem statement: It is unclear what insight does the paper provide, the observed specialization is a largely expected consequence of optimizing a mixture likelihood. A generative model is trained with a mixture of two heads, with one head exclusively meant for copying, and at inference this head is removed, leading to reduced performance.
  2. Pangram assigns a 100% probability to the paper being AI written (I pasted several paragraphs of the paper into the free pangram app). While, the authors acknowledge generative AI usage, the paper can benefit from a substantial rewrite as it is very difficult to parse.
Questions for authors
  1. Can the author clarify what exactly does experimental insight does the paper provide besides:
    1. Training a model with an explicit copying mechanism improves performance on retrieval tasks, and removing the copying mechanism at inference time leads to a training-inference mismatch, leading to reduced performance.
  2. What if the training includes randomly dropping the pointer mechanism? would that improve the vocabulary pathway’s performance on copying at inference time?
6 comments

(Part 1/2) Thank you for the review. Numbering follows the revised PDF.

Weakness 1 and Q1: the insight. You asked what the insight is beyond mixture-likelihood specialization. The objection is fair against the submitted framing, and we no longer present the existence of offloading as the finding. The revision leads with this: inference-time removal measures reliance within a jointly trained system and can overstate or understate what including a component during training does. On LAMBADA it overstates the benefit: for Mamba-1 at 5B tokens, removing the head costs 20.9 points, while the full model is 2.7 points above a baseline trained without a head; the other 18.2 points are a backbone deficit (Section 5.4; single seed at 5B, three seeds at 130M/2B in Table 44). On WikiText-103 non-copyable tokens removal instead overstates the head's cost (2.1- to 2.8-fold, three Variant A pairs), and over all tokens it overstates the log-perplexity benefit 1.7- to 2.6-fold (Appendix AC; one seed, not pre-registered). On MQAR-update, a retraction task, the larger share is the head's: it carries most of the regression in both GDN pairs scored, and removal understates the harm by the backbone's remaining deficit, resolved in one of them; in one of two Mamba-1 pairs the backbone shows no resolved deficit and the head carries the regression (Section 6, Figure 1(c)). The mixture argument predicts a backbone deficit on copyable positions, which covers the LAMBADA deficit and could cover GDN's remaining MQAR-update backbone deficit; it does not predict the head's own vote for the superseded value, or the absence of a resolved Mamba-1 backbone deficit. We now compute a reference for it: with the trained head fixed and the baseline standing in for the data, the objective's optimum reproduces 0.70 of the LAMBADA deficit in one 60M GPT-2-style pair (uncapped head, one seed, the initialization caveat applies; Appendix AD) and 0.47-0.63 in four 130M pairs with the capped head (pre-registered; unified-recipe checkpoints, one seed each; Appendix AE). We do not claim the LAMBADA deficit exceeds what that objective requires. The same reference asks for no deficit on targets whose most recent earlier occurrence lies beyond the windowed head's reach (1.8% of scored WikiText-103 tokens), yet on those targets the closed-gate backbones of those four pairs are 0.10-0.23 nats per token worse than their baselines, every 95% interval above zero (Appendix AE; the stratum was chosen post hoc on the main-table pairs, and these are other checkpoints). On targets with no earlier occurrence they stay better than their baselines (Table 49). We have not identified the cause of the beyond-window deficit.

Concurrent work. Concurrent work (Godey & Artzi, 2026) reports gains from a jointly trained pointer head at up to 1.7B parameters and fewer induction heads in those backbones, but no scores with the head removed. Our closed-gate scores supply them: LAMBADA 3.6-6.5%, against 18-23% without a head. Whether reliance persists to 1.7B as their gains do is open. The heads differ, so the results are complementary, not a replication.

Weakness 2: presentation. We accept that the submitted paper was hard to parse. We do not dispute the detector's output, and we make no statement about the writing beyond the paper's Declaration on Generative AI, which is unchanged: ChatGPT and Claude Opus were used to enhance grammar, rectify spelling mistakes and rephrase writing for clarity, and the authors take full responsibility for the content. The revision is not the substantial rewrite you recommend. What changed for the reader: the abstract opens with the claim and one worked case (Mamba-1: removal costs 20.9 LAMBADA points, the gain is 2.7) where the submitted abstract listed results; the introduction states two questions and four numbered contributions where it had six result bullets; Sections 5.3 and 5.4 are retitled for what they show ("Copy Gains Are Concentrated in Copyable Targets", "Closing the Gate: Reliance Is Not Benefit"); Figure 1 has a third panel for MQAR-update; Appendix M defines each task; and the scoring corrections are collected in Appendix P.1 with a before/after table (Table 28). The revision is also longer, 53 pages against 31, and the text is still dense in places.

Q2: dropping the pointer at random during training. It behaves like the auxiliary loss. At 60M over three seeds, closed-gate LAMBADA rises above the jointly trained value but stays below the baselines, and the full model keeps 93-96% of the LAMBADA gain (Table 40). These comparisons use the original baseline scorer, whose effect is bounded in Appendix P.1.

(Part 2/2)

Your review also cites two statements of the submitted version that the revision corrects.

Hidden-state probes. The submitted claim that copyable identities stay about as decodable from the hidden states was too strong, and rqDY is right that a probe near 40% against a native readout near 50% cannot rule out a loss. A full-vocabulary linear readout refit on all five families narrows but does not close the baseline-copy gap on copyable targets, which stays significant in every family (Table 36). The refit lowers both arms' accuracy, so it does not localize the gap's cause or answer rqDY's Q2, and we no longer claim a share of the deficit for the readout. We have since run the pre-registered native-initialized readout test on the Llama-style pair; it was underpowered (at the largest Adam learning rate that left the baseline's own readout undamaged on held-out text, the optimizer recovered only 13% of a known perturbation of that readout in a power control), so its result, R = 0.004 [95% CI -0.004, 0.013], is a lower bound at that setting, does not localize the deficit, and the question remains open.

"Improved perplexity on non-copyable tokens". The submitted wording overstated this. Most of that improvement on the SSM-family backbones is probability shifted away from in-context tokens. Scored only among tokens outside the window, the closed-gate backbone's perplexity is approximately unchanged or lower, and the shift accounts for about half to all of the unconditional change, depending on family (Section 5.4 and its footnote; supplement data/noncopyable_check). This is single-seed at 130M, and the split is approximate: its 512-token window matches neither head's reach exactly. We have not rescored it under the source mask, so we write "concentrated in", not "confined to"; Appendix N bounds, by a tokenizer-only count, how many items change category under either mask. The deployed mixture is still worse than baseline on non-copyable LAMBADA targets in every pair, which we read as a property of our gates (Section 5.3).

  1. The rebuttal still does not provide a clear, readable answer on what the paper actually is about.

Can the authors explain why this inference-time removal is being studied? Precisely, what question about the remaining architecture can be answered by this removal, that one could not already reasonably infer.

  1. Can the authors provide a precise explanation of what they mean by this statement:

The revision leads with this: inference-time removal measures reliance within a jointly trained system and can overstate or understate what including a component during training does

How does it relate to the paper?

  1. As other reviewers have complained about readability as well, I maintain my score.

Comments on the paper

1 comment

Thank you for the reviews. The revision narrows the claim and corrects the submitted version where you found it wrong; numbering follows the revised PDF. Each review has its own reply under it; this comment summarizes what changed.

Main claim. We no longer present the existence of offloading as the finding. The revision leads with this: inference-time removal measures reliance within a jointly trained system and can overstate or understate what including a component during training does. On LAMBADA it overstates the benefit: for Mamba-1 at 5B tokens, removing the head costs 20.9 points, while the full model is 2.7 points above a baseline trained without a head; the other 18.2 points are a backbone deficit (Section 5.4; single seed at 5B, three seeds at 130M/2B in Table 44). On MQAR-update, a retraction task, the larger share is the head's: it carries most of the regression in both GDN pairs scored, and removal understates the harm by the backbone's remaining deficit, resolved in one of them; in one of two Mamba-1 pairs the backbone shows no resolved deficit and the head carries the regression (Section 6, Figure 1(c)).

New reference result (Appendices AD and AE). The mixture argument predicts a backbone deficit on copyable positions, which covers the LAMBADA deficit and could cover GDN's remaining MQAR-update backbone deficit; it does not predict the head's own vote for the superseded value, or the absence of a resolved Mamba-1 backbone deficit. We now compute a reference for it: with the trained head fixed and the baseline standing in for the data, the objective's optimum reproduces 0.70 of the LAMBADA deficit in one 60M GPT-2-style pair (uncapped head, one seed, the initialization caveat applies; Appendix AD) and 0.47-0.63 in four 130M pairs with the capped head (pre-registered; unified-recipe checkpoints, one seed each; Appendix AE). We do not claim the LAMBADA deficit exceeds what that objective requires. The same reference asks for no deficit on targets whose most recent earlier occurrence lies beyond the windowed head's reach (1.8% of scored WikiText-103 tokens), yet on those targets the closed-gate backbones of those four pairs are 0.10-0.23 nats per token worse than their baselines, every 95% interval above zero (Appendix AE; the stratum was chosen post hoc on the main-table pairs, and these are other checkpoints). On targets with no earlier occurrence they stay better than their baselines (Table 49). We have not identified the cause of the beyond-window deficit.

Corrections to the submitted version.

  • 2B baselines (rqDY). Every 2B cell is now rescored in one environment, baselines head-free (Table 7, Appendix P.1). The GDN 2B WikiText-103 reduction is -12.7%, not the -18.3% we reported.
  • Llama-style WikiText-103 discrepancy (rqDY). It was neither the scorer nor bfloat16 attention: both numbers came from the freshly initialized, unseeded copy head the backend attached to baselines. The corrected reduction is -24.4% (submitted -21.7%); see Appendix P, item (8), Appendix P.1 and Appendix Z.
  • Hidden-state probes (atSF, Lw9x, rqDY). The submitted claim that copyable identities stay about as decodable from the hidden states was too strong. A full-vocabulary linear readout refit on all five families narrows but does not close the baseline-copy gap on copyable targets, which stays significant in every family (Table 36). The refit lowers both arms' accuracy, so it does not localize the gap's cause or answer rqDY's Q2, and we no longer claim a share of the deficit for the readout.
  • "Improved perplexity on non-copyable tokens" (atSF). The submitted wording overstated this. Most of that improvement on the SSM-family backbones is probability shifted away from in-context tokens. See Section 5.4 and its footnote.
  • Weight initialization (LXVy). The GPT-2-style stack is built with nn.TransformerEncoder, which clones one layer, so all layers start with identical weights in both arms at 60M, 130M and 350M; those runs also keep N(0,1) embeddings. This was not disclosed before; it is now in the caption of Table 6, Section 4 and Section 7. We keep the caveat (Table 6, Section 7) that GPT-2-style magnitudes are not comparable across families.
  • Paired runs (rqDY). We now call the pairs runs on one corpus and nominal recipe, not controlled pairs.

Added results. Appendix AA trains one Variant A head configuration on six backbones under one recipe (Table 43). Three seeds at 130M/2B are in Table 44. Table 45 adds 10B-token rows at 130M and about 800M parameters, one seed each. We scored MQAR-update on the same 5B checkpoints, gate closed and per pathway, under a pre-registered protocol (Appendix N, Table 23).

Presentation (all reviewers). The revision reorganizes the abstract and introduction around one claim; it is not a full rewrite, and it is longer (53 pages against 31). Details are in our reply to atSF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.