- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a modular attention-based component that supports copying by calculating the output token prediction by aggregating attention weights over prior occurrences of tokens, then interpolating with the distribution determined by the main model’s backbone with a learned gate. The authors demonstrate that models with this learnable modular attachment improve on overall perplexity and accuracy on WikiText-103 and LAMBADA over baselines; however, the gains are concentrated to when the target token has previously occurred in the context window. Experiments primarily between the 125M and 350M scale, trained from 2B to 5B tokens, across various model families show that this “copying head” improves performance on measured downstream tasks within these experimental configurations but can degrade tasks such as MQAR.
Critical strengths
The paper demonstrates that the copying head improves performance on the small scale under their experimental setup. The authors test variations of the copy head, explore the limitations of the methodology at two scales, sweep cap c values, and replicate findings with multiple seeds.
Critical weaknesses
The experimental setup raises concerns about whether the improvements observed in this paper would persist under stronger training configurations. For instance, the Mamba series’ improved training recipe trains the 125M and 350M models with a peak lr of 3e-3 and 1.5e-3 respectively. Although the baseline and copy-head models utilize the same lr, it is unclear whether the gains would hold under better hyperparameters. In addition, upon looking at some of the scripts in the supplemental material, it seems like there are potential issues with weight initialization where the layers are initialized with the same parameters (see GPT2TF initialization in train_gpt2_tf_130m_copy.py). Furthermore, the context length of 1K is smaller than the more standard 2K/4K context lengths in architecture research literature. It remains to be seen whether the improvements hold at longer training context, as it may be possible that a shorter context length may bias certain patterns in the data.
While the authors provide several ablations and comparisons, I believe that there is a lack of important experiments which isolate the functionality of the copy head and establish what the copy head contributes beyond a context-dependent repetition prior. For instance, there should be additional ablations that try to determine out whether the “copy head” is just upweighting previously seen tokens: a simple baseline would be to replace the attention weights with a uniform weight based on the distribution of previous tokens. Furthermore, a standard baseline which replaces the copy head with a SWA head (or a slightly adjusted variant with an identity value projection) would additionally help determine whenever the copy-head provides benefits beyond an additional attention-based retrieval mechanism.
Questions for authors
See weaknesses. How were the experimental hyperparameters selected, and do the gains persist with stronger baselines? Furthermore, do the gains hold across scale? Further fleshing out the role the copy-head plans would be helpful.
For Appendix M, is there a reference to what these tasks are based on or where they specifically measure? Based on my understanding, these types of tasks are usually toy-tasks that require a model to be trained from scratch instead of testing an already trained model on them (see Mechanistic Design and Scaling of Hybrid Architectures).
6 comments
(Part 2/2)
The copy head's role, as far as the paper establishes it. (1) Reliance is not benefit: removal costs 20.9-25.6 LAMBADA points in the five 5B pairs; 2.7-7.9 are gain over a head-free baseline, 13.9-18.2 a backbone deficit (Section 5.4). (2) The gain sits on copyable positions: +4 to +11 LAMBADA points where the target occurs in the prior 512 tokens; elsewhere accuracy is 1-4% with or without the head (Section 5.3, Table 19; the split is approximate). (3) It needs position-specific identities: permuting identities within each query's exact candidate set, at inference, drops each of the five main 5B models below its baseline (Table 4). (4) Reliance persists when the head's gradients are kept out of the backbone: with the backbone detached, closed-gate LAMBADA is 0.7% and 1.5% against baselines of 10.7% and 13.3% (60M, three seeds; GPT-2-style, initialization caveat of Section 7, and Mamba-1; Table 41). (5) Beyond the head's window (1.8% of WikiText-103 tokens) four 130M closed-gate backbones are 0.10-0.23 nats per token worse than their baselines; stratum chosen post hoc, one seed each, cause not identified (Appendix AE).
Appendix M. Appendix M now defines each task and states that these are zero-shot probes we designed, not train-from-scratch tasks.
Thank you for your response and additional experiments.
Unfortunately, this response raises additional concerns regarding the experimental methodology, reproducibility, and presentation.
The new runs used a different GPU type and possibly a different PyTorch build, which three seeds cannot separate from the initialization.
The authors should confirm whether there were any changes to the GPU type and PyTorch build. Combined with the weight initialization issue from before, this uncertainty about experimental setup is frankly concerning. I would expect authors to be able to verify their own experimental setup.
In addition, my original concerns remain unanswered, particularly those that ask for a simple context-dependent baseline.
Finally, the new rewrite has further decreased readability. For instance, this is the first sentence of the abstract:
Inference-time removal measures reliance within a jointly trained system and can overstate or understate what including a component during training does.
It has come to my attention that another reviewer (atSF) felt like the submission was heavily AI generated. Pangram also assigns the joint response, my specific response, and abstract as 100% AI generated.
Notwithstanding the empirical findings and soundness, the current submission's presentation needs to be rethought.
(Part 1/2)
Thank you for the review. Numbering follows the revised PDF.
Learning rate. The peak learning rate is the same for every family at 130M-198M and was not tuned per family; higher peak rates are untested (Sections 4 and 7).
How the hyperparameters were selected. The paper reports the values used and describes no search, and we do not present the recipe as tuned. At 130M-198M every family uses peak learning rate 6 x 10^-4, not tuned per family (Section 4, Appendix A); weight decay, warmup, learning-rate floor and micro-batching are set per architecture (Table 5). The 350M runs use an effective batch of 32 sequences and peak 3 x 10^-4 (Appendix A); the 800M-class runs use peak 2.5 x 10^-4 (Appendix AB). The paper gives no selection procedure for either. The unified recipe was fixed before any run (Appendix AA). All these rates are below the ones you cite; higher peak rates are untested (Section 7), so whether the gains hold under a stronger recipe is open.
Weight initialization. You were right. The GPT-2-style stack is built with
nn.TransformerEncoder, which clones one layer, so all layers start with identical weights in both arms at 60M, 130M and 350M; those runs also keep N(0,1) embeddings. This was not disclosed before; it is now in the caption of Table 6, Section 4 and Section 7. The other four families build layers independently and all show the collapse. A pre-registered control (GPT-2-style 60M, 1B tokens, three seeds; layers redrawn independently, every other draw kept; published checkpoints rescored in the same environment; baselines head-free on both sides, unlike Table 17) moved none of the three pre-registered quantities (closed-gate LAMBADA accuracy, LAMBADA and WikiText-103 copy gains) past its threshold. The closed-gate threshold (about 15 items, one LAMBADA pass's binomial resolution) was fixed before any run; under our first rule, three literal seed s.d., the closed-gate move (26 -> 33 of 5,153) would have been flagged, so we report it. At either value the closed gate stays about 10 points below the head-free baseline (11.0%). Two cells outside the decision moved by several published seed s.d.; we attribute no cause. This excludes large moves, not small ones, and was not run at 130M or 350M, at other token budgets, or with N(0,0.02) embeddings. The new runs used a different GPU type and possibly a different PyTorch build, which three seeds cannot separate from the initialization. We therefore keep the caveat (Table 6, Section 7) that GPT-2-style magnitudes are not comparable across families; files: supplement code/init60m/ and data/init60m/.Context length. Training context is 1,024 tokens throughout; at inference, for one Mamba-1 pair, the head holds at 2,048 tokens and reverses on WikiText-103 at 4,096 (Appendix AB), and training at longer context has not been run.
Controls not run. We have no trained uniform-attention head and no sliding-window attention head. The causal and binding shuffles (Table 4) and context partition (Table 40) are inference-time and set-based controls; they do not replace the training-time experiment you ask for. Truncating the Variant B window at inference removes 5-10% of the measured WikiText-103 and holdout gains and none of the LAMBADA gain (Appendix Q), but shorter-window training was not isolated.
Seeds and scale. Three seeds at 130M/2B are in Table 44: removal costs 19.8 +/- 0.7 (GPT-2-style) and 21.1 +/- 0.6 LAMBADA points (Mamba-1) against gains of 5.9 +/- 1.4 and 6.1 +/- 0.7, so removal exceeds the gain 2.9- to 4.3-fold in every seed (Section 5.4). Table 45 adds 10B-token rows at 130M and about 800M parameters, one seed each: the closed/baseline LAMBADA ratio is unchanged from 5B to 10B at 130M and higher at 800M; the holdout gain shrinks, and the other gains show no consistent trend.