- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper identifies an implicit reweighting of token input embeddings in pretrained decoder-only Transformers. Although the input embedding enters the residual stream with a fixed coefficient of one, its projection coefficient along the token's own embedding grows substantially with depth, and it is much larger than projections onto other token embeddings across six pretrained models. Based on this observation, the paper introduces Explicit Embedding Reweighting (EER), which adds a state-dependent positive multiple of the original embedding after residual updates. EER adds merely about 0.007% parameters and improves validation loss by 0.022–0.025 nats across three model scales. At the largest scale, it improves average zero-shot accuracy on seven tasks by 1.39 points and complements several recent residual-stream methods.
Critical strengths
- The motivation of the paper is clear and well-illustrated: The model shows substantial growth of the projection coefficient onto each token’s own token embedding across the layers.
- The paper is well-presented, and thus it is easy to understand.
Critical weaknesses
- While the paper shows a good motivation and presents a simple method, the method does not explicitly connect to the motivation. In other words, it does not directly show that increasing the own-embedding projection improves EER.
- Experiments go up to 900M total parameters (511M non-embedding) and 41B tokens. Embedding-skip designs are known to help small models most, and with a 152k vocabulary the embedding is a large share of parameters at this size, so it is unclear whether the gain persists at modern LLM scales.
Questions for authors
- The paper should specify their AI use statement in more detail.
- What do you mean when you state that the reference vectors are centered? Why is this needed? (Figure 1)
- How robust is EER when evaluated on substantially larger pretrained or pretrained-then-finetuned models? In particular, does the EER gain remain meaningful as model scale and training token budget increase?
4 comments
Response to Reviewer X4Tj (Part 2/2)
W2 / Q3. Scale, vocabulary share, token budget, pretrained models
Larger runs are the most direct test, so since the submission we have trained a larger pair, 1.45B parameters (982M non-embedding) on 78.5B tokens, with the same protocol [new] (Appendix E.1). EER lowers validation loss from 2.286 to 2.264 and raises the seven-task zero-shot average from 64.28% to 65.50% (+1.22 points), for 2.8% more training GPU-hours. This pair has one seed so far; the other two seeds are running, and we will update the results here once they finish. Across the four scales:
| Scale | Non-embedding params | Training tokens | Embedding and head share | Reduction (nats) |
|---|---|---|---|---|
| Small | 122.7M | 9.8B | 66% | 0.0250 |
| Medium | 272.7M | 21.8B | 53% | 0.0217 |
| Large | 511.2M | 40.9B | 43% | 0.0249 |
| 1.45B [new] | 982.0M | 78.5B | 32% | 0.0221 |
- Small-model effect. The gain stays at 0.022–0.025 nats while non-embedding parameters and training tokens both grow 8×, so it is not concentrated in the smallest models. EER also differs from a plain embedding skip: re-adding the embedding with a learned per-layer scalar gives only 0.009 nats even at Small scale, where such skips should help most, against 0.025 for EER (Table 3a).
- Vocabulary share. EER adds two width-sized vectors per block and no token-indexed table, so a 152k vocabulary gives it no extra capacity. In the table above, the gain stays in the same range as the share of the input embedding and output head falls from 66% to 32%.
- Token budget. Every run uses about 80 tokens per non-embedding parameter, four times the compute-optimal ratio (Hoffmann et al., 2022), so our models are trained well past the point around which scaling-law studies are usually run. EER is ahead at every one of the 10, 22, and 41 intermediate and final evaluations at Small, Medium, and Large (Appendix B.1), and at Large the gap is 0.0255 nats at 61% of the training budget and 0.0249 at the end [new]. We have not trained at higher token-to-parameter ratios.
- Modern scale. Figure 1 is measured on 7B–32B models, so the implicit reweighting behind EER is not a small-model effect.
- Pretrained models. We also tried adding EER to a pretrained model [new] (Appendix E.9). We inserted EER into Qwen3-8B-Base and continued training for 2B tokens; the loss shows no consistent difference from continued training of the unchanged model on the same tokens (within 0.005 nats), and inserting iHC in the same way gives the same result. We read this as a property of modifying a pretrained model rather than of EER: after pretraining on far more tokens, the model has settled how it uses its residual stream, and 2B tokens appear to be too few for it to make use of a new path. Like iHC, EER is meant to be part of the architecture from the start of training. This is the setting of all our other results, and there its gain stays in the same range up to 1.45B parameters.
Q1. AI use statement
We apologize for the overly brief statement. We have replaced it in the revised PDF with the following text.
We used OpenAI Codex, an LLM-based assistant, in three ways: (i) to check the training and evaluation code for errors; (ii) to write an initial draft of the manuscript, which we then revised; and (iii) after our revision, to check grammar and fluency of wording. The research idea, method, experimental design, and interpretation of results are our own. We reviewed and verified all AI-assisted text and code and take full responsibility for the content.
Q2. What does "centered" mean in Figure 1, and why is it needed?
Centering subtracts the mean embedding from the reference embeddings we project onto; the hidden states are unchanged. We use it because of one model, Qwen3-8B-Base, whose hidden states have a large component along this shared mean direction. That component raises the projection onto every token's embedding, including the control, and centering removes it so that the figure shows what is specific to each token's own embedding (Appendix C gives the numbers).
For the other five models centering makes little difference. Figure 7 of the revised PDF (Figure 6 in the submission) shows the curves without centering: the control stays near zero in those five models, and in all six models the projection onto the token's own embedding stays above the control at every block (Figure 6 of the revised PDF). The finding therefore does not depend on centering, and the Figure 1 caption now points to the uncentered curves.
Thank you again for your time. We hope these results address your main concerns, in particular the link between the observation and the method, and that you will take them into account in your final assessment. We would be glad to clarify anything further.
The review addressed my concerns. I will raise the score from 2 to 3.
Thank you for reading our response and for raising your score, we really appreciate it. Your point about connecting the observation to the method pushed us to run the analysis that is now Section 4.6, and we think the paper is clearly better for it. In the final version we will move this into the main text, along with the clarification about centering and the updated AI use statement.
Response to Reviewer X4Tj (Part 1/2)
Thank you for the review and the clear questions. Results not in the submission are marked [new]; they are in the revised PDF (Section 4.6 and Appendix E).
W1 (a). Does EER raise the own-embedding projection, and how does it relate to the implicit route?
We agree that the submission put the observation and the method side by side without testing the link in the same models. We have now tested it directly, in (a) and (b) below.
The link we have in mind is this. Models raise the embedding's weight through layers that see the embedding only inside a normalized mix, in which it keeps fading (Section 2). EER supplies the embedding itself, with a weight chosen from the current state, which changes exactly the quantity the implicit route is adjusting (Equation 5), and it keeps the embedding visible to the layers that follow.
We measured this on our Large Baseline and EER [new] (seed 42, 256 held-out documents). The Baseline behaves like the pretrained models: its projection coefficient onto the token's own embedding grows from 1 to about 10. Its embedding, however, fades from the state, while with EER it does not. Cosine between the state and the token's own embedding:
With EER, the projection coefficient is much higher, and the sublayers write more along the embedding, not less:
Within the trained EER model, switching off only the largest injections (blocks 14–19) makes the MLPs in blocks 17–22 write 12–39% less along the embedding, measured before each block's own injection. The explicit path therefore raises the projection both directly and by keeping the embedding visible to deeper layers, whose MLPs then write more along it. This analysis is in the new Section 4.6 and Figure 5.
W1 (b). Does changing the own-embedding projection change the loss?
Yes, and the projection set by EER's controller gives the lowest loss among the alternatives we tested. Raising the projection is not enough by itself: a fixed coefficient of one also raises it at every block but gives only 0.007 nats. What matters is where and for which tokens it is raised, which is what the state-dependent controller sets. Across separately trained models that differ only in how the added weight is set (Table 3a, three seeds):
Within a single trained model [new] (Appendix E.8), we changed the injection coefficients of the Large EER model without retraining. Moving them away from what the controller chose, in either direction, raises the loss:
Values are loss increases in nats. Such a change takes the model off its training distribution, which is why we read this table next to Table 3a and not by itself.
In short, EER raises the own-embedding projection, keeps the embedding visible to deeper layers, and the projection it chooses gives the lowest loss among the alternatives we tested. The revised Section 4.6 now makes this link explicit.