acceptodds
Under review as a conference paper at ICLR 2027

Residual Connections with Explicit Embedding Reweighting

Abstract

In a pre-norm Transformer, each token's input embedding enters the residual stream with a fixed coefficient of one. We find that pretrained language models nevertheless reweight the embedding implicitly: across six models from four families, the residual state's projection onto each token's own embedding grows substantially with depth and far exceeds its projection onto other tokens' embeddings. This reweighting is produced indirectly, by attention and MLP layers that, after the first, see the embedding only mixed with earlier updates. To provide a direct path for it, we introduce Explicit Embedding Reweighting (EER), a lightweight module that adds the original embedding with a positive coefficient predicted from the current state. EER keeps the identity shortcut and stores no layer outputs or extra embedding tables. At three scales (356M–900M parameters), EER lowers validation loss by 0.022–0.025 nats with about 0.007% additional parameters. At 900M parameters, it raises average zero-shot accuracy on seven tasks by 1.39 points, is competitive with Attention Residuals, and further improves SATFormer and identity Hyper-Connections. Ablations show that its state-dependent coefficient outperforms fixed, per-layer, and token-only coefficients.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformer, Large Language Model, Architecture, Residual Connection, Embedding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_X4TjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies an implicit reweighting of token input embeddings in pretrained decoder-only Transformers. Although the input embedding enters the residual stream with a fixed coefficient of one, its projection coefficient along the token's own embedding grows substantially with depth, and it is much larger than projections onto other token embeddings across six pretrained models. Based on this observation, the paper introduces Explicit Embedding Reweighting (EER), which adds a state-dependent positive multiple of the original embedding after residual updates. EER adds merely about 0.007% parameters and improves validation loss by 0.022–0.025 nats across three model scales. At the largest scale, it improves average zero-shot accuracy on seven tasks by 1.39 points and complements several recent residual-stream methods.

Critical strengths
  1. The motivation of the paper is clear and well-illustrated: The model shows substantial growth of the projection coefficient onto each token’s own token embedding across the layers.
  2. The paper is well-presented, and thus it is easy to understand.
Critical weaknesses
  1. While the paper shows a good motivation and presents a simple method, the method does not explicitly connect to the motivation. In other words, it does not directly show that increasing the own-embedding projection improves EER.
  2. Experiments go up to 900M total parameters (511M non-embedding) and 41B tokens. Embedding-skip designs are known to help small models most, and with a 152k vocabulary the embedding is a large share of parameters at this size, so it is unclear whether the gain persists at modern LLM scales.
Questions for authors
  1. The paper should specify their AI use statement in more detail.
  2. What do you mean when you state that the reference vectors are centered? Why is this needed? (Figure 1)
  3. How robust is EER when evaluated on substantially larger pretrained or pretrained-then-finetuned models? In particular, does the EER gain remain meaningful as model scale and training token budget increase?
4 comments
AuthorsOpenReview ↗

Response to Reviewer X4Tj (Part 1/2)

Thank you for the review and the clear questions. Results not in the submission are marked [new]; they are in the revised PDF (Section 4.6 and Appendix E).

W1 (a). Does EER raise the own-embedding projection, and how does it relate to the implicit route?

We agree that the submission put the observation and the method side by side without testing the link in the same models. We have now tested it directly, in (a) and (b) below.

The link we have in mind is this. Models raise the embedding's weight through layers that see the embedding only inside a normalized mix, in which it keeps fading (Section 2). EER supplies the embedding itself, with a weight chosen from the current state, which changes exactly the quantity the implicit route is adjusting (Equation 5), and it keeps the embedding visible to the layers that follow.

We measured this on our Large Baseline and EER [new] (seed 42, 256 held-out documents). The Baseline behaves like the pretrained models: its projection coefficient onto the token's own embedding grows from 1 to about 10. Its embedding, however, fades from the state, while with EER it does not. Cosine between the state and the token's own embedding:

After block181624
Baseline0.2620.0950.0640.034
EER0.5430.3080.3690.223

With EER, the projection coefficient is much higher, and the sublayers write more along the embedding, not less:

Projection coefficient after the final blockBaselineEER
Written by all attention and MLP outputs9.018.5
From the explicit path (one plus the sum of injections)1.035.1
Total10.053.6

Within the trained EER model, switching off only the largest injections (blocks 14–19) makes the MLPs in blocks 17–22 write 12–39% less along the embedding, measured before each block's own injection. The explicit path therefore raises the projection both directly and by keeping the embedding visible to deeper layers, whose MLPs then write more along it. This analysis is in the new Section 4.6 and Figure 5.

W1 (b). Does changing the own-embedding projection change the loss?

Yes, and the projection set by EER's controller gives the lowest loss among the alternatives we tested. Raising the projection is not enough by itself: a fixed coefficient of one also raises it at every block but gives only 0.007 nats. What matters is where and for which tokens it is raised, which is what the state-dependent controller sets. Across separately trained models that differ only in how the added weight is set (Table 3a, three seeds):

Coefficient ruleLoss reduction (nats)
Fixed coefficient of one0.007
Learned per-layer scalar0.009
Token-only controller0.014
State-dependent (EER)0.025

Within a single trained model [new] (Appendix E.8), we changed the injection coefficients of the Large EER model without retraining. Moving them away from what the controller chose, in either direction, raises the loss:

Change to the injection coefficientsFineWeb-EduWikiTextC4
Multiply all by 0.5+0.042+0.054+0.047
Multiply all by 2+0.040+0.059+0.055
Replace by the per-block mean+0.178+0.195+0.199
Replace by a per-token, per-block mean+0.134+0.147+0.154
Remove the block-1 injection only+0.034+0.034+0.029

Values are loss increases in nats. Such a change takes the model off its training distribution, which is why we read this table next to Table 3a and not by itself.

In short, EER raises the own-embedding projection, keeps the embedding visible to deeper layers, and the projection it chooses gives the lowest loss among the alternatives we tested. The revised Section 4.6 now makes this link explicit.

AuthorsOpenReview ↗

Response to Reviewer X4Tj (Part 2/2)

W2 / Q3. Scale, vocabulary share, token budget, pretrained models

Larger runs are the most direct test, so since the submission we have trained a larger pair, 1.45B parameters (982M non-embedding) on 78.5B tokens, with the same protocol [new] (Appendix E.1). EER lowers validation loss from 2.286 to 2.264 and raises the seven-task zero-shot average from 64.28% to 65.50% (+1.22 points), for 2.8% more training GPU-hours. This pair has one seed so far; the other two seeds are running, and we will update the results here once they finish. Across the four scales:

ScaleNon-embedding paramsTraining tokensEmbedding and head shareReduction (nats)
Small122.7M9.8B66%0.0250
Medium272.7M21.8B53%0.0217
Large511.2M40.9B43%0.0249
1.45B [new]982.0M78.5B32%0.0221
  • Small-model effect. The gain stays at 0.022–0.025 nats while non-embedding parameters and training tokens both grow 8×, so it is not concentrated in the smallest models. EER also differs from a plain embedding skip: re-adding the embedding with a learned per-layer scalar gives only 0.009 nats even at Small scale, where such skips should help most, against 0.025 for EER (Table 3a).
  • Vocabulary share. EER adds two width-sized vectors per block and no token-indexed table, so a 152k vocabulary gives it no extra capacity. In the table above, the gain stays in the same range as the share of the input embedding and output head falls from 66% to 32%.
  • Token budget. Every run uses about 80 tokens per non-embedding parameter, four times the compute-optimal ratio (Hoffmann et al., 2022), so our models are trained well past the point around which scaling-law studies are usually run. EER is ahead at every one of the 10, 22, and 41 intermediate and final evaluations at Small, Medium, and Large (Appendix B.1), and at Large the gap is 0.0255 nats at 61% of the training budget and 0.0249 at the end [new]. We have not trained at higher token-to-parameter ratios.
  • Modern scale. Figure 1 is measured on 7B–32B models, so the implicit reweighting behind EER is not a small-model effect.
  • Pretrained models. We also tried adding EER to a pretrained model [new] (Appendix E.9). We inserted EER into Qwen3-8B-Base and continued training for 2B tokens; the loss shows no consistent difference from continued training of the unchanged model on the same tokens (within 0.005 nats), and inserting iHC in the same way gives the same result. We read this as a property of modifying a pretrained model rather than of EER: after pretraining on far more tokens, the model has settled how it uses its residual stream, and 2B tokens appear to be too few for it to make use of a new path. Like iHC, EER is meant to be part of the architecture from the start of training. This is the setting of all our other results, and there its gain stays in the same range up to 1.45B parameters.

Q1. AI use statement

We apologize for the overly brief statement. We have replaced it in the revised PDF with the following text.

We used OpenAI Codex, an LLM-based assistant, in three ways: (i) to check the training and evaluation code for errors; (ii) to write an initial draft of the manuscript, which we then revised; and (iii) after our revision, to check grammar and fluency of wording. The research idea, method, experimental design, and interpretation of results are our own. We reviewed and verified all AI-assisted text and code and take full responsibility for the content.

Q2. What does "centered" mean in Figure 1, and why is it needed?

Centering subtracts the mean embedding from the reference embeddings we project onto; the hidden states are unchanged. We use it because of one model, Qwen3-8B-Base, whose hidden states have a large component along this shared mean direction. That component raises the projection onto every token's embedding, including the control, and centering removes it so that the figure shows what is specific to each token's own embedding (Appendix C gives the numbers).

For the other five models centering makes little difference. Figure 7 of the revised PDF (Figure 6 in the submission) shows the curves without centering: the control stays near zero in those five models, and in all six models the projection onto the token's own embedding stays above the control at every block (Figure 6 of the revised PDF). The finding therefore does not depend on centering, and the Figure 1 caption now points to the uncentered curves.

Thank you again for your time. We hope these results address your main concerns, in particular the link between the observation and the method, and that you will take them into account in your final assessment. We would be glad to clarify anything further.

The review addressed my concerns. I will raise the score from 2 to 3.

Thank you for reading our response and for raising your score, we really appreciate it. Your point about connecting the observation to the method pushed us to run the analysis that is now Section 4.6, and we think the paper is clearly better for it. In the final version we will move this into the main text, along with the clarification about centering and the updated AI use statement.

Reviewer_AshKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Explicit Embedding Reweighting (EER), a lightweight modification to Transformer residual connections that dynamically injects the original token embedding with a state-dependent coefficient. Motivated by the observation that pretrained Transformers implicitly increase the projection of hidden states onto their original embeddings, EER achieves consistent improvements in language modeling performance with small parameter overhead.

Critical strengths
  1. The proposed method is simple, intuitive, and straightforward to implement, with very few additional parameters.

  2. Solid empirical evaluation, including analysis of models from different families, training at multiple model scales, comparisons against alternative residual architectures, and well-designed ablations.

  3. Consistent and meaningful improvements over the standard Transformer baseline, with competitive performance and better training efficiency than Attention Residuals.

Critical weaknesses
  1. Limited improvements over stronger residual architectures. Although EER substantially improves over the standard Transformer baseline, the improvements over alternative residual designs are relatively modest, particularly when combined with iHC.

  2. Limited scalability evaluation. Experiments only cover models up to 900M parameters, leaving it unclear whether the improvements generalize to larger-scale LLMs.

  3. (minor) Mechanistic justification. While the observed increase in projection onto the original token embedding is interesting, the paper does not fully establish why this behavior is beneficial or whether it directly explains the improvements from EER.

Questions for authors
  1. Does EER maintain its performance and efficiency advantages at larger model scales?

  2. How does EER affect inference-time latency and throughput, beyond the reported training overhead?

3 comments
AuthorsOpenReview ↗

Response to Reviewer AshK (Part 1/2)

Thank you for the encouraging review and the constructive questions. Results not in the submission are marked [new]; they are in the revised PDF (Section 4.6 and Appendix E).

W1. Limited improvement over stronger residual designs, especially with iHC

The gain on top of the stronger designs is indeed smaller than on the plain Baseline, most clearly on iHC. We see EER as useful in two roles: on its own, as a much cheaper alternative to these designs, and on top of them. The Large results of Table 2 (three-seed means; bold marks the better perplexity and accuracy within each group):

LargePPLAvg. accuracy (%)GPU-hours
Baseline11.4759.36310.63
+ AttnRes11.2460.44387.26 (+24.7%)
+ EER11.1960.75322.36 (+3.8%)
SATFormer11.3859.84314.42
SATFormer + EER11.1560.76328.59 (+4.5%)
iHC11.0460.92372.30
iHC + EER10.9661.16389.54 (+4.6%)
  • Alone, EER does slightly better than AttnRes at about one sixth of its extra training cost. It also obtains about two thirds of iHC's loss reduction over Baseline (0.025 of 0.038 nats) at about a fifth of iHC's extra cost (3.8% against 19.9%).
  • On SATFormer, most of the gain survives: 0.020 of the 0.025 nats EER gives on the plain Baseline, and all seven tasks improve.
  • On iHC, the gain is relatively smaller, as you note, but consistent: perplexity drops by 0.7% at Large (11.04 ± 0.02 to 10.96 ± 0.02 over three seeds) and by 0.8% at Medium (12.76 to 12.67) [new]. A four-stream design probably already keeps more of the early information that EER supplies (Section 4.3).

In short, EER is both a cheap alternative to these designs and a complement to them, at under 5% extra training time in either role.

W2 / Q1. Do the performance and efficiency advantages hold at larger scale?

To test this directly, we have since trained a larger pair, 1.45B parameters (982M non-embedding) on 78.5B tokens, with the same protocol [new] (Appendix E.1). Both advantages hold at this scale:

ScaleNon-embedding paramsBaseline lossEER lossReduction (nats)Training overhead
Small122.7M2.7892.7640.0250+4.2%
Medium272.7M2.5812.5590.0217+3.5%
Large511.2M2.4402.4150.0249+3.8%
1.45B [new]982.0M2.2862.2640.0221+2.8%

Performance. The gain stays at 0.022–0.025 nats across the four scales. At 1.45B, EER also raises the seven-task zero-shot average from 64.28% to 65.50% (+1.22 points):

1.45B (one seed), accuracy (%)LAMBADAPIQAHellaSwagWinoGrandeARC-EARC-CSciQAverage
Baseline50.6974.0560.1159.3573.2340.7091.8064.28
EER51.2573.9961.2661.5674.7942.2493.4065.50

Six of the seven tasks improve, and PIQA is essentially unchanged (73.99 against 74.05). The 1.45B pair has one seed so far; the other two seeds are running, and we will update the results here once they finish.

Efficiency. The training overhead is lowest at 1.45B: 2.8% (1,126 against 1,096 GPU-hours), against 3.5–4.2% at the three smaller scales. This fits the design of EER, which adds two width-sized vectors per block, so its share of parameters and FLOPs falls as the model gets wider: from 0.020% to 0.012% of non-embedding parameters and from 0.018% to 0.015% of forward FLOPs between Small and Large (Table 1).

We would add that the phenomenon EER builds on is not a small-model effect, since Figure 1 is measured on 7B–32B production models. Testing EER at that scale is a natural next step.

AuthorsOpenReview ↗

Response to Reviewer AshK (Part 2/2)

W3 (minor). Why is the growing projection beneficial, and does it explain the gain?

The submission indeed left this link implicit. Our account is that the implicit route works from an input in which the embedding keeps fading, and EER gives the model a direct handle on the embedding's weight, alongside the implicit route rather than in place of it. We now have measurements for each step of this account [new], on our Large Baseline and EER (seed 42, 256 held-out documents).

The embedding fades, in pretrained models and in our own Baseline. After the first sublayer, a layer only sees a normalized mix of the embedding and the outputs of earlier layers. In the six models of Figure 1, the cosine between state and embedding falls from 0.13–0.58 after the first block to 0.008–0.044 after the last. Our Baseline shows the same, while with EER the embedding stays visible. Cosine between the state and the token's own embedding:

After block181624
Baseline0.2620.0950.0640.034
EER0.5430.3080.3690.223

With the embedding visible, the sublayers write more along it. Their contribution to the projection coefficient at the output is 18.5 with EER against 9.0 in the Baseline, on top of 35.1 from the explicit path. Switching off only the largest injections (blocks 14–19) in the trained EER model makes the MLPs in blocks 17–22 write 12–39% less along the embedding, measured before each block's own injection, so the extra writing depends on the explicit path.

The weight matters for prediction. Across separately trained models, the coefficient rule alone moves the gain from 0.007 nats (fixed) to 0.025 (state-dependent) (Table 3a). Within the trained EER model, halving or doubling all injection coefficients costs 0.04–0.06 nats, and replacing them with per-token constants costs 0.13–0.15 nats; since such changes take the model off its training distribution, we read them together with Table 3a.

So the explicit path does not replace the implicit route: it keeps the embedding visible to deeper layers, and their MLPs then write more along it. These results are in the new Section 4.6 and Figure 5.

Q2. Inference latency and throughput

These are now in Appendix E.6 [new]. We measured both on the trained Large checkpoints, on a single GPU with BF16 autocast and torch.compile, batch size 1, a 2,048-token prompt followed by 128 decoded tokens, taking the median of six timed runs:

Large, torch.compileBaselineEEROverhead
Prefill, 2,048 tokens (ms)14.2314.45+1.5%
Prefill with KV-cache construction (ms)14.3914.60+1.5%
Decoding with KV cache (ms per token)7.277.33+0.8%

Decoding throughput goes from 137.5 to 136.4 tokens/s. EER leaves the KV cache unchanged (Section 3.2) and adds no matrix product: each module is one pass over the residual state, a softplus, and a scaled addition, and all modules together add 0.015% of forward FLOPs at Large (Table 1).

Thank you again for your time and the helpful suggestions. We hope the 1.45B results and the inference measurements answer your two questions, and we would be glad to clarify anything further.

Dear Reviewer AshK,

Just a quick follow-up before the discussion period ends. On your two questions: the gain holds at 1.45B (0.022 nats lower loss, +1.22 points zero-shot) with a slightly lower training overhead of 2.8%, and the inference overhead is about 1–1.5%. We’d be happy to answer anything else if helpful.

Reviewer_2vSeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EER, a simple add-on that puts the original input embedding back into the model’s residual stream after every MLP update, using a positive coefficient that’s predicted from the current hidden state. The authors train models from scratch and scale up to 900 million parameters and report consistently lower validation loss, all with very few extra parameters. EER performs about as well as Attention Residuals but is cheaper to train. It also further improves SATFormer and identity Hyper-Connections when added to them. The experiments also show that letting the coefficient depend on the current state beats using a fixed value or simpler alternatives.

Critical strengths
  1. EER is straightforward to implement and introduces only a small number of additional parameters. Despite its simplicity, it consistently reduces validation loss across all tested model sizes and achieves performance comparable to Attention Residuals, while incurring lower training costs.
  2. The paper further observes that in six pretrained LLMs, each token’s residual state accumulates a growing component along its own embedding across layers, much larger than along the embeddings of other tokens. This finding is supported by control tests and mean-removal analyses.
  3. The experiments are rigorously controlled. The primary comparisons and coefficient tests are averaged over three random seeds, and the study directly compares the use of a state-dependent coefficient with fixed, per-layer, and token-only rules at equivalent locations in the model.
Critical weaknesses
  1. Although the observation is sound, the motivation for explicit injection remains unclear. The paper explains the effect of injection on the normalized input but does not clarify why an explicit path is necessary when pretrained models already achieve this reweighting implicitly. The specific cost or limitation of the indirect route is not analyzed.
  2. As a follow-up, the paper does not empirically test the connection between its observation and the proposed method. While it demonstrates that pretrained large language models amplify the embedding component, it does not measure this effect in its own Baseline and EER models, such as whether EER reduces the embedding component added by sublayers. The observed improvement may also result from re-exposing token identity to deeper layers, as seen in value embeddings or input-skip designs.
  3. The reported improvements are modest, and there is limited evidence of generalizability. Gains are approximately 0.025 nats for models up to 900 million parameters and around 80 tokens per parameter. Some differences, such as the accuracy gain on identity Hyper-Connections and the insertion-site ablations, are comparable to the seed variation observed elsewhere in the paper, and some tables omit standard deviations. The experiments do not include comparisons to depth-aggregation methods that also weight the embedding, such as DenseFormer and MUDDFormer.
  4. The efficiency claim is not fully substantiated. Although the paper emphasizes that EER adds very few parameters and floating-point operations, training time increases by 3.8% at the Large scale, and inference latency or throughput is not reported. Given that practicality is a primary advantage of EER, the actual wall-clock cost for both training and inference should be provided.
Questions for authors

Most of my concerns are included in the weakness section. Specifically, I would like to ask the authors:

  1. Given that pretrained models achieve embedding reweighting implicitly, could the authors clarify the specific cost or limitation associated with the indirect approach, and explain how the explicit path introduced by EER addresses this issue?
  2. Have the authors measured the projection coefficient α\alpha in their own Baseline and EER models? Does EER demonstrably reduce the embedding component written by sublayers, and how do the authors distinguish their explanation from the possibility of merely re-exposing token identity to deeper layers, as seen in value embeddings or input-skip architectures?
  3. Are the results presented in Tables 1, 3b, and 7 averaged across three seeds? If so, could the authors report the corresponding standard deviations?
  4. What are the inference latency and throughput overheads associated with EER, and to what extent could the reported 3.8% training overhead be mitigated through the use of fused kernels?
  5. In the Small-scale Add/Total comparison, the Add rule yields a value of 2.766, whereas the same configuration yields 2.764 in Table 3a. Could the authors clarify the origin of this discrepancy?
  6. Table 6 presents results for SATFormer at Medium scale only. Were iHC or AttnRes combinations also evaluated at this scale, and if so, what were the corresponding outcomes?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 2vSe (Part 1/5)

Thank you for the careful and detailed review. Your questions on how the observation connects to the method were especially helpful and led us to several new analyses. Results not in the submission are marked [new]; they are in the revised PDF (Section 4.6 and Appendix E).

Summary. We now measure the link between the observation and the method in our own models (new Section 4.6 and Figure 5). In our Baseline, as in the pretrained models, the token's embedding fades from the state with depth; with EER it stays visible, and the deeper sublayers write more along it, not less. Re-adding the embedding with a uniform or token-only weight reaches only a third to a half of EER's gain. New results include a 1.45B pair (0.0221 nats, one seed so far), comparisons with DenseFormer and MUDDFormer, the standard deviations you asked for, and inference latency (+1.5% for prefill and +0.8% for decoding with torch.compile).

W1 / Q1. Models already reweight the embedding implicitly. Why add an explicit path, and what does the indirect route cost?

We agree that the submission did not explain this clearly. In short, the sublayers have to adjust the embedding's weight while seeing the embedding only as a fading part of their input. EER keeps the embedding directly available, and the sublayers then act on it more. The explicit path works alongside the implicit route rather than replacing it.

The implicit route faces two difficulties.

  • The embedding fades from what the sublayers read. After the first sublayer, a layer sees a normalized mix of the embedding and the outputs of all earlier layers (Section 2), and the embedding becomes a smaller and smaller part of that mix. In the six pretrained models of Figure 1, the cosine between state and embedding falls from 0.13–0.58 after the first block to 0.008–0.044 after the last, even though the projection coefficient grows over most of the depth [new]. Our own Large Baseline shows the same, from 0.26 to 0.034.
  • The sublayers have to find the embedding direction themselves. To raise the embedding's weight for a token, a layer must write along that token's own embedding, a direction that differs for each of the 152k tokens, and it has to recover this direction from the faded mix with the same weights that also produce contextual features.

EER eases both. It adds the token's embedding directly, with a weight read from the current state; by Equation 5, this changes exactly the quantity the implicit route adjusts, the size of the embedding relative to the accumulated updates. The injections also keep the embedding prominent in what deeper layers read (cosine 0.22 at the output, against 0.034 in the Baseline), so the sublayers no longer have to work from a faded direction. As W2/Q2 shows, they then write more along the embedding, not less. This is the cost of relying on the implicit route alone: without an explicit path, the embedding becomes very faint in the last layers.

AuthorsOpenReview ↗

Response to Reviewer 2vSe (Part 2/5)

W2 / Q2 (b). Does EER reduce the embedding component written by the sublayers?

No, and this was the most informative result for us. The explicit path could have taken over the adjustment from the sublayers. Instead, by the final block they write twice as much along the embedding as in the Baseline, on top of the explicit path:

Projection coefficient after the final blockBaselineEER
Written by all attention and MLP outputs9.018.5
From the explicit path (one plus the sum of injections)1.035.1
Total10.053.6

Across depth, the two paths share the work (Section 4.6 and Figure 5):

  • Early blocks. The embedding is still prominent in both models, and the explicit path takes over part of the adjustment: with EER the sublayers write less than in the Baseline (1.21 against 1.90 after block 5).
  • Deeper blocks. From block 11 on, the embedding fades in the Baseline (cosine 0.08 or less) but stays visible with EER (0.22–0.38), and there the sublayers of EER write more along it. The largest increases are in blocks 17–22, which follow the largest injections (blocks 14–19).

Is the extra writing caused by the explicit path? Two separately trained models cannot show this, so we tested it within one model [new]. In the trained Large EER model, we switched off only the injections in blocks 14–19 and measured what the sublayers in blocks 17–22 write along the token's embedding, on the same 256 documents and before each block's own injection.

The MLP outputs write less in all six blocks:

Block171819202122
Change in MLP writing along the embedding−39%−37%−25%−20%−12%−12%

The drop is largest in the blocks closest to the removed injections, and the 95% bootstrap interval of the change is below zero in every block. The changes in attention are smaller and differ in sign across blocks. This fits the usual division of labor: attention mostly gathers information from other positions, while the MLP works position by position and is often described as a key-value memory (Geva et al., 2021), so its output depends more directly on how much of the token's embedding is in its input.

So the explicit path does not take over from the sublayers: by keeping the embedding visible, it lets the deeper MLPs act on it more. This test changes the trained model at inference time; it is not a model retrained without these injections.

W2 / Q2 (c). Does the embedding's weight matter for prediction?

Yes; moving it in either direction hurts [new] (Appendix E.8). We changed the injection coefficients of the trained Large EER model without retraining:

Change to the injection coefficientsFineWeb-EduWikiTextC4
Multiply all by 0.5+0.042+0.054+0.047
Multiply all by 2+0.040+0.059+0.055
Replace by the per-block mean+0.178+0.195+0.199
Replace by a per-token, per-block mean+0.134+0.147+0.154
Remove the block-1 injection only+0.034+0.034+0.029

Values are loss increases in nats. After such a change the model is off its training distribution, so we read this table together with Table 3a. There, separately trained models that differ only in the coefficient rule show the same ordering: state-dependent coefficients are best, per-token ones next, and a single coefficient per layer worst.

AuthorsOpenReview ↗

Response to Reviewer 2vSe (Part 3/5)

W2 / Q2 (d). Is the gain just re-exposing token identity, as in value embeddings or input skips?

Partly. EER does re-expose the token's embedding, but re-exposure explains only part of the gain. Table 3a keeps the insertion sites fixed and changes only how the embedding's weight is set:

How the embedding is re-addedLoss reduction (nats)Share of EER's gain
Fixed coefficient of one0.00728%
Learned per-layer scalar (a plain input skip)0.00936%
Token-only controller (EER's parameter count)0.01456%
State-dependent (EER)0.025100%

Uniform re-exposure thus reaches about a third of EER's gain and token-dependent re-exposure about half. The most direct comparison is the last two rows: with the same embedding, insertion sites, and parameter count, reading the weight from the current state instead of the token lowers the loss from 2.775 ± 0.003 to 2.764 ± 0.002. Likewise, in the trained EER model, replacing the coefficients by per-token constants costs 0.13–0.15 nats (table above).

Value-style re-exposure. SATFormer [1], which we compare against, is a gated form of Value Residual Learning (Section 5): later attention layers reuse the values of the first layer. These values depend only on the input tokens, so they carry token identity into deeper layers, as value embeddings do. EER alone does better than SATFormer (perplexity 11.19 against 11.38, Table 2), and adding EER to SATFormer still helps (11.38 to 11.15 at Large, three seeds; 0.0116 nats at Medium, Table 6). If the two did the same job, the combination would gain little. The revised Related Work now states this connection.

[1] SATFormer: Learning When to Use Early Value Residuals in Transformers.

W3 (a). The improvements are modest

Measured against scaling, the gain is sizeable for its cost. Going from Medium to Large, with 1.9× the parameters and 1.9× the tokens, lowers Baseline loss by 0.141 nats (Table 1); EER gives about 18% of that step for 3.8% more GPU-hours. Among the residual designs in Table 2, AttnRes reaches perplexity 11.24 for 25% more GPU-hours and iHC 11.04 for 20% more; EER reaches 11.19 for 3.8% more.

W3 (b). Scale and generalization

Within the paper's range the gain does not decline: 0.0250, 0.0217, and 0.0249 nats while parameters and tokens both grow 4.2×. Since the submission we have also trained a larger pair, 1.45B parameters (982M non-embedding) on 78.5B tokens, with the same protocol [new] (Appendix E.1). EER lowers validation loss from 2.286 to 2.264 (0.0221 nats) and raises the seven-task zero-shot average from 64.28% to 65.50%. This pair has one seed so far; the other two seeds are running, and we will update the results here once they finish.

On the training budget: every run uses about 80 tokens per non-embedding parameter. This is four times the compute-optimal ratio of Hoffmann et al. (2022), around which scaling-law studies are usually run, so our models are trained well past that point. EER is ahead at every one of the 10, 22, and 41 intermediate and final evaluations at Small, Medium, and Large (Appendix B.1), and at Large the gap is 0.0255 nats at 61% of the training budget and 0.0249 at the end [new].

Downstream, EER improves the seven-task zero-shot average by 1.39 points at Large (Table 2, three seeds). It also lowers perplexity on corpora outside the training data [new] (Appendix E.7):

CorpusBaseline PPLEER PPLReduction
WikiText13.0712.454.7%
C419.9819.492.5%

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful and constructive feedback. We are glad that they found EER simple to implement and its gains consistent (2vSe, AshK), the experiments well controlled (2vSe, AshK), and the motivation clear and well presented (X4Tj). Several questions were raised by more than one reviewer, so we summarize the main new results here. Each individual response is self-contained and marks results not in the submission as [new]. All changes in the revised PDF are shown in dark red.

1. The link between the observation and the method (2vSe, AshK, X4Tj). We repeated the Section 2 measurement on our own Large Baseline and EER models (new Section 4.6 and Figure 5).

  • Our Baseline shows the same pattern as the pretrained models in Figure 1, and its embedding is diluted over depth: the cosine between state and embedding falls from 0.26 after the first block to 0.034 at the output. With EER the embedding stays visible, with a cosine of 0.22 at the output.
  • With the embedding visible, the sublayers act on it more, not less: at the output, attention and MLP outputs write twice as much along the embedding as in the Baseline (18.5 against 9.0), on top of the explicit path.
  • Switching off the largest injections (blocks 14–19) in the trained EER model makes the MLPs in blocks 17–22 write 12–39% less along the embedding.

The explicit path therefore works together with the implicit route: it keeps the embedding visible to deeper layers, and their MLPs then write more along it.

2. Scale (2vSe, AshK, X4Tj). We trained a larger pair, 1.45B parameters (982M non-embedding) on 78.5B tokens, with the same protocol. EER lowers validation loss from 2.286 to 2.264 and raises the seven-task zero-shot average by 1.22 points. The reduction, 0.0221 nats, is in line with 0.0250, 0.0217, and 0.0249 at the three scales in the paper. This pair has one seed so far. At the three smaller scales the standard deviation across seeds is 0.001–0.003 nats, about a tenth of the gain; the other two seeds are running, and we will update the results here once they finish.

3. Cost (2vSe, AshK). At 1.45B, EER adds 2.8% to training GPU-hours (1,126 against 1,096). At Large, where we compare with other residual designs, it adds 3.8%, against 25% for AttnRes and 20% for iHC. With torch.compile, inference latency on the Large checkpoints rises by 1.5% for prefill and 0.8% per decoded token.

4. Other additions. Following the reviewers' suggestions, the revised PDF also adds comparisons with DenseFormer and MUDDFormer, the standard deviations and seed counts that were requested, a three-seed Add/Total comparison, iHC with EER at Medium, perplexity on WikiText and C4, and tests that change the coefficients of the trained model. The mechanism analysis is in Section 4.6; all other new results are collected in Appendix E, and we will integrate them into the relevant sections in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.