- Rating
- 3(+1)
- Confidence
- 2(+1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes using N-gram embeddings as a source of token-level control signals for autoregressive Transformers. A learned certainty score jointly controls attention-output gating, input-embedding reinjection, attention span, and locality bias.
Critical strengths
-
Repurposing N-gram embeddings as token-level control signals is an interesting idea that could make local contextual information useful for guiding subsequent Transformer processing.
-
The experiments cover multiple model scales and relevant architectural baselines, providing useful evidence of the method’s effectiveness across the evaluated settings.
-
The context-removal probes and signal-replacement ablations provide useful diagnostics for understanding what the learned control signal captures and how it affects model behavior.
Critical weaknesses
My biggest concern is that the method introduces numerous named components and control-inspired equations, but their necessity and relationship to the central hypothesis remain unclear. For example, why are both span control and a separate distance penalty needed? Why should higher certainty favor reinjection of the initial embedding? What motivates accumulated deviations and cross-layer differences in the feedback update? These choices are specified operationally, but their motivations are not developed into a coherent explanation. Consequently, the framework reads as a collection of heuristics, making its essential contribution difficult to identify.
Questions for authors
See weakness.
7 comments
Part 2. Why both a span and a distance penalty? They control reach and shape.
2. Why both a span and a distance penalty? They control reach and shape.
Section 3.3 introduces them as two controls on different aspects of attention, and Appendix A.7 gives the design rationale: SpanCtrl penalizes distance relative to a chosen span, while LocalBias changes the distance preference throughout the causal prefix, adapting ALiBi's per-head slope to certainty. Concretely, SpanCtrl sets, per head and per token, a distance beyond which a steep penalty applies (Eq. 13), and LocalBias decides how strongly nearer keys are preferred within that reach. One mechanism cannot express both "attend roughly evenly over the last 200 tokens" and "strongly prefer the last 10 tokens but still allow 200". Only the span also gives each query a window, which deployment can round to a few fixed sizes and run with efficient windowed attention (Appendix A.5). Each is needed given the other: removing SpanCtrl costs +0.21 PPL and removing LocalBias costs +0.16 (Table 9).
Part 3. Why should higher certainty favor re-injecting the initial embedding?
3. Why should higher certainty favor re-injecting the initial embedding? Because a locally resolved token's input already holds much of what it needs, and depth dilutes it.
Section 3.2 gives the roles: recall provides another path from the fixed enriched input, the gate decides how much newly computed attention output is added to that recalled state, and the learned recall coefficient can preserve more of the input at high-certainty positions. Section 4.3 ("What changes inside the residual stream?") and Appendix C.5 measure the dilution that recall counters. In the 1.3B backbone, the hidden state's cosine with the input falls from 0.84 after the first block to 0.02 after block 32. ResidualGate alone raises the final value to 0.08, adding recall raises it to 0.21, and the full method reaches 0.19. Appendix A.12.5 explains why the recall rate shrinks with depth. For a low-certainty token, the missing information has to come from context, so keeping the input matters less and a larger attention write matters more. The direction is learned, not imposed: the certainty weight in recall starts at zero, so recall is initially identical for every certainty value (Appendix A.4), and training makes recall increase with certainty in 82% of layer entries. Recall restores the complete input embedding, which includes more than the N-gram branch that certainty is read from, and the FFN still transforms the recalled state, so recall preserves access to the input without freezing the token.
Part 1. The hypothesis and the contribution
Thank you for the review. Below we answer your question on how the components relate to the central hypothesis and whether each is necessary, and the three specific questions on the span and distance penalty, recall at high certainty, and the feedback terms. Table numbers refer to the revised PDF, where new and changed text is in dark red.
1. The hypothesis and the contribution
Section 1 (third paragraph) frames the method as two per-token decisions. Along depth, the model decides how strongly attention should rewrite a token's representation. Along the sequence, it decides how much distant context that attention should consider. A locally predictable position benefits from a smaller attention write and a stronger locality preference, and an uncertain position from broader access. Section 3.1 defines N-gram certainty as the estimate that drives these decisions: it is read once from the N-gram branch before any attention, is designed to be high at locally resolvable positions, and is trained through the overall objective (the language-modeling loss plus the span-budget and gate penalties). Figure 1 shows the intended responses for high- and low-certainty tokens.
The contribution, stated in Section 1, is to turn the N-gram branch into a shared control source for all four actuators, and to show that its estimate predicts sensitivity to distant context: cutting the context to 128 tokens raises the loss by 0.39 nats for the least certain fifth of tokens and by 0.05 for the most certain fifth (Section 4.2, Table 5). Each actuator on its own has precedent (gated residual writes, adaptive spans, ALiBi-style slopes), and Appendix B (Table 27) compares signal sources and insertion points with that prior work.
The table collects, for each component (the four actuators and feedback, which the paper calls predictive-reactive certainty refinement), the per-token decision it makes (Sections 3.2 to 3.4), its learned direction in the three 1.3B full-method runs (raw parameters averaged over seeds, shares pooled over layers or heads and seeds), and its leave-one-out cost at 350M (Table 9).
λ_c= 0.5λ_c= 0.68, positive in 97% of layer entriesw_α= 0 (neutral)w_α= 0.14, positive in 82% of layer entriesβ_1= 0 (neutral)β_1= 0.07, positive in 71% of head entriesw_c= 0.5w_c= 0.82, positive in 3 of 3 seedsThe trained parameters follow the hypothesized directions. The two certainty weights that start at zero become positive on average, in 82% of LocalRecall's layer entries and in 71% of LocalBias's head entries, the weakest dependence. ResidualGate keeps its initialized sign in 97% of layer entries, and SpanCtrl's direction, set by Eq. 5, holds in every head because the learned maximum span stays above the minimum. In the perturbation study of Table 21, one added unit of certainty shrinks the attention write by 0.124 and raises recall by 0.087. These weights give directions, and point 5 measures how much PPL depends on the signal.