acceptodds
Under review as a conference paper at ICLR 2027

Context-Demand-Aware Control for N-gram Transformers

Abstract

N-gram embeddings encode local context before attention, but using them only as input features leaves their potential to guide later computation unexplored. We introduce context-demand-aware control for N-gram Transformers, repurposing the N-gram branch as a token-level runtime control signal. A learned certainty score coordinates attention-output gating and embedding recall along depth with attention span and locality bias along the sequence. A shared predictive-reactive controller updates this score using layerwise attention statistics. At 1.3B parameters, the full model reduces held-out perplexity from to across three seeds, a improvement at training-time overhead. A context-removal probe links low certainty to greater dependence on distant context, with the contrast persisting after matching token frequency. Matched-data comparisons and signal-replacement ablations further separate the control signal's contribution from the enriched input representation. Single-seed results at 2.8B and 3.93B support the observed ordering. At 1.3B, optional certainty-driven skipping yields a measured speedup for a PPL increase. These results support using an existing N-gram branch to coordinate computation, within the evaluated N-gram architectures and language-modeling setting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
N-gram embeddings, N-gram Transformers, language modeling, adaptive computation, residual gating, adaptive attention span, attention feedback control, layer skipping

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KTAdOpenReview ↗
Rating
3(+1)
Confidence
2(+1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes using N-gram embeddings as a source of token-level control signals for autoregressive Transformers. A learned certainty score jointly controls attention-output gating, input-embedding reinjection, attention span, and locality bias.

Critical strengths
  1. Repurposing N-gram embeddings as token-level control signals is an interesting idea that could make local contextual information useful for guiding subsequent Transformer processing.

  2. The experiments cover multiple model scales and relevant architectural baselines, providing useful evidence of the method’s effectiveness across the evaluated settings.

  3. The context-removal probes and signal-replacement ablations provide useful diagnostics for understanding what the learned control signal captures and how it affects model behavior.

Critical weaknesses

My biggest concern is that the method introduces numerous named components and control-inspired equations, but their necessity and relationship to the central hypothesis remain unclear. For example, why are both span control and a separate distance penalty needed? Why should higher certainty favor reinjection of the initial embedding? What motivates accumulated deviations and cross-layer differences in the feedback update? These choices are specified operationally, but their motivations are not developed into a coherent explanation. Consequently, the framework reads as a collection of heuristics, making its essential contribution difficult to identify.

Questions for authors

See weakness.

7 comments
AuthorsOpenReview ↗

Part 1. The hypothesis and the contribution

Thank you for the review. Below we answer your question on how the components relate to the central hypothesis and whether each is necessary, and the three specific questions on the span and distance penalty, recall at high certainty, and the feedback terms. Table numbers refer to the revised PDF, where new and changed text is in dark red.

1. The hypothesis and the contribution

Section 1 (third paragraph) frames the method as two per-token decisions. Along depth, the model decides how strongly attention should rewrite a token's representation. Along the sequence, it decides how much distant context that attention should consider. A locally predictable position benefits from a smaller attention write and a stronger locality preference, and an uncertain position from broader access. Section 3.1 defines N-gram certainty as the estimate that drives these decisions: it is read once from the N-gram branch before any attention, is designed to be high at locally resolvable positions, and is trained through the overall objective (the language-modeling loss plus the span-budget and gate penalties). Figure 1 shows the intended responses for high- and low-certainty tokens.

The contribution, stated in Section 1, is to turn the N-gram branch into a shared control source for all four actuators, and to show that its estimate predicts sensitivity to distant context: cutting the context to 128 tokens raises the loss by 0.39 nats for the least certain fifth of tokens and by 0.05 for the most certain fifth (Section 4.2, Table 5). Each actuator on its own has precedent (gated residual writes, adaptive spans, ALiBi-style slopes), and Appendix B (Table 27) compares signal sources and insertion points with that prior work.

The table collects, for each component (the four actuators and feedback, which the paper calls predictive-reactive certainty refinement), the per-token decision it makes (Sections 3.2 to 3.4), its learned direction in the three 1.3B full-method runs (raw parameters averaged over seeds, shares pooled over layers or heads and seeds), and its leave-one-out cost at 350M (Table 9).

Question for each tokenComponentEffect of higher certaintyInitial certainty weightLearned at 1.3BRemoval cost
How much should attention rewrite it?ResidualGatesmaller attention writeλ_c = 0.5λ_c = 0.68, positive in 97% of layer entries+0.39
Should it keep its local evidence?LocalRecallmore re-injection of the inputw_α = 0 (neutral)w_α = 0.14, positive in 82% of layer entries+0.28
How far may attention reach?SpanCtrlshorter spansign set by Eq. 5learned span range positive in every head+0.21
How strongly should nearer keys be preferred?LocalBiassteeper distance penaltyβ_1 = 0 (neutral)β_1 = 0.07, positive in 71% of head entries+0.16
Did attention confirm the estimate?Feedbackadjusts certainty for the next layerw_c = 0.5w_c = 0.82, positive in 3 of 3 seeds+0.11

The trained parameters follow the hypothesized directions. The two certainty weights that start at zero become positive on average, in 82% of LocalRecall's layer entries and in 71% of LocalBias's head entries, the weakest dependence. ResidualGate keeps its initialized sign in 97% of layer entries, and SpanCtrl's direction, set by Eq. 5, holds in every head because the learned maximum span stays above the minimum. In the perturbation study of Table 21, one added unit of certainty shrinks the attention write by 0.124 and raises recall by 0.087. These weights give directions, and point 5 measures how much PPL depends on the signal.

AuthorsOpenReview ↗

Part 2. Why both a span and a distance penalty? They control reach and shape.

2. Why both a span and a distance penalty? They control reach and shape.

Section 3.3 introduces them as two controls on different aspects of attention, and Appendix A.7 gives the design rationale: SpanCtrl penalizes distance relative to a chosen span, while LocalBias changes the distance preference throughout the causal prefix, adapting ALiBi's per-head slope to certainty. Concretely, SpanCtrl sets, per head and per token, a distance beyond which a steep penalty applies (Eq. 13), and LocalBias decides how strongly nearer keys are preferred within that reach. One mechanism cannot express both "attend roughly evenly over the last 200 tokens" and "strongly prefer the last 10 tokens but still allow 200". Only the span also gives each query a window, which deployment can round to a few fixed sizes and run with efficient windowed attention (Appendix A.5). Each is needed given the other: removing SpanCtrl costs +0.21 PPL and removing LocalBias costs +0.16 (Table 9).

AuthorsOpenReview ↗

Part 3. Why should higher certainty favor re-injecting the initial embedding?

3. Why should higher certainty favor re-injecting the initial embedding? Because a locally resolved token's input already holds much of what it needs, and depth dilutes it.

Section 3.2 gives the roles: recall provides another path from the fixed enriched input, the gate decides how much newly computed attention output is added to that recalled state, and the learned recall coefficient can preserve more of the input at high-certainty positions. Section 4.3 ("What changes inside the residual stream?") and Appendix C.5 measure the dilution that recall counters. In the 1.3B backbone, the hidden state's cosine with the input falls from 0.84 after the first block to 0.02 after block 32. ResidualGate alone raises the final value to 0.08, adding recall raises it to 0.21, and the full method reaches 0.19. Appendix A.12.5 explains why the recall rate shrinks with depth. For a low-certainty token, the missing information has to come from context, so keeping the input matters less and a larger attention write matters more. The direction is learned, not imposed: the certainty weight in recall starts at zero, so recall is initially identical for every certainty value (Appendix A.4), and training makes recall increase with certainty in 82% of layer entries. Recall restores the complete input embedding, which includes more than the N-gram branch that certainty is read from, and the FFN still transforms the recalled state, so recall preserves access to the input without freezing the token.

Reviewer_1tb9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes using the local information already available in an N-gram embedding branch to guide subsequent Transformer computation. A learned certainty signal coordinates attention writes, reuse of the initial embedding, attention span, and preference for nearby context. Attention statistics update this signal across layers. The main experiments show improved language modeling and downstream accuracy on N-gram Transformers

Critical strengths

Reusing an existing local representation to guide later computation is an appealing idea. It provides a relatively inexpensive way to coordinate decisions that are usually handled separately, namely, how much to update a representation and how broadly to attend. The context-removal results give this idea some empirical substance. Specifically, lower-certainty tokens are more affected by losing distant context, including after controlling for token frequency. If this relationship remains useful across workloads, it could offer a practical basis for adaptive computation without requiring an expensive additional router.

Critical weaknesses

To me, the benefit of the coordinated design is still difficult to separate from the benefit of its individual architectural choices. The complete method combines several mechanisms, auxiliary objectives, and a staged training schedule. The ablations show that components contribute, and the signal comparisons support using informative control inputs. However, they leave an important practical question unresolved: how much of the improvement requires the proposed token-dependent coordination? A matched version with learned but token-independent span and locality settings would help distinguish adaptive context allocation from the benefit of adding useful attention constraints. The paper acknowledges this missing comparison. It matters because adopting the full method involves several interacting choices, and the evidence does not yet establish how much of that complexity is necessary.

Questions for authors

At the reported 10.7% speedup operating point, what happens to downstream accuracy on tasks where the model performs meaningfully above chance, or to correctness in generated continuations? This would help determine whether the small perplexity penalty translates into an acceptable practical tradeoff.

3 comments
AuthorsOpenReview ↗

Part 1. How much of the gain needs token-dependent control? [Weakness]

Thank you for the careful review. Below we answer your two questions: how much of the gain needs token-dependent control, and what skipping costs at the 10.7% speedup point. Table numbers refer to the revised PDF, where new and changed text is in dark red.

1. How much of the gain needs token-dependent control? [Weakness]

All new arms reuse the 350M protocol: the same 20B tokens and data order, optimizer, schedule, curriculum, and auxiliary losses, seeds paired with the existing full-method runs, and the Table 8 scorer.

  • Static sequence control (the requested comparison). SpanCtrl replaces certainty in Eq. 5 with one learned constant per layer and head (a sigmoid of a parameter initialized at 0, so it starts at 0.5) and keeps its learned maximum span and budget loss. LocalBias drops its certainty term (β_1 = 0), so its slope is a learned constant per layer and head. ResidualGate, LocalRecall, and feedback (predictive-reactive certainty refinement) are unchanged.
  • Static depth control. The mirror image: ResidualGate and LocalRecall read a learned constant per layer, while SpanCtrl and LocalBias keep per-token certainty.
  • Static control for all four actuators. The certainty head is removed, and every actuator in layer l reads one learned constant. Feedback is off, since no signal remains to refine. ResidualGate keeps its norm-ratio term (Eq. 3) in every arm, so only the certainty input becomes constant.
  • Separate signal per actuator. The certainty features are kept, and each actuator gets its own readout.
Arm (350M)Gate, recall readSpan, bias readFeedbackSeedsPPLΔPPLCompared with
N-gram backbonen/an/an/a319.47 ± 0.07
Full methodcertaintycertaintyon318.03 ± 0.10
Static sequence controlcertaintyconstanton318.27 ± 0.10+0.24full method
Static depth controlconstantcertaintyon118.22+0.19full method
Full method without feedback (Table 9)certaintycertaintyoff318.14 ± 0.09+0.11full method
Static, all four actuatorsconstantconstantoff318.49 ± 0.11+0.35without feedback
Separate signal per actuatorown readoutown readoutoff118.17+0.03without feedback
Random signal, matched statistics (Table 8)randomrandomon318.99 ± 0.12+0.96full method

Most of the span and locality benefit needs per-token certainty. A constant span and slope cost +0.24, about as much as dropping either control outright (Table 9: +0.21 for SpanCtrl, +0.16 for LocalBias). The depth side behaves differently: static depth control costs +0.19 in one seed, against +0.39 and +0.28 for removing ResidualGate or LocalRecall, so constants keep more of the depth-side benefit.

Across all four actuators, the 1.44 PPL gain splits into 0.98 (68%) that learned constants keep, 0.35 (24%) from reading per-token certainty, and 0.11 (8%) from feedback. The certainty step is nearly as large as removing ResidualGate, the largest single component (+0.39), and it comes from what the signal encodes: a random signal with matched statistics is worse than the constants (18.99 vs 18.49). Four separate readouts give no gain over one shared readout (+0.03, one seed), so a single estimate serves all four actuators, and it is the same quantity the context probe and skipping use.

For adoption, the results give a simpler configuration. Learned constants for all four actuators, with no certainty head or feedback, keep about two thirds of the 350M gain. N-gram certainty and its feedback add the remaining third, and certainty also makes skipping possible. Every arm keeps the staged curriculum and the two auxiliary penalties, whose local sensitivity is in Table 40 (Appendix C.12). We added this table to Section 4.3 (Table 10, with the protocol in Appendix A.15), replaced the Appendix B.4 sentence that listed this ablation as future work, and stated the split in the abstract, contributions, and conclusion.

AuthorsOpenReview ↗

Part 2. At 1.3B, skipping keeps 90% of the five-task gain at the 10.7% speedup point [Q1]

2. At 1.3B, skipping keeps 90% of the five-task gain at the 10.7% speedup point [Q1]

We evaluated the 1.3B full-method checkpoint behind Tables 5 and 12 with the Table 3 harness settings, applying the Table 12 skipping implementation at τ = 0.80 (2.1 skipped layers per token on average) to the scoring passes. No-skip values are this checkpoint's Table 3 scores. Intervals are paired item bootstraps with 2,000 resamples.

TaskBackboneFull, no skipFull, skip τ = 0.80Skip minus no skip [95% CI]Share of gain kept
ARC-Challenge32.936.336.0-0.3 [-0.8, 0.2]91%
HellaSwag58.462.361.9-0.4 [-0.9, 0.1]90%
PIQA73.876.276.0-0.2 [-0.6, 0.2]92%
WinoGrande57.960.360.1-0.2 [-0.7, 0.3]92%
LAMBADA62.866.866.3-0.5 [-0.9, -0.1]88%
Average of the five57.1660.3860.06-0.32 [-0.48, -0.16]90%
MMLU (near chance)25.526.426.2-0.2 [-0.5, 0.1]n/a

The cost of skipping is small and measurable: the five-task average falls by 0.32 points, with an interval that excludes zero. Per task, only LAMBADA's interval excludes zero. LAMBADA also tests correctness in generation, since an item counts as correct only when greedy decoding reproduces the exact final word. With skipping, the model still scores above the strongest reproduced baseline without skipping, Block AttnRes, on all five tasks (average 60.06 vs 59.14, point estimates from Table 3).

AuthorsOpenReview ↗

Part 3. Certainty skipping changes fewer continuations than random skipping, at lower NLL cost [Q1]

3. Certainty skipping changes fewer continuations than random skipping, at lower NLL cost [Q1]

Open-ended web continuations have no single correct answer, so this test measures fidelity to full computation, and the tasks in point 2 measure correctness. Prompts are the first 512 tokens of 500 held-out FineWeb-Edu documents, disjoint from the probe documents. We decode 128 tokens greedily in BF16 with the Table 12 kernels, the workload behind its speedups.

SettingSkipped layers per tokenDecode speedup (Tables 12, 13)Identical to no skipMean tokens before first divergence (128 if identical)NLL under no-skip modelDistinct-2
No skip (repeat run)0n/a100%128.00.64243.8
Certainty, τ = 0.900.62.8%74%104.80.65143.7
Certainty, τ = 0.802.110.7%52%84.00.66843.4
Certainty, τ = 0.704.321.9%29%62.80.71342.5
Random, matched to τ = 0.802.110.7%38%72.60.70442.1

The repeated no-skip run is token-identical, so every divergence comes from skipping. At τ = 0.80, half of the continuations match full computation for all 128 tokens. Averaged over all 500, NLL under the no-skip model rises by 0.026 nats per token (about 4%) with Distinct-2 nearly unchanged (43.4 vs 43.8), while random skipping more than doubles that increase (+0.062). Among continuations that diverge, certainty first differs after about 36 tokens and random after about 39, and the NLL increase is about 0.05 nats per token for certainty against about 0.10 for random. Certainty skipping therefore makes divergence rarer and cheaper. τ = 0.90 is the conservative setting (74% identical, +0.009 nats).

Section 4.4 now summarizes both results, Appendix C.7 gives both tables (Tables 36 and 37), and the Section 5 limitations report the accuracy cost of skipping.

Summary. (1) Token-independent span and locality (Weakness). Replacing N-gram certainty in SpanCtrl and LocalBias with learned per-layer, per-head constants costs +0.24 PPL (3 seeds). Across all four actuators, learned constants keep about two thirds of the 350M gain, and per-token certainty lowers PPL by a further 0.35. (2) Skipping at τ = 0.80 (Q1). The five-task average falls by 0.32 points, the skipped model keeps 90% of its gain over the backbone, and 52% of 128-token greedy continuations match full computation, against 38% for random skipping at the same budget.

Thank you again. We would be glad to provide further analysis.

Reviewer_Su2gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at how the local representations in an n-gram transformer can help control later computation. The authors predict a certainty signal for each token from the n-gram branch. The same signal controls four operations: gating the attention output, adding the initial embeddings back into the model, changing the attention span, and adding a distance bias. The signal is also updated through feedback from attention statistics at each layer. On the 1.3b model, the full method reduces perplexity from 13.79 to 12.75, with about 3.1% more training time. It also gets better results on several downstream tasks. The context truncation and signal replacement experiments suggest a link between certainty and the need for distant context. The authors also test an optional layer-skipping strategy based on certainty. They report a 10.7% inference speedup with a 0.16 increase in ppl. Overall, the method shows clear gains in the experiments. But I think it still needs more evidence about why the control should be shared, what the signal actually means, and how well it works in deployment.

Critical strengths
  • Besides providing input features, it also helps decide how later computation is done. This naturally connects information updates across layers with how much context the model uses along the sequence. Appendix a.1 explains the order of using the control signal and how the residual paths work. These details makes the method easier to follow.

  • The 1.3b results are strong under the tested settings. With the same n-gram backbone and training conditions, the full model performs better than the original backbone and several comparison methods reproduced by the authors. The backbone, block attnres, and the full method all have results from three seeds. The full method beats block attnres by 0.44 ppl. This gap is clearly larger than the standard deviation reported across seeds for either model.

  • The authors check the signal in several ways. They remove context, match word frequency, replace the signal and retrain the model, and change the signal during inference. They also compare routing methods with the same layer-skipping budget. One result I find useful is the test with a 128-token window. After matching word frequency, the loss gap between the lowest and highest certainty groups is still 0.27 nats. This suggests that certainty contains information beyond just how common a word is.

Critical weaknesses
  • My main concern is how much the shared signal really adds. There are no comparisons using fixed controls or separate signals for the four modules under the same settings. Some gains may come from the modules themselves. The hidden-state baseline also predicts the signal only at the input layer, so it does not test layer-by-layer updates.
  • The meaning of certainty is still unclear to me. The tested model has already learned to use this controller, and gating, LocalRecall, and LocalBias remain active. Turning off spanctrl cannot fully separate existing context needs from patterns learned under the controller. Word frequency matching helps, but local prediction difficulty, position, and text type also needs checking.
  • The experiments cover a fairly narrow setting. The main ppl results use a private fine web-edu split, and most larger-model comparisons use one seed. Training length is always 2,048, with no direct tests of long-distance retrieval or dependencies. The standard transformer extension also lacks matched training results. It is still unclear whether the gains hold beyond this n-gram setup.
Questions for authors
  • With the same training budget, what happens if certainty is replaced with a learnable constant for each layer? What if each of the four modules uses its own predicted signal? Could these comparisons show how much extra benefit comes from sharing a dynamic control signal?
  • Could you use certainty from the full model to group tokens when testing the backbone trained without this controller? Would context removal show a similar trend across the groups? After also matching local surprisal, position, and text type, does this relationship still remain?
  • Could you compare simple feedback based only on the current attention statistics, the full feedback method, and a hidden-state predictor at each layer? I would also like to see the learned feedback weights, the certainty distribution, and how much of the certainty values reach saturation.
  • Could you add at least one language modeling result on a public dataset and one task that directly measures the use of distant information? This would help show whether the gains also holds beyond the current private split and local prediction setting.
7 comments
AuthorsOpenReview ↗

Part 1. Per-token N-gram certainty lowers PPL by a further 0.35 on top of the modules [Q1, W1]

Thank you for the detailed review. Below we answer Q1 to Q4 and the points raised in W1 to W3. Table numbers refer to the revised PDF, where new and changed text is in dark red.

1. Per-token N-gram certainty lowers PPL by a further 0.35 on top of the modules [Q1, W1]

All new 350M arms on the N-gram backbone reuse the protocol of the existing runs: the same 20B tokens and data order, optimizer, schedule, curriculum, and auxiliary losses, seeds paired with the full-method runs, and the Table 8 scorer. Feedback below is the predictive-reactive certainty refinement of Section 3.4.

Arm (350M)Gate, recall readSpan, bias readFeedbackSeedsPPLΔPPLCompared with
N-gram backbonen/an/an/a319.47 ± 0.07
Full methodcertaintycertaintyon318.03 ± 0.10
Full method without feedback (Table 9)certaintycertaintyoff318.14 ± 0.09+0.11full method
Learnable constant per layer, all four actuatorsconstantconstantoff318.49 ± 0.11+0.35without feedback
Separate signal per actuatorown readoutown readoutoff118.17+0.03without feedback
Static sequence controlcertaintyconstant per layer and headon318.27 ± 0.10+0.24full method
Static depth controlconstantcertaintyon118.22+0.19full method
Random signal, matched statistics (Table 8)randomrandomon318.99 ± 0.12+0.96full method

The constant arm is your first comparison: the certainty head is removed, and every actuator in layer l reads σ(κ_l), one learnable scalar initialized at 0. ResidualGate keeps its norm-ratio term (Eq. 3) in every arm, so only the certainty input becomes constant. The separate-signal arm keeps the certainty features (LN and W_u in Eq. 1) and gives each actuator its own readout, initialized like the original, so the four signals share features and differ only in their readouts. Both run without feedback and are compared with the no-feedback model, since feedback refines a single signal.

The 1.44 PPL gain splits into 0.98 (68%) that learnable constants keep, 0.35 (24%) from reading per-token certainty, and 0.11 (8%) from feedback. As you suspected in W1, the modules themselves carry most of the gain. The dynamic part is nearly as large as removing ResidualGate, the largest single component (+0.39, Table 9), and it comes from what the signal encodes, since a random signal with matched statistics is worse than constants (18.99 vs 18.49). Both axes use it, the sequence side more: a static sequence side costs +0.24, more than removing SpanCtrl (+0.21), and a static depth side +0.19 (one seed), less than removing ResidualGate (+0.39) or LocalRecall (+0.28). Separate signals give no gain over the shared one (18.17 vs 18.14, one seed), so one estimate serves all four actuators, and it is the quantity we probe (point 3) and threshold for skipping.

AuthorsOpenReview ↗

Part 2. Feedback works through its cross-layer terms, and certainty rarely saturates [Q3, W1]

2. Feedback works through its cross-layer terms, and certainty rarely saturates [Q3, W1]

Variant (350M)How certainty changes across layersFeedback or predictor parametersSeedsPPLΔPPL vs full feedback
Full feedbackupdated from each layer's attention statistic5 scalars318.03 ± 0.10reference
Current statistic only (w_I = w_D = 0)same, current layer only3 scalars318.13 ± 0.09+0.10
No feedbackinitial certainty at every layer0318.14 ± 0.09+0.11
Current and accumulated (w_D = 0)no change term4 scalars118.08+0.05
Current and change (w_I = 0)no accumulated term4 scalars118.10+0.07
Hidden-state projection at layer 0 (Table 8)computed once from the layer-0 state≈0.26M318.38 ± 0.08+0.35
Layerwise hidden-state predictor (new)recomputed at each layer≈0.26M118.29+0.26

The layerwise predictor removes the N-gram certainty head and the feedback, and sets the certainty for layer l to σ(MLP(LN(h_l))), read from the incoming residual state before LocalRecall. One MLP with the Table 8 shape (d to 256 to 1) is shared across layers and active from the start of training. Two findings follow. First, the feedback benefit comes from the cross-layer terms: removing them costs about as much as removing feedback entirely, and with three seeds we read the two rows as one gain of about 0.1 PPL. Second, in one seed, a predictor recomputed from hidden states at every layer does not reach N-gram certainty: it lands at 18.29, between the static layer-0 projection (18.38) and N-gram certainty without feedback (18.14). The Section 5 limitations now report it as a single-seed result.

In the three 1.3B runs, the weights on the initial certainty and the current statistic are positive in every seed, and the two cross-layer weights in two of three (raw values, mean of three seeds, initialization in parentheses): w_c 0.82 (0.5), w_P 0.47 (0.5), w_I 0.21 (0.3), w_D 0.34 (0.5), b -0.16 (0). Together with the 350M ablation, this points to a short memory of the attention statistic rather than a specific control law. Raw weights are not comparable across inputs, since the change term is pre-scaled by γ = 10 (Table 26), so we measure anchoring with the rank correlations below.

Certainty distribution in the 1.3B full model (skipping off, the 512 probe documents, first 2,048 tokens each). Layer l gives the certainty that layer consumes.

LayerMeanSD5th pct.Median95th pct.Below 0.05Above 0.95Spearman with initialMean abs. change
Initial (0)0.5800.2400.1600.6000.9400.56%3.67%n/an/a
10.5780.2400.1600.5960.9360.58%3.60%0.970.018
40.5660.2410.1540.5820.9350.65%3.58%0.920.037
80.5500.2430.1450.5650.9330.80%3.42%0.860.055
160.5300.2470.1300.5430.9301.05%3.15%0.760.078
240.5120.2510.1180.5230.9271.30%2.90%0.680.094
310.4980.2550.1080.5060.9231.55%2.66%0.620.108
Layers 1 to 31 pooled0.5350.2480.1390.5520.9321.00%3.25%0.790.073

At every reported layer, and pooled over layers 1 to 31, about 4% of values lie below 0.05 or above 0.95. Feedback lowers certainty gradually with depth and reorders tokens moderately (rank correlation 0.97 at layer 1, 0.62 at layer 31), so deep layers receive an updated signal that stays anchored to the initial estimate.

AuthorsOpenReview ↗

Part 3. Certainty predicts context sensitivity in a model that never used the controller [Q2, W2]

3. Certainty predicts context sensitivity in a model that never used the controller [Q2, W2]

As you suggested, we grouped the probe targets by the full model's initial certainty, keeping the Table 5 documents, targets, and quintile cuts. Eq. 1 depends only on the input tokens, through the full model's N-gram branch, so it labels the same targets for any model. We then ran context removal on the backbone trained without any controller, so ResidualGate, LocalRecall, SpanCtrl, and LocalBias are all absent.

ΔNLL in nats by context window64128256512Top-1 drop at 128 (points)
Backbone, Q1 (lowest certainty)0.500.350.190.093.0
Backbone, Q20.410.270.140.072.2
Backbone, Q30.310.200.100.051.5
Backbone, Q40.220.140.070.030.8
Backbone, Q5 (highest)0.110.090.040.020.5
Backbone, Q1 minus Q50.390.260.150.072.5
Full method, Q1 minus Q5 (Table 5)0.520.340.190.103.2

The backbone shows the same monotone trend across all five quintiles at every window, and its Q1 minus Q5 contrast is 70% to 79% of the full model's (at 128 tokens, 0.26 [0.232, 0.288] versus 0.34). The relationship therefore exists without the controller. In the full model, where ResidualGate, LocalRecall, and LocalBias stay active during the probe, the contrast is wider at both ends: low-certainty tokens rely slightly more on distant context and high-certainty tokens slightly less.

With all controls applied jointly, 47% of the contrast survives in the full model and 35% in the backbone. The table gives Q1 minus Q5 at 128 tokens in nats, with document-bootstrap 95% intervals (2,000 resamples, repeating the whole procedure in each resample). Local surprisal is the target's surprisal under the backoff N-gram estimator behind the count-based source (Appendix A.2). The joint row regresses the context-removal loss increase on quintile indicators, log frequency, local surprisal, NLL at 2,048, log position, and document fixed effects, and reports the Q1 coefficient.

ControlFull methodShare keptBackboneShare kept
None0.34 [0.312, 0.368]100%0.26 [0.232, 0.288]100%
Token frequency0.27 [0.244, 0.296]79%0.18 [0.156, 0.204]69%
Local surprisal0.22 [0.190, 0.250]65%0.14 [0.114, 0.166]54%
Full-context difficulty (NLL at 2,048)0.19 [0.158, 0.222]56%0.12 [0.094, 0.146]46%
Position0.29 [0.260, 0.320]85%0.22 [0.192, 0.248]85%
Within documents (text type)0.25 [0.220, 0.280]74%0.17 [0.144, 0.196]65%
All jointly (regression with document fixed effects)0.16 [0.124, 0.196]47%0.09 [0.058, 0.122]35%
WikiText-103 test documents, no control0.29 [0.198, 0.382]n/a0.20 [0.118, 0.282]n/a

Among controls that are not part of the outcome, local surprisal absorbs the most, as expected for a signal designed to estimate local predictability: matching on it removes part of what certainty is meant to capture. Full-context difficulty enters the outcome itself (the loss increase is measured from NLL at 2,048), so part of its attenuation is mechanical. Position and document matter less. With all controls jointly, 0.16 remains in the full model and 0.09 in the backbone, both with intervals excluding zero, and the uncontrolled contrast replicates on public WikiText-103 text. Certainty is therefore a learned estimate of local predictability that carries additional information about sensitivity to distant context, and Sections 3.1 and 4.2 now describe it this way.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for concrete, actionable requests. Every requested experiment is complete, and the full tables are in each reviewer's thread and in the revised PDF. This post summarizes them in one place, with seed counts marked.

New resultRequested byHeadline
Certainty replaced by learned constants (350M, 3 seeds unless noted)1tb9, Su2g Q1, KTAdConstants for all four actuators: +0.35 PPL versus per-token certainty, both without feedback. Constant span and bias only: +0.24 versus the full method. Separate signal per actuator, without feedback: +0.03 (one seed).
Feedback (predictive-reactive certainty refinement) variants and a layerwise predictor (350M)Su2g Q3, KTAdCurrent-statistic-only feedback: +0.10 versus full feedback, no measurable gain over no feedback. Layerwise hidden-state predictor without feedback: 18.29 versus 18.14 for N-gram certainty without feedback (one seed).
Learned controller parameters and certainty distribution (1.3B)Su2g Q3, KTAdLocalRecall and LocalBias certainty weights start at zero and become positive on average (82% of layer entries, 71% of head entries). About 4% of certainty values lie below 0.05 or above 0.95.
Context probe on the backbone, with added controls (1.3B, one checkpoint per model)Su2g Q2The backbone trained without the controller shows the same monotone trend (Q1 minus Q5 at 128 tokens: 0.26 versus 0.34 for the full method). With frequency, local surprisal, difficulty, position, and document controls applied jointly, 0.16 [0.124, 0.196] remains for the full method and 0.09 [0.058, 0.122] for the backbone.
WikiText-103 (public data, 3 seeds)Su2g Q4Full method: BPB -0.028 [-0.045, -0.012] relative to the backbone.
Passkey retrieval and repeated-token copy (one checkpoint per model)Su2g Q4The full method is better at every distance from 256 to 1,900 tokens, with the gap widening with distance.
Accuracy and generation at the 10.7% speedup point (1.3B, one checkpoint)1tb9 Q1Five-task average -0.32 points [-0.48, -0.16], 90% of the gain kept. 52% of 128-token greedy continuations identical to full computation, against 38% for random skipping at the same budget.
Controller on an ordinary 350M Transformer (pilot, one seed)Su2g W3PPL 3.8% to 4.8% lower than the vanilla model, versus 7.4% for the full method on the N-gram backbone.

Two findings change how the method is described. First, of the 1.44 PPL gain over the backbone at 350M, 0.98 (68%) remains when the four actuators read learned constants, 0.35 (24%) comes from reading per-token certainty, and 0.11 (8%) from feedback. One shared signal serves all four actuators, since separate signals give no gain (one seed). Second, feedback works through its cross-layer terms, while the current attention statistic alone gives no measurable gain.

The revised PDF (i) adds to Section 3 a summary table (Table 1) linking each component to the per-token decision it makes, its learned direction, and its removal cost, (ii) states the split above in the abstract, contributions, and conclusion, (iii) adds the new controls and evaluations to Sections 4.1 to 4.4 and Appendices A and C, and (iv) revises the limitations to report the public benchmark, the distant-information tests, and the layerwise predictor (one seed) as new evidence, while keeping the ordinary-Transformer pilot and the 2.8B and 3.93B runs labeled single-seed. New and changed text is in dark red, and table numbers in our replies refer to the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.