acceptodds
Under review as a conference paper at ICLR 2027

Which State Should Prediction Read? A Mechanistic Analysis of State-Prediction Separation

Abstract

State-Prediction Separation (SPS) improves language modeling with separate persistent-state and next-token prediction slots, but sharing all weights leaves it unclear whether and how their roles specialize. We trace information flow after giving the streams separate weights. State concentrates predecessor information, while prediction shows stronger induction matching, retrieving continuations of earlier token occurrences. Ablating state's previous-token heads selectively weakens prediction's retrieval, and restricting access to older states worsens loss. Restricting prediction to the deepest state levels costs far less than restricting it to the shallowest, yet SPS never reads its top state layer. We therefore give every prediction layer the final state (a final-state read), a model we call Sequential. With each tower as deep as the Transformer, Sequential lowers seed-mean loss by 0.031 nats/token versus the same-depth read (Two-tower) at matched parameters and floating-point operations (FLOPs). A smaller Sequential model with half the layers per tower comes within 0.005 nats/token of SPS at 55% of its FLOPs and 0.064 below the Transformer at matched non-embedding parameters and FLOPs.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, induction heads, language models, transformer architecture, state–prediction separation, compute efficiency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_D8qsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper studies State-Prediction Separation (SPS), a decoder-only model that gives each token separate state and prediction slots but shares all weights between them. To expose the role of each stream, the paper trains separate-weight variants in which a prediction tower cross-attends to a state tower. It compares two ways for prediction to read state: a same-depth read (Two-tower) and a read of the final state at every prediction layer (Sequential). Using several interpretability tools, the paper argues that state carries predecessor information while prediction does more of induction matching. Prediction needs distant state and, in Two-tower, relies more on deep state levels than on shallow ones. The paper also notes that SPS never reads its top state layer. These findings motivate the final-state read, which the paper tests by training both models from scratch as small language models on a single web-text corpus. At matched cost, Sequential outperforms Two-tower, and a smaller Sequential model nearly matches SPS at the Transformer's compute.

Critical strengths
  • Giving the two streams separate weights is a reasonable way to make the roles of state and prediction measurable.

  • The interpretability analysis combines several complementary tools with explicit controls and reports which effects replicate across seeds.

  • The architecture comparison holds parameters and FLOPs fixed and includes controls that help rule out simple capacity explanations.

Critical weaknesses
  • The architectural contribution appears to overlap substantially with previous final-state designs, which are not included as baselines. Reading the final output of one stack from every layer of a second stack is the standard encoder-decoder pattern. For causal language models, the decoder-decoder YOCO uses a similar layout: its cross-decoder has no self-attention and attends causally to the final output of its self-decoder. The paper distinguishes Sequential from YOCO mainly by its separate state and prediction streams with their own weights and input. However, the paper shows that replacing the prediction embedding table with a single learned token barely changes loss, so the prediction input does not seem to drive the result. Other design differences from YOCO, such as sharing one K/V projection across prediction layers, are not evaluated on their own. It is therefore unclear whether Sequential offers a benefit beyond existing final-state designs.

  • Although the paper frames the final-state read as derived from its mechanistic analysis, the link between the analysis and the design appears weak. The analysis centers on induction retrieval, but Sequential's gain is not concentrated on copyable tokens (Table 4). Moreover, the read-depth evidence is based on restricting a trained Two-tower model at evaluation time, which partly mixes read depth with reads the model was not trained on. This leaves open which property of the final-state read drives the gain.

  • The generality of the result is not well established. All models are trained on a single corpus and evaluated only by validation loss. The gains over Two-tower seem modest. In addition, the preference for the final-state read appears to reverse when the two streams share weights, where the Two-tower read gives slightly lower loss (Table 3). This suggests that the benefit of the final-state read may depend on the architecture setting, and the reason is not yet clear. Separately, the efficiency comparison with SPS is partly post hoc, since the paper adopted this framing after a pre-registered test did not meet its target. The paper does not clearly show that the final-state read helps at larger scale or on downstream tasks.

Questions for authors
  • Which design difference from YOCO is expected to give Sequential an advantage?

  • Why does Two-tower 6+6 outperform the Transformer at matched compute, given that each tower is half as deep as the Transformer?

3 comments

Thank you for the careful review. You raise three concerns: whether Sequential adds anything beyond YOCO and encoder–decoders, whether the analysis leads to the design, and whether the result generalizes. The revision answers each with newly trained models. (Your "Table 3" shared-weight rows are now in App. C.6, and "Table 4" is Tab. 15.)

Novelty over YOCO and encoder–decoders. We agree that the final-state read is not new, and §8 says so. What is new is evidence for why it helps in a decoder-only model, plus a direct test against YOCO. Encoder–decoders read a separate source sequence and change the whole architecture. Our comparisons hold the model fixed and change only which state level prediction reads. A YOCO-style cross-decoder trained with our recipe at equal FLOPs ends 0.049 behind Sequential 6+6 (Tab. 14, App. C.4).

Which difference from YOCO matters? (Q1) Both, tested one at a time at equal FLOPs. First, running prediction as its own stream instead of continuing the state residual is worth 0.025. This is not the embedding table. Against a Sequential with a single ⟨predict⟩ token and the same 169.3M total parameters, the residual-continuing model is still 0.021 worse. You are right that the input's content barely matters (2.801 vs 2.805), but a separate prediction stream does. Second, giving each prediction layer its own K/V projection of the final state, instead of YOCO's single shared set, is worth 0.015 with parameters matched (App. C.4–C.5).

Link between analysis and design. The analysis predicts where prediction should read, not which tokens should gain. We used the induction circuit to find that prediction depends on later, distant state. Reading that state helps every kind of token, so the gain is spread over tokens in proportion to their loss, and induction tokens get 8–14% of it (§5.1, Tab. 15). We agree that evaluation-time restriction alone cannot settle what a trained read prefers (App. B.6), so the revision adds two trained tests. First, when Sequential's prediction layers learn their own mixture of state levels, the final level is the largest source in every layer (App. B.7). Second, a Two-tower with only six prediction layers reading levels 7–12, trained from scratch at Two-tower 12+12's parameters, beats it (2.722 vs 2.733) with 10% fewer FLOPs (App. C.1). Reading later state is what drives the gain.

Shared weights. The reversal is confounded. The tied Shared Sequential reads the final state through one K/V set. That choice alone costs 0.015 (App. C.4), about the size of its 0.012 deficit. With its own prediction table and per-layer K/V reads, Shared Sequential reaches 2.806, 0.009 below Shared Two-tower. A shared pair matched in head, input and K/V would settle the ordering, and App. C.6 says so. Our claim concerns separate weights: sharing removes most of the benefit of separation (App. C.6).

Generality. Training still uses one corpus, but evaluation no longer does. Sequential beats the Transformer in loss and zero-shot accuracy at all four sizes from 11.5M to 327M. It has lower bits per byte on five out-of-domain corpora at 92M and 327M, and it keeps a flat loss lead to 32K tokens after continued pretraining (§6–7, App. E).

Are the gains over Two-tower modest? They are consistent and far beyond noise: 0.037 at 6+6 and 0.031 at 12+12 in both seeds, against a pooled seed standard deviation of about 0.002. Two-tower needs 1.15–1.23× the training FLOPs to reach Sequential's loss (Tab. 17, App. A.3).

Post hoc SPS framing. You are right about the history, and App. D.1 states it: Sequential 6+6 missed its original target of matching Two-tower 12+12. The revision therefore adds a comparison fixed in advance, against SPS at equal training FLOPs. Sequential wins it at both sizes, by 0.037 at 92M and 0.025 at 327M (App. D.1).

Larger scale and downstream tasks. At 327M, Sequential beats the Transformer by 0.042 nats/token and 0.85 macro accuracy points (App. D.5, E.1).

Why does Two-tower 6+6 beat the Transformer? (Q2) It is not specific to Two-tower. Every 6+6 model we trained that gives six layers to a prediction stack reading state by cross-attention beats the 12-layer Transformer, by 0.014 to 0.064. That includes the YOCO-style model, which has fewer parameters and FLOPs. The pattern repeats at the larger budget, where Two-tower 12+12 beats a 24-layer Transformer by 0.023. At these sizes, a dedicated prediction stack with direct reads of state is worth more than the depth it gives up (App. C.3–C.4).

Novelty is now tested directly against YOCO, the read-depth claim rests on trained models, and the evaluation covers downstream, out-of-domain and long-context settings. We hope this addresses the concerns behind your rating, and we would be grateful if you would reconsider it.

Thank you for the detailed response and additional experiments. The rebuttal has addressed my main concerns on novelty, the link between analysis and design, and the generality of results. There are still some concerns on limited models scale, but that seems reasonable for this work. I will increase my score accordingly.

Thank you for your feedback and for reconsidering your score! We appreciate your engagement with our work and are glad that our response addressed your main concerns.

Reviewer_3PyjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the roles of state and prediction streams in SPS. Through mechanistic analysis, the authors find that state retains predecessor information while prediction performs induction matching. Motivated by these findings, they propose Sequential, where every prediction layer reads the final state. Sequential improves validation loss over Two-tower at matched parameters and FLOPs, while a smaller variant approaches SPS performance with substantially less compute.

Critical strengths
  1. Mechanistic insights: The paper provides useful evidence of functional specialization between state and prediction streams through probing, ablation, and patching experiments.

  2. Effective architectural improvement: Sequential consistently outperforms Two-tower at matched parameters and FLOPs. The 6+6 variant also achieves near-SPS validation loss with only 55% of its training FLOPs.

Critical weaknesses

Lack of downstream evaluation. The paper only reports validation loss. It is unclear whether these improvements translate into actual downstream capabilities.

Questions for authors
  1. Downstream evaluation: Please report downstream accuracy on at least standard benchmarks such as LAMBADA, HellaSwag, PIQA, and ARC. Long-context retrieval results would also be useful.

  2. Wall-clock efficiency: The paper uses Triton for SPS but FlexAttention for Sequential. Does the SPS implementation actually skip masked QK computations? Please provide a fair wall-clock comparison using equally optimized kernels. Otherwise, the reported speedup may simply reflect implementation differences.

1 comment

Thank you for the review. Your concerns were the lack of downstream evaluation and whether the speed comparison reflected implementation rather than compute. The revision answers both. The loss gain carries over to downstream accuracy, and on one kernel the remaining speed gap matches the FLOP gap.

Downstream accuracy. We now report zero-shot LAMBADA, HellaSwag, PIQA, ARC-Easy, ARC-Challenge, WinoGrande, OpenBookQA and SciQ for every model and size. Sequential has higher macro accuracy than the Transformer at every size, significantly at three of the four. At 92M, Sequential 6+6 leads by 0.72 points (95% CI [+0.19, +1.24]) (§7, Tabs. 4 and 19).

Long-context retrieval. Within the trained 4K context, Sequential 6+6 finds passkeys more often than the Transformer (0.71 vs 0.59). After continued pretraining at 32K it trails on passkeys beyond 16K, but its 0.043 loss lead stays flat across 32K-token documents (App. E.3, Fig. 5, Tab. 22).

Does the SPS kernel skip masked QK? Only partly, and your concern was justified. Our Triton kernel skips only tiles above the causal diagonal and applies SPS's other masks as dense biases, so it computes about half of the (2T)² score matrix. On Triton, SPS took 2.23× the Transformer's step time against a 1.83× FLOP ratio (App. D.2).

A fair comparison. We re-timed SPS on FlexAttention with block-sparse masks, which skips its masked tiles. On the same kernel and node, SPS takes 1.75× Sequential 6+6's step time, below its 1.83× FLOP ratio. The gap is therefore compute, not implementation. Sequential 6+6 matches the Transformer's speed, within node-to-node variation, and we claim no advantage over it (Tab. 16). The efficiency result also holds at equal training FLOPs: Sequential beats SPS by 0.037 at 92M and 0.025 at 327M (App. D.1).

Reviewer_VgeGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. Method. This paper studies which state representations the prediction stream should read in state–prediction separated language models. Building on SPS, the authors give the state and prediction streams separate weights and compare two architectures: Two-tower, where each prediction layer reads the corresponding state layer's output, and Sequential, where every prediction layer reads the final state layer's output. State uses causal self-attention, while prediction cross-attends to causally visible state positions without attending to other prediction positions.

  2. Experimental setup. The authors train models with 6+6 and 12+12 state/prediction layers, approximately 208M and 300M parameters, on 20B FineWeb-Edu tokens at 4K context length. They compare against Transformer, SPS, and Almost-Free SPS baselines, and test controls for prediction embeddings, LM-head tying, weight sharing, and tower-depth allocation. Head ablations, activation patching, probes, and read restrictions examine the computations learned by each stream and which state representations prediction uses.

  3. Results. At matched parameters and FLOPs, Sequential lowers validation NLL relative to Two-tower from 2.838 to 2.801 at 6+6 and from 2.733 to 2.702 at 12+12. Replacing the prediction embedding table with a single learned token gives a 169M Sequential 6+6 model with NLL 2.805, compared with 2.865 for the Transformer and 2.797 for SPS. This variant matches the Transformer's parameter count and FLOPs while using approximately 55% of SPS's FLOPs. The mechanistic experiments support a division of labor in which state encodes predecessor information and prediction uses it for induction matching.

Critical strengths
  1. Controlled architectural comparison and useful ablations. The main Two-tower versus Sequential comparison changes which state layer prediction reads while keeping parameters, FLOPs, and the training setup fixed. Sequential improves validation NLL at both tested depths and in both training seeds. The prediction-embedding and LM-head-tying controls also strengthen the efficiency result: the smaller Sequential model retains its advantage over the Transformer after removing the extra prediction embedding table and tying the output head. The fixed-budget comparison of 3+9, 6+6, and 9+3 further tests how depth should be allocated between the two towers.

  2. Causal evidence for complementary stream roles. The authors go beyond attention visualizations and probes by ablating previous-token state heads and testing whether restoring their clean activations after corruption recovers retrieval. These experiments support the claim that prediction uses predecessor information computed by state. In particular, the head-ablation effect in Sequential 12+12 is significant against both random and pattern-matched controls across both seeds. This provides useful evidence for the proposed division of labor, rather than simply observing different attention patterns in the two streams.

Critical weaknesses
  1. The quality evaluation is too limited to establish whether the validation-loss gains translate to broader model capabilities.

    The main quality results are held-out FineWeb-Edu NLL on 4K sequences. These improvements are useful, but do not by themselves establish an advantage on downstream tasks. The induction experiments examine a specific retrieval mechanism rather than broader retrieval performance. Although the authors acknowledge the evaluation limitation, the existing checkpoints would already permit several informative comparisons:

    • Short-context performance: Evaluate held-out LM perplexity on a corpus outside FineWeb-Edu, standard downstream benchmarks (e.g., HellaSwag, PIQA, ARC), and retrieval (NIAH/RULER) within the trained 4K context window. I am particularly interested in comparing Sequential with Transformer, SPS, and Two-tower to determine whether their validation-loss ordering carries over to these capabilities. Likelihood-based evaluation, such as bits per byte (BPB) on the scored answers or continuations of the LM-7 tasks, would also be useful alongside accuracy.

    • Long-context performance: I would also encourage continued pretraining at 32K, using the same long-context training data, token budget, objective, and learning-rate schedule across architectures, followed by evaluation within that context window. In addition to long-document perplexity, report next-token loss broken down by token position, as studied in TTT-E2E [1, Section 3.4.1, Figure 6]. Plotting both the loss and its difference from the Transformer would show whether Sequential's advantage persists toward the end of the document or is concentrated in earlier tokens. NIAH/RULER across context lengths and evidence positions would provide a complementary test of retrieval. LongBench would also be useful on tasks where the baseline models perform meaningfully.

    These comparisons would help establish whether final-state reads improve broader model capabilities and remain beneficial as the available context grows, rather than only lowering average loss in the current 4K setting.

[1] Tandon et al. End-to-End Test-Time Training for Long Context. arXiv:2512.23675, 2025.

  1. The baseline comparisons are incomplete across compute budgets, and the gains have not been tested at larger scales.

    The matched Two-tower–Sequential comparisons establish that final-state reads improve NLL in the tested setting. However, the comparisons against Transformer and SPS do not establish which architecture offers the best quality at each compute budget, and all experiments remain at or below 300M parameters.

    • Matched baselines: At the lower budget, Sequential 6+6 is compared with a compute-matched 12-layer Transformer, but SPS uses approximately 1.8x as many FLOPs per token. A shallower SPS model, approximately six layers, would complete the comparison at this budget. At the higher budget, the 12+12 models are compared with 12-layer SPS, but not a 24-layer Transformer with comparable FLOPs and non-embedding parameters. The latter would test whether separation is preferable to spending the same budget on additional standard Transformer layers. Total and non-embedding parameter counts should both be reported, since matching compute does not also match parameters when weights are shared.

    • Scaling: I would like to see these comparisons extended toward 1.5B parameters, for example using approximately 180M, 440M, 880M, and 1.5B Transformer configurations as reference compute budgets. At each budget, compare Transformer, SPS, Two-tower, and Sequential with comparable FLOPs per token, the same training data and token budget, and a consistent training recipe. Report scaling plots for held-out NLL and downstream performance, including LM-7 accuracy or answer/continuation BPB. Figure 4 provides useful training-compute trajectories for the existing configurations, but does not establish whether the architectural advantage persists as model size increases.

  2. The prediction-depth ablation changes depth, parameter count, and compute together.

    • Appendix D.5 uses the top-aligned Two-tower 12+6 experiment to argue that access to deep state is insufficient without enough prediction depth. This model retains all 12 state layers and makes its six prediction layers read state levels 7–12. Relative to Two-tower 12+12, NLL increases from 2.732 to 2.757 in the seed-1 runs. However, parameters also decrease from 300M to 251M and forward compute from 0.751 to 0.577 GFLOPs/token (Table 5). The comparison therefore shows that this smaller configuration performs worse, but does not distinguish the effect of prediction depth from the reduction in capacity and compute.

    • A more informative comparison would retain the 12-layer state tower and top-aligned reads, widen the six prediction layers, and compare against Two-tower 12+12 under matched budgets. For example, increasing prediction MLP width would allow parameter-matched and FLOP-matched controls, reported separately if both cannot be matched simultaneously. This would better test whether a deeper prediction tower is preferable to a shallower, wider one, rather than simply whether removing six layers hurts performance.

Questions for authors
  1. Short-context LM and retrieval performance.

    • Could the authors evaluate the existing 4K checkpoints on held-out LM perplexity using a corpus outside FineWeb-Edu, standard downstream tasks (e.g., HellaSwag, PIQA, ARC), and retrieval benchmarks (e.g., NIAH/RULER) within the trained context window?

    • I am particularly interested in comparing Transformer, SPS, Two-tower, and Sequential. Does Sequential's advantage over the Transformer at the lower compute budget, and over Two-tower at both tested depths, persist on these tasks? Including SPS, while retaining the distinction between the two compute budgets, would also help establish whether Sequential preserves its capabilities at lower cost.

    • Likelihood-based evaluation on the LM-7 tasks would also be useful, such as bits per byte (BPB) on the correct answers or continuations. Please specify the scored spans and report per-task results alongside the aggregate. These measurements would complement accuracy and help determine whether the improvements extend beyond the pretraining distribution.

  2. Long-context performance after continued pretraining at 32K.

    • Could the authors continue pretraining Transformer, SPS, Two-tower, and Sequential at 32K? The long-context training data, additional token budget, objective, and learning-rate schedule should be kept the same across methods, with all model parameters updated. Each model should retain its intended attention structure and be evaluated within the trained context window. Please also report the continuation FLOPs, since equal token budgets do not imply equal compute across these architectures.

    • I would be interested in held-out long-document perplexity, including next-token loss broken down by token position, as studied in TTT-E2E [1, Section 3.4.1, Figure 6]. Specifically, evaluate the same long documents and average the next-token loss at each position or over fixed position bins. Plotting both these losses and Sequential's loss difference from the Transformer would show whether the advantage persists as more context becomes available, or whether the aggregate improvement is concentrated in earlier tokens.

    • Could the authors also evaluate retrieval on NIAH/RULER across context lengths up to 32K and different evidence positions? LongBench would be a useful additional evaluation on tasks where the baseline models perform meaningfully. Together, these results would help establish whether final-state reads remain beneficial for both long-document language modeling and retrieval.

[1] Tandon et al. End-to-End Test-Time Training for Long Context. arXiv:2512.23675, 2025.

  1. Matched baselines and scaling toward 1.5B parameters.

    • Could the authors complete the baseline comparisons at both existing compute budgets? At the lower budget, add a shallower SPS model (approximately six layers) alongside the 12-layer Transformer, Two-tower 6+6, and Sequential 6+6. At the higher budget, add a 24-layer Transformer alongside SPS 12, Two-tower 12+12, and Sequential 12+12. These baselines should use the same 20B-token training data, tokenizer, context length, and learning-rate schedule. This would test whether Sequential is preferable to reducing SPS depth or increasing standard Transformer depth at comparable compute.

    • Could the authors extend these comparisons toward 1.5B parameters, for example using approximately 180M, 440M, 880M, and 1.5B Transformer configurations as reference compute budgets? At each scale, compare Transformer, SPS, Two-tower, and Sequential with comparable FLOPs per token and the same training data and token budget. Please report total and non-embedding parameter counts as well as training FLOPs, since SPS's weight sharing means that matching compute does not also match parameter count.

    • I would be interested in scaling plots for both held-out NLL and downstream LM performance, including LM-7 accuracy or answer/continuation BPB using the scoring protocol in Question 1. Reporting these metrics against model size and training compute would help establish whether the gains persist, narrow, or grow with scale. Figure 4 provides useful training trajectories for the existing configurations, but does not answer this model-size scaling question.

  2. Prediction depth at matched parameter and compute budgets.

    • Could the authors add a widened version of the top-aligned Two-tower 12+6 model to the comparison in Appendix D.5? Keep the 12-layer state tower and prediction reads from state levels 7–12 unchanged, but increase the prediction MLP width to approximately match the Two-tower 12+12 budget. Parameter-matched and FLOP-matched controls could be reported separately if both budgets cannot be matched simultaneously.

    • Please train these controls using the same data, 20B-token budget, 4K context length, and learning-rate schedule, and report total and non-embedding parameters, forward FLOPs/token, and validation NLL. Does the gap to Two-tower 12+12 persist? This would better test whether a deeper prediction tower is preferable to a shallower, wider one, rather than simply whether reducing its parameter and compute budgets hurts performance.

3 comments

Thank you for the detailed review and the concrete protocols. You raised three concerns: that validation loss may not reflect broader capability, that the baselines and scale were incomplete, and that the prediction-depth ablation confounded depth with capacity. We ran the experiments you proposed for each. In short, the ordering carries over to downstream, out-of-domain and long-context evaluation, and Sequential beats the added baselines at every size. Your widened control also sharpens our depth conclusion. (Your "Appendix D.5" is now App. C.1.)

Beyond validation loss. On eight zero-shot tasks, Sequential 6+6 beats the Transformer by 0.72 macro points (paired 95% CI [+0.19, +1.24]). Sequential 12+12 beats Two-tower 12+12 by 0.87 points and a 24-layer Transformer by 1.69. Against Two-tower 6+6, the 0.39-point accuracy lead is not significant, but the bits-per-byte lead is. SPS only ties Sequential 6+6 while spending 1.83× its FLOPs. Bits per byte is scored on lm-eval's gold continuation and reported per task (Tab. 19). Out of domain, Sequential has lower bits per byte than the Transformer on all five corpora at 92M and 327M, and lower than Two-tower on all but one (Tab. 21). Within 4K tokens, it finds passkeys (0.71 vs 0.59) and multi-key needles (0.54 vs 0.41) more often than the Transformer (§7, App. E.1–E.3).

For long context we followed your protocol. All six size-s models continue for 1.05B tokens at 32K with the same data, objective and schedule. All parameters are updated, and each model keeps its own attention structure. A continuation costs 4.7×10¹⁸ FLOPs for the Transformer and the 6+6 models, and 9.1×10¹⁸ for SPS and the 12+12 models. Loss by position answers your main question: on PG19, Sequential 6+6's lead over the Transformer is 0.042 at 2K–4K and 0.043 at 16K–32K, so the advantage does not fade with context (Figs. 5 and 12). Retrieval is the one mixed result. Across seven needle depths, Sequential 6+6 trails the Transformer on passkeys at 16K–32K, while both 12+12 models are best from 8K on (Tab. 22). RULER and LongBench are left to follow-up.

Baselines and scale. The 24-layer Transformer you asked for reaches 2.756, behind Two-tower 12+12 (2.733) and Sequential 12+12 (2.702), so separation beats spending the same budget on depth. We did not train a 6-layer SPS. Interpolating SPS's own size ladder to the Transformer's FLOPs gives 2.958, against 2.801 for Sequential 6+6 (Fig. 9). Tabs. 3 and 5 give total and non-embedding parameters and FLOPs/token for every model. We also trained all four architectures at four sizes, from 11.5M to 327M non-embedding parameters, with the same 20B tokens and recipe. Sequential beats the Transformer and Two-tower in loss at every size, and the Transformer in accuracy. The margin over the Transformer narrows from 0.064 to 0.042, still eight times our tie threshold. SPS has lower per-token loss at 327M but uses 1.92× the FLOPs; at equal training FLOPs, Sequential beats it by 0.025 (Tab. 17, Figs. 8 and 11, App. D.1, D.6). Baselines at 880M and 1.5B did not fit our compute.

Prediction depth. You were right to separate depth from capacity, and with that done, the gap does not persist. With its prediction MLP widened, the top-aligned 12+6 beats Two-tower 12+12 at matched parameters (2.722 vs 2.733) while using 10% fewer FLOPs. At matched FLOPs it reaches 2.699, level with Sequential 12+12 (2.702). Access to later state matters more than prediction depth, which is what Sequential is built on (Tab. 3, App. C.1).

Together, these results show that the final-state read improves capability beyond validation loss and holds across budgets and sizes. We hope they address the concerns behind your rating, and we would be grateful if you would reconsider it.

I thank the authors for their response and I am satisfied with their answers to my questions. I have raised the score to a 3.

Thank you for your feedback and for reconsidering your score! We appreciate your engagement with our work and are glad that our response addressed your main concerns.

Comments on the paper

1 comment

Thank you for the reviews. We answer each reviewer in the reply to that review. Here is the case the revised paper makes:

  • Reading the final state improves a decoder-only model at a fixed cost. In Sequential, every prediction layer reads the final state. At equal non-embedding parameters and FLOPs, it beats the Transformer at all four sizes from 11.5M to 327M, by 0.042 to 0.079 nats/token (Fig. 4e, Tab. 17).
  • The gain goes beyond validation loss. Sequential has higher zero-shot accuracy on eight tasks at every size, significantly at 27M, 92M and 327M. It also has lower out-of-domain bits per byte at 92M and 327M, and its loss lead stays flat across 32K-token documents after continued pretraining (§7, Tab. 21, Figs. 5 and 11).
  • It beats the strongest alternatives at matched budgets. At 12+12, Sequential (2.702) beats both a 24-layer Transformer (2.756) and Two-tower (2.733), and at 6+6 it beats a YOCO-style cross-decoder by 0.049. At equal training FLOPs it beats SPS by 0.037 at 92M and 0.025 at 327M (Tabs. 3 and 14, App. D.1).
  • The analysis explains why the design works. With separate weights, state heads record each token's previous token and prediction heads match against that record. Read restrictions and lesions show that prediction relies most on later, distant state, while SPS never reads its top state layer. Sequential reads that state directly, and when given a learned read over all levels, it puts the most weight on the final level (§3–4, App. B.7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.