- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies State-Prediction Separation (SPS), a decoder-only model that gives each token separate state and prediction slots but shares all weights between them. To expose the role of each stream, the paper trains separate-weight variants in which a prediction tower cross-attends to a state tower. It compares two ways for prediction to read state: a same-depth read (Two-tower) and a read of the final state at every prediction layer (Sequential). Using several interpretability tools, the paper argues that state carries predecessor information while prediction does more of induction matching. Prediction needs distant state and, in Two-tower, relies more on deep state levels than on shallow ones. The paper also notes that SPS never reads its top state layer. These findings motivate the final-state read, which the paper tests by training both models from scratch as small language models on a single web-text corpus. At matched cost, Sequential outperforms Two-tower, and a smaller Sequential model nearly matches SPS at the Transformer's compute.
Critical strengths
-
Giving the two streams separate weights is a reasonable way to make the roles of state and prediction measurable.
-
The interpretability analysis combines several complementary tools with explicit controls and reports which effects replicate across seeds.
-
The architecture comparison holds parameters and FLOPs fixed and includes controls that help rule out simple capacity explanations.
Critical weaknesses
-
The architectural contribution appears to overlap substantially with previous final-state designs, which are not included as baselines. Reading the final output of one stack from every layer of a second stack is the standard encoder-decoder pattern. For causal language models, the decoder-decoder YOCO uses a similar layout: its cross-decoder has no self-attention and attends causally to the final output of its self-decoder. The paper distinguishes Sequential from YOCO mainly by its separate state and prediction streams with their own weights and input. However, the paper shows that replacing the prediction embedding table with a single learned token barely changes loss, so the prediction input does not seem to drive the result. Other design differences from YOCO, such as sharing one K/V projection across prediction layers, are not evaluated on their own. It is therefore unclear whether Sequential offers a benefit beyond existing final-state designs.
-
Although the paper frames the final-state read as derived from its mechanistic analysis, the link between the analysis and the design appears weak. The analysis centers on induction retrieval, but Sequential's gain is not concentrated on copyable tokens (Table 4). Moreover, the read-depth evidence is based on restricting a trained Two-tower model at evaluation time, which partly mixes read depth with reads the model was not trained on. This leaves open which property of the final-state read drives the gain.
-
The generality of the result is not well established. All models are trained on a single corpus and evaluated only by validation loss. The gains over Two-tower seem modest. In addition, the preference for the final-state read appears to reverse when the two streams share weights, where the Two-tower read gives slightly lower loss (Table 3). This suggests that the benefit of the final-state read may depend on the architecture setting, and the reason is not yet clear. Separately, the efficiency comparison with SPS is partly post hoc, since the paper adopted this framing after a pre-registered test did not meet its target. The paper does not clearly show that the final-state read helps at larger scale or on downstream tasks.
Questions for authors
-
Which design difference from YOCO is expected to give Sequential an advantage?
-
Why does Two-tower 6+6 outperform the Transformer at matched compute, given that each tower is half as deep as the Transformer?
3 comments
Thank you for the detailed response and additional experiments. The rebuttal has addressed my main concerns on novelty, the link between analysis and design, and the generality of results. There are still some concerns on limited models scale, but that seems reasonable for this work. I will increase my score accordingly.
Thank you for your feedback and for reconsidering your score! We appreciate your engagement with our work and are glad that our response addressed your main concerns.
Thank you for the careful review. You raise three concerns: whether Sequential adds anything beyond YOCO and encoder–decoders, whether the analysis leads to the design, and whether the result generalizes. The revision answers each with newly trained models. (Your "Table 3" shared-weight rows are now in App. C.6, and "Table 4" is Tab. 15.)
Novelty over YOCO and encoder–decoders. We agree that the final-state read is not new, and §8 says so. What is new is evidence for why it helps in a decoder-only model, plus a direct test against YOCO. Encoder–decoders read a separate source sequence and change the whole architecture. Our comparisons hold the model fixed and change only which state level prediction reads. A YOCO-style cross-decoder trained with our recipe at equal FLOPs ends 0.049 behind Sequential 6+6 (Tab. 14, App. C.4).
Which difference from YOCO matters? (Q1) Both, tested one at a time at equal FLOPs. First, running prediction as its own stream instead of continuing the state residual is worth 0.025. This is not the embedding table. Against a Sequential with a single ⟨predict⟩ token and the same 169.3M total parameters, the residual-continuing model is still 0.021 worse. You are right that the input's content barely matters (2.801 vs 2.805), but a separate prediction stream does. Second, giving each prediction layer its own K/V projection of the final state, instead of YOCO's single shared set, is worth 0.015 with parameters matched (App. C.4–C.5).
Link between analysis and design. The analysis predicts where prediction should read, not which tokens should gain. We used the induction circuit to find that prediction depends on later, distant state. Reading that state helps every kind of token, so the gain is spread over tokens in proportion to their loss, and induction tokens get 8–14% of it (§5.1, Tab. 15). We agree that evaluation-time restriction alone cannot settle what a trained read prefers (App. B.6), so the revision adds two trained tests. First, when Sequential's prediction layers learn their own mixture of state levels, the final level is the largest source in every layer (App. B.7). Second, a Two-tower with only six prediction layers reading levels 7–12, trained from scratch at Two-tower 12+12's parameters, beats it (2.722 vs 2.733) with 10% fewer FLOPs (App. C.1). Reading later state is what drives the gain.
Shared weights. The reversal is confounded. The tied Shared Sequential reads the final state through one K/V set. That choice alone costs 0.015 (App. C.4), about the size of its 0.012 deficit. With its own prediction table and per-layer K/V reads, Shared Sequential reaches 2.806, 0.009 below Shared Two-tower. A shared pair matched in head, input and K/V would settle the ordering, and App. C.6 says so. Our claim concerns separate weights: sharing removes most of the benefit of separation (App. C.6).
Generality. Training still uses one corpus, but evaluation no longer does. Sequential beats the Transformer in loss and zero-shot accuracy at all four sizes from 11.5M to 327M. It has lower bits per byte on five out-of-domain corpora at 92M and 327M, and it keeps a flat loss lead to 32K tokens after continued pretraining (§6–7, App. E).
Are the gains over Two-tower modest? They are consistent and far beyond noise: 0.037 at 6+6 and 0.031 at 12+12 in both seeds, against a pooled seed standard deviation of about 0.002. Two-tower needs 1.15–1.23× the training FLOPs to reach Sequential's loss (Tab. 17, App. A.3).
Post hoc SPS framing. You are right about the history, and App. D.1 states it: Sequential 6+6 missed its original target of matching Two-tower 12+12. The revision therefore adds a comparison fixed in advance, against SPS at equal training FLOPs. Sequential wins it at both sizes, by 0.037 at 92M and 0.025 at 327M (App. D.1).
Larger scale and downstream tasks. At 327M, Sequential beats the Transformer by 0.042 nats/token and 0.85 macro accuracy points (App. D.5, E.1).
Why does Two-tower 6+6 beat the Transformer? (Q2) It is not specific to Two-tower. Every 6+6 model we trained that gives six layers to a prediction stack reading state by cross-attention beats the 12-layer Transformer, by 0.014 to 0.064. That includes the YOCO-style model, which has fewer parameters and FLOPs. The pattern repeats at the larger budget, where Two-tower 12+12 beats a 24-layer Transformer by 0.023. At these sizes, a dedicated prediction stack with direct reads of state is worth more than the depth it gives up (App. C.3–C.4).
Novelty is now tested directly against YOCO, the read-depth claim rests on trained models, and the evaluation covers downstream, out-of-domain and long-context settings. We hope this addresses the concerns behind your rating, and we would be grateful if you would reconsider it.