acceptodds
Under review as a conference paper at ICLR 2027

Single-Layer, Two Axes: Communication and Refinement in Looped Transformers

Abstract

Looped Transformers increase computation by reusing parameters and outperform standard Transformers of the same size, but their architectures often change two things together: how a token is processed and which intermediate representations it can read. This coupling obscures the source of their gains. To disentangle the two, we introduce a Single-Layer framework that separates refinement, the choice of feed-forward operators, from communication, access to historical token–depth states. Unfolding recurrent depth into an addressable sequence of latent states lets one shared attention module support controlled changes to either axis. In 274M-parameter looped MoE language models with matched FFN compute, cross-depth access lowers held-out perplexity by 2.4–8.1% relative to same-depth access under all four refinement schedules, yet depth-specific expert banks still outperform a shared bank by 11.8–15.5%. Better communication and more effective parameter reuse are therefore distinct design questions, and they interact: the benefit of start injection depends on the available historical context. Varying the retained depths for recent and distant tokens shows that the preferred depth depends on token age and refinement, and that retaining all tested depths is worse than keeping the best one. These observations motivate a Single-Layer Transformer combining routed expert refinement with recent working memory and selective long-term memory. A compact 20M-parameter comparison at matched total capacity confirms the cross-depth advantage under both bank assignments, with a smaller gap between shared and depth-specific refinement. In the shared-bank model, W = 8 lowers perplexity by 7.9% relative to same-depth access and by 0.37% relative to full cross-depth access, while storing 84.8% fewer historical states.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recurrent transformers, looped transformers, parameter sharing, language modeling, autoregressive decoding, persistent KV memory, mixture of experts, recurrent computation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Lnv1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies looped MoE Transformers through a unified Single-Layer framework that decouples communication (which historical token-depth states are readable via a shared attention module) from refinement (how FFN expert banks are scheduled across RR recurrent steps). Using an alternating even/odd parallel training scheme and exact token-major serial evaluation on FineWeb-Edu (at 274M and 20M resident params, ctx length T=256T = 256), the authors show that cross-depth historical access consistently lowers Dev PPL across all FFN schedules, while depth-specific expert banks still beat a single shared bank. They further test a selective memory policy (retaining full depth trajectories for the last WW tokens and a single shallow anchor aa for older tokens), cutting analytic KV state storage by 84.8% at W=8W = 8 w/o hurting PPL.

Critical strengths
  • The paper cleanly separates cross-depth KV access from FFN sharing in looped MoE transformers. This makes it easy to see how historical context access and expert scheduling each affect performance.
  • The selective window and anchor memory idea is very practical. It keeps full depth states only for recent tokens and one shallow state for older tokens, which cuts KV cache size a lot without hurting perplexity.
Critical weaknesses

My biggest concern is the gap between the paper's training/inference mechanics and practical efficiency, combined with the lack of wall-clock or compute-matched baselines. Exact causal execution (Sec. 3.2) is strictly token-major because shallow steps of token ii read deeper steps of token m<im < i, which means even prompt prefill cannot be parallelized across tokens w/o falling back to the multi-pass approximation. Meanwhile, parallel training requires 10 sequential passes per step (approx. 6 full-scan equivalents in FLOPs, Sec. D.2), yet all comparisons are only token-matched (400M--450M tokens) rather than training-FLOP matched. On top of that, Tab. 2 only reports analytic matmul counts where same-depth and full cross-depth are listed at the exact same 0.061 GFLOPs/token simply because the current kernel computes dense attention before masking (line 795), with zero actual latency, throughput, or peak memory numbers.

Second, the headline result in Sec. 5.2 that retaining all tested boundaries {z0,z8,z16}\{z_0, z_8, z_{16}\} hurts PPL by 10.5%--90.8% (Fig. 4b, Tab. 6) looks heavily confounded by the training approximation. In Tab. 6, adding {z0,z8,z16}\{z_0, z_8, z_{16}\} makes the shared-bank model blow up from 28.28 to 53.96 PPL, whereas in Tab. 1, full cross-depth (which exposes all 16 depths) reaches 24.32 PPL on the same 274M model. Looking at App. F.2, Tab. 6 switched to a 2-sweep block-Jacobi training contract (and lacks paired parallel/exact gap checks, line 904), whereas the no-extra-anchor baseline used exact-graph training and Tab. 1 used a K=8K = 8 schedule. It seems very likely that the 53.96 PPL spike is an optimization/Jacobi discrepancy artifact from exposing unconverged deep states (z16z_{16}) under only 2 sweeps, rather than more historical information inherently hurting the architecture.

Finally, the empirical scope feels somewhat thin and incomplete in places. All main experiments use a very short ctx length (T=256T = 256), a small token budget (400M--450M tokens, which is quite undertrained for a 274M MoE pool with 48 experts and likely exaggerates the gap between 1×481 \times 48 shared routing and 16×316 \times 3 depth-specific banks), a single seed, and Dev-only PPL with no downstream evaluations. The text also contains several odd notes about missing experimental logs (e.g., lines 613--614, Tab. 4 missing train PPL, Tab. 7 missing the K=16K = 16 endpoint, and line 904 missing parallel/exact scores for Tab. 6), which makes the experimental record feel a bit unfinished.

Questions for authors
  1. In Tab. 6, why does retaining {z0,z8,z16}\{z_0, z_8, z_{16}\} degrade shared-bank Exact Dev PPL to 53.96 when retaining all 16 depths in Tab. 1 achieves 24.32? Can you report the parallel-vs-exact discrepancy ∣Δgraph∣|\Delta_{\text{graph}}| for the Tab. 6 runs, and check if that degradation disappears under the stronger alternating schedule used in Tab. 1?

  2. How does the model handle prompt prefill at inference time? If exact execution is strictly serial token-by-token (Fig. 2c), doesn't prefill latency scale sequentially with T×RT \times R? Could you provide actual wall-clock throughput and peak GPU memory for same-depth, full cross-depth, and selective memory (W=8W = 8)?

  3. In the 274M study, each expert in the 1×481 \times 48 shared bank is routed from a 48-expert pool at every step, whereas 16×316 \times 3 only routes among 3 experts per depth. At 450M tokens, could the 11.8%--15.5% shared-bank deficit simply be router under-training or load-balancing friction in a 48×48\times larger action space? Do you have expert utilization stats or longer token-horizon runs?

  4. Since same-depth access does not have cross-token depth dependencies, it can be trained in a single exact pass rather than 10 alternating passes (~6×6\times fewer training scans). How does same-depth compare to cross-depth when matched on total training FLOPs instead of training tokens?

5 comments
AuthorsOpenReview ↗

Response to Lnv1 (1/3): stronger training and equal-compute results

Thank you for asking us to check the training schedule and the equal-compute comparison.

In Tab. 6, why does retaining [...] degrade shared-bank Exact Dev PPL to 53.96 when retaining all 16 depths in Tab. 1 achieves 24.32?

Can you report the parallel-vs-exact discrepancy [...] for the Tab. 6 runs, and check if that degradation disappears under the stronger alternating schedule used in Tab. 1?

We understand the concern as whether added historical boundaries help under stronger state estimation and exact execution. We test this with a newly trained paired follow-up.

Under the stronger schedule, added anchors give lower Exact PPL (27.5258 vs 28.3792), with both absolute graph CE gaps below 0.001. The pair uses a 274.43M shared-bank model, R=16, context 256, W32 and 400,031,744 training tokens.

Added distant anchorsExact PPLTraining-graph PPLAbsolute CE gap
z028.379228.39470.000548
z0, z8, z1627.525827.49960.000953

Both arms retain the segment-entry state matched to the query and are based on the all-boundary configuration. Their model/training configurations differ only in the added-anchor policy. The differentiable serial warm start and ten gradient-tracked updates cost seven full-token scans. Both use the same 128 validation windows.

The all-boundary arm finished its last 132 updates on H800 with a different PyTorch version, while z0 stayed on H100. Appendix G documents the original 53.96 endpoint and its earlier settings.

The new pair shows a prediction benefit from richer historical access under exact execution while the expert bank remains shared. It supplies a controlled design judgment about communication within shared refinement.

How does same-depth compare to cross-depth when matched on total training FLOPs instead of training tokens?

The current architectural payoff is that useful history can improve a shared bank even within one-pass training. At fixed bank and per-query historical key count, the 100M source control reaches CE 2.40 with less data (0.8B vs 2.0B tokens), and CE 2.30 with 1.2B vs 2.6B tokens (Figure 6).

For the original 20M recipe, the one-pass controls achieve lower PPL at equal analytic training FLOPs.

Historical access and trainingTraining tokensAnalytic training FLOPsExact PPL
Full cross-depth, ten passes400,031,7443.403489e1731.2819
Same-depth, one exact pass2,455,044,0963.403507e1727.5167
Fully untied dense Standard Transformer2,752,905,2163.403489e1725.4879

The cross-depth reference has 20,203,648 parameters, R=8 and context 256. The same-depth control shares attention, differentiates all eight updates and retains the original routing. Warmup and cosine follow the same fraction of the compute budget. Integer-step rounding changes budgets by at most 0.00052%. All endpoints use 128 context-256 windows. FLOPs count logical forward/backward matrix work.

The dense Standard Transformer has eight independent attention/FFN sets, 19.97M parameters (1.16% fewer) and width 320, compared with the width-512 MoE shared reference. The one-pass controls use the fixed budget to process 2.455B/2.753B tokens, versus 0.400B for repeated cross-depth scans.

Fresh KV reduces repeated scans in the newer 100M, context-4096 study. At 185,057,280 tokens, two scans give lower CE (2.8341 vs 2.9163/2.8703) than the older four/five-scan schedules. At 2,000,076,800 tokens, two/four scans reach 2.1233/2.1442. These compare complete schedules, including supervision distributions (Figure 3, Appendix J.2). Progressive pass schedules, as explored by Full-bandwidth Transformer, offer a further candidate for this architecture.

At matched data, the completed 100M Single-Layer model improves CE and downstream mean over the 105M Standard Transformer. At 200M, the mean is 42.28 vs 42.37 for the 288M control with 30.7% fewer parameters (Section 3.1, Table 1). The source control identifies useful communication within shared refinement, and the complete models measure the value of their combination.

AuthorsOpenReview ↗

Response to Lnv1 (2/3): Fresh KV training and parallel prefill

Meanwhile, parallel training requires 10 sequential passes per step (approx. 6 full-scan equivalents in FLOPs, Sec. D.2), yet all comparisons are only token-matched (400M--450M tokens) rather than training-FLOP matched.

Fresh KV supplies updated historical states with lower Exact CE and fewer full-token scans at the common measured checkpoints. At query depth j, sources up to j come from the current pass, while deeper sources use previous-pass estimates. Later depths can use fresh shallow states within the same scan.

The 100M, R=8, context-4096 comparison uses 185,057,280 training tokens per row. Exact evaluation uses the same 32 windows and 131,072 scored tokens.

Training scheduleFull-token scans / sequential callsExact CE
Fresh KV, two full passes2 / 22.8341
Older alternating schedule4 / 62.9163
Older alternating schedule5 / 82.8703

At 2,000,076,800 tokens, two/four scans reach CE 2.1233/2.1442. All schedules supervise updated positions and differentiate every update. This compares complete schedules, including their distributions of supervision.

Fresh KV reaches these results with two scans and two sequential calls, versus four/five scans and six/eight calls (Figure 3, Appendix J.2). This gives the shared block useful historical context through a training schedule with fewer repeated scans.

How does the model handle prompt prefill at inference time? If exact execution is strictly serial token-by-token [...]?

Could you provide actual wall-clock throughput and peak GPU memory for same-depth, full cross-depth, and selective memory [...]?

We understand this as asking whether cross-depth access admits fast prefill, including its execution costs and subsequent prediction quality.

The 100M/200M Single-Layer checkpoints obtain lower prefill latency with closely matched mean continuation NLL using two-pass Fresh KV. Measurements use one H100, batch 1, 4096 prompt tokens and BF16 execution/KV.

ModelExact / two-pass prefill (s)Exact / two-pass peak GiB
100M W3256.586 / 0.0770.716 / 1.079
200M W3287.725 / 0.2381.400 / 2.182

Exact prefill executes prompt tokens serially. Fresh KV uses two depth-major passes with parallel prompt positions, fresh shallow K/V and previous-pass deeper estimates. Continuation uses the second pass's cache and exact single-token steps.

With 4032-token prompts and 64 teacher-forced continuation tokens across 32 windows, exact/two-pass mean NLL is 1.80309/1.80593 at 100M and 1.53873/1.53864 at 200M. First-token NLL changes are +0.03888/-0.00824.

The requested same-depth/full/W8 comparison uses the earlier 20M cohort on the same H100, batch 1, 256 prompt tokens and BF16 execution/KV.

ExecutionPrefill msPrefill tok/sDecode tok/sPeak GiB
Same-depth, exact18.1514107.189.70.162
Full, exact2845.1390.089.00.162
Full, two-pass43.895832.489.00.162
W8, exact2071.38123.6122.50.159
W8, two-pass39.436492.4121.00.159

Prefill includes cache construction and final logits. Medians use one warmup and three repeats for main models, five for compact models. Decode generates 32 tokens in each of three repeats. Memory is PyTorch peak allocated during prefill.

For the earlier 20M full/W8 checkpoints, two-pass mean NLL rises by +0.1070/+0.2840 after 224-token prompts (32 windows, 32 continuation tokens). The completed 100M/200M models prefill 4096 tokens in less than one second. Their continuation checks connect parallel construction of historical context to its use by subsequent shared updates.

On top of that, Tab. 2 only reports analytic matmul counts [...] with zero actual latency, throughput, or peak memory numbers.

The measurements above provide actual prefill latency, decode throughput and peak allocated memory. Peak allocated memory includes weights, activations and cache allocations in this implementation.

The original same-depth kernel computed dense attention before masking, which explains its identical analytic inference count to full cross-depth. Figure 2 separately counts historical storage. At sixteen updates and a 4096-token history, Single-Layer's W32 policy retains 7.0% of full cross-depth historical K/V records (4,576 vs 65,536).

The timings and memory quantify complete execution, while the record counts show how selecting historical context reduces storage for the shared update architecture. This makes its practical quality and resource tradeoff inspectable.

AuthorsOpenReview ↗

Response to Lnv1 (3/3): longer training, refinement, and downstream results

At 450M tokens, could the 11.8%--15.5% shared-bank deficit simply be router under-training or load-balancing friction [...]?

Do you have expert utilization stats or longer token-horizon runs?

We examine shared-bank quality with a longer training horizon and explicit bank/routing controls. The completed 100M/4B and 200M/8B models use context 4096, a shared bank of 24 experts and about 40 training tokens per total parameter (Section 3, Appendix J).

At 100M, Single-Layer gives lower CE (1.9440 vs 1.9709) and a higher seven-task mean (39.94 vs 39.17) than the 105M Standard Transformer. At 200M, its mean is 42.28 vs 42.37 for the 288M control with 30.7% fewer parameters.

The 4B-token control then studies FFN allocation at fixed one-pass same-depth access and shared attention. Depth-specific banks give lower CE (1.9956 vs 2.0152) and a higher seven-task mean (38.98 vs 38.53) than the shared bank (Section 4.1, Table 2, Figure 5). Appendix J specifies router projections and auxiliary-loss grouping. This quantifies dedicated FFNs under a specified historical context.

Routing also affects shared processing. In the three-seed compact 20M study, orthogonal-code routing gives lower shared-bank PPL (32.22 vs 32.67) than logarithmic-code routing. The depth-specific reference gives 32.58 (Table 9b, Appendix H).

Shared updates can prepare gains through later computation. On the completed 100M checkpoint, with historical K/V fixed, the third update's immediate NLL reduction averages -0.0035 nat, while skipping it raises final NLL by 0.1453 nat. At 18 of 64 positions, its immediate readout worsens while its final contribution is beneficial. The intervention skips a complete update and executes its suffix (Section 4.4, Figure 8, Appendix K).

The longer runs demonstrate useful shared processing. The bank/routing controls assess refinement under a fixed historical read rule, and the token intervention judges shared updates by their contribution to the final prediction. Together they support studying FFN reuse separately from the context attention supplies.

All main experiments use a very short ctx length [...] a small token budget [...] a single seed, and Dev-only PPL with no downstream evaluations.

The completed Single-Layer models now have seven-task zero-shot evaluation. The 100M prediction gain and 200M parameter benefit extend to downstream capability. The studies use context 4096 and 4B/8B training tokens (Table 1).

Every model scores the same 19,149 documents and 68,428 candidate continuations with its own tokenizer and causal graph. Appendix I specifies metrics and numerical checks. Table 1 also includes nine re-evaluated public checkpoints and their released training budgets.

Each 100M/200M run uses one seed. The compact routing study uses seeds 42, 43 and 44, with paired windows within each seed. The paired 100M results connect better language-modeling quality to a higher downstream mean. At 200M, the near-matching mean with 30.7% fewer parameters demonstrates the resource benefit of selected historical context combined with FFN reuse.

The text also contains several odd notes about missing experimental logs [...] which makes the experimental record feel a bit unfinished.

The current 100M/200M conclusions use complete, traceable evaluation records. Each archived configuration identifies the training data, checkpoint, historical read rule, FFN allocation and evaluation graph. Exact CE uses 131,072 held-out tokens. The downstream archive retains per-task scores and fixed evaluation requests, linking each reported result to its actual inputs and configuration (Appendix J).

The new boundary follow-up reports paired training-graph and exact-graph scores, as detailed in our first reply. Recovered W2/W3 evaluations and the fully untied equal-compute endpoint provide completed compact-model comparisons. Appendix G documents the original settings.

These records make both architectural choices and their predictive effects traceable, so readers can verify the comparisons supporting separate historical-source and FFN-allocation decisions.

Reviewer_ctHUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors introduce the single layer formulation for recurrent transformers. Specifically, the authors reframe the transformer encoding process as a stream of token states through history. Through this relatively simple reframing, the authors are able to systematically examine various techniques used for making looping transformers, such as attending to higher depth token states, attending to a subsequence's higher depth token states, and various loop injection methods. Furthermore, the author introduces a new even-odd training paradigm, whereby every other tokens are encoded in parallel for each round through a few iterations with gradients passing through all in order to supervise final states. The authors demonstrate that almost all circumstances, attending to future state of past tokens provides gains to performance.

Critical strengths
  • As the authors suggest, their reframing of the attention procedure allows for the flexible redefinition of methods that are usually tied such as historical residual passthrough or skipping states.
  • The authors introduce an interesting even-odd training paradigm inspired by Gauss-Seidel iteration that seems reasonably general for incorporating historical state injection as a part of their training.
  • The authors so-called "long-short term design" where attention is performed from later stages to early stages only for the recent few tokens yields only small perplexity drop while dramatically saving memory.
Critical weaknesses
  • Although the framework for training is nice, I'm not sure I see a particular payoff for this reframing; in particular, most approaches proposed here beyond even-odd training seems to already exist in literature.
  • There really isn't a lot of insights as to why cross depth attention worked in this case. The authors' empirical evidence seems to generally point to simply adding more information improves quality; the one case where this isn't true is when the author calibrated under held-out test data under the same distribution, which indicates that all depth is not necessary but is not fully explored.
  • I would've like to see more validation on the even-odd training, with respect to matched baselines.
Questions for authors
  • How does the proposed system of long-short term design change as a function of the amount of context inclusion? In particular, recent [1] works demonstrate that only a few tokens (2-3) of historical state access is required for performance gains seen here?
  • How does the method of even-odd training compare to actually training FLOPs matched / parameter matched (through additional MoEs in the baseline to boost parameters artificially) untied approaches? What can this tell us about the efficiency of the even-odd approach?

[1] Wang, X., et al., "Full-bandwidth transformer", arXiV preprint 2608.08888 (2026).

2 comments
AuthorsOpenReview ↗

Response to ctHU (1/2): motivation, payoff, and historical information

Thank you for these questions.

Although the framework for training is nice, I'm not sure I see a particular payoff for this reframing; in particular, most approaches proposed here beyond even-odd training seems to already exist in literature.

The Single-Layer model, at 100M/4B tokens, achieves lower CE (1.9440 vs 1.9709) and a higher seven-task mean (39.94 vs 39.17) than a 105M Standard Transformer. At 200M/8B tokens, it scores 42.28 vs 42.37 for the 288M Standard Transformer, using 30.7% fewer parameters. Its sixteen-update cache retains 7.0% of full cross-depth historical records at a 4096-token history (Section 3, Figures 1, 2 and 4, Table 1).

Our motivation is to reconsider how communication and refinement are coupled in looped-model design. Communication chooses the historical token-depth states available to attention. Refinement chooses the FFN or expert bank. During cached decoding, past tokens have finished computing while the current token is still evolving, allowing historical inputs to be chosen separately from FFN reuse.

Shared updates and historical feedback have precedents in Universal, Feedback and Full-bandwidth Transformers. Single-Layer makes the two choices separately testable. In 100M one-pass controls, updated history lowers CE from 2.1884 to 2.0173 with the shared bank fixed. Under a separate fixed same-depth read rule, depth-specific FFNs lower CE from 2.0152 to 1.9956. These controls identify useful history for shared processing and quantify dedicated FFNs under a specified context (Sections 4.1-4.2).

There really isn't a lot of insights as to why cross depth attention worked in this case. The authors' empirical evidence seems to generally point to simply adding more information improves quality; [...]

Deeper historical context steers early expert selection with current-token identity and position fixed. We swap historical K/V between matched contexts with the same current token ID and absolute position. At j=1 (0-based), swapping recent-8 history at depths 4-7 recovers the donor expert in 66/74 divergent cases (89.2%), versus 5/74 (6.8%) at depths 0-3. Both replace 32 K/V slots.

This uses a 20M R8 full-cross-depth checkpoint, context 256 and 400M training tokens, with 164 matched pairs and 328 directed swaps. The 74 cases are the j=1 subset of the 522 Full-model native routing disagreements in Table 14b (Appendix H.4).

Late-value direction swaps with recipient norms fixed recover 58/74 choices, while norm-only swaps recover 2/74. Standard Transformers read same-layer history. Cross-depth access lets an early shared update read deeper states already computed for past tokens, exposing context that changes its expert selection.

The 100M one-pass source control then tests prediction quality with the shared bank, recent reads and per-query historical key count fixed.

Distant sourceExact CE at 4BTokens to CE 2.40Tokens to CE 2.30
Token embedding2.18842.0B2.6B
State after min(j,3) updates2.01730.8B1.2B

Here j is query depth. The policies retain depths {0,1,2,3} and {0} across queries. Thresholds use the first evaluated checkpoint. The embedding control requires 2.17-2.50x as many training tokens to reach the targets. Selecting the historical computation stage improves prediction while preserving FFN reuse (Section 4.2, Figure 6).

the one case where this isn't true is when the author calibrated under held-out test data under the same distribution, which indicates that all depth is not necessary but is not fully explored.

If "calibrated" refers to the submitted Section 5.2 boundary experiment, its policies were trained separately and evaluated on held-out Dev data. The current fixed-checkpoint interventions examine which receiving updates benefit from a retained distant state.

A distant state written after four updates helps prediction at later receiving updates. On the completed 200M model, historical K/V is frozen and the read set changes at 776 positions. The distant source stays at z4. Removing all distant reads raises mean final target NLL by 0.4506 nats.

Receiving updates 12-15 account for 74% of the summed single-update removal cost (Section 4.3, Figure 7). Replay has a 0.0100-nat mean absolute NLL mismatch against exact execution (Appendix L). Thus the source's write depth and the updates that benefit from reading it are distinct.

A complementary exploratory compact checkpoint tests historical content with native routing replayed. Masking history raises CE from 3.4541 to 5.1124. Replaying its original expert IDs and mixture weights gives 4.9738 after masking (Appendix H.3). Historical content contributes beyond preserving routing choices.

These interventions connect selected historical reads to the final prediction while FFN reuse is retained, supporting separate choices of readable context and shared processing.

AuthorsOpenReview ↗

Response to ctHU (2/2): historical access and training efficiency

How does the proposed system of long-short term design change as a function of the amount of context inclusion? In particular, recent [1] works demonstrate that only a few tokens (2-3) of historical state access is required for performance gains seen here?

We understand this question as asking how quality changes with historical access, and whether only two or three historical tokens suffice for the gains.

In Single-Layer, W counts recent tokens with multi-depth access. Older tokens remain readable through a retained state, including at W2/W3. In Full-bandwidth Transformer [1], two/three concerns prefill passes. Its recurrent-suffix probe applies feedback at the last k={1,2,4} positions while retaining the earlier prefix.

We therefore test W2/W3 as small recent multi-depth windows with distant history retained. In the 20M, context-256 sweep, W2/W3 reach PPL 32.27/31.88, versus 31.28 for full cross-depth. The training budget is 400,031,744 tokens. W2/W3 have paired exact/training-graph evaluation on the same 128 validation windows, with absolute CE gaps 0.00269/0.00289.

Recent windowExact PPLHistorical K/V records
W=232.27270
W=331.88277
W=831.17312
W=1631.71368
W=3231.45480
Full cross-depth31.282048

The curve is non-monotonic. W8 gives the lowest measured PPL and uses fewer historical K/V records (312 vs 2048) than full cross-depth.

At the larger scales, Single-Layer's W32 policy gives near-full cross-depth quality. At 100M, CE differs by at most 0.0093 over common 0.4B-1.8B checkpoints. At 200M, it differs by 0.0136 at the unrestricted run's 2.4B-token endpoint. Pass count and loss weights are fixed (Table 6, Figure 14).

At sixteen updates and a 4096-token history, W32 retains 7.0% of full cross-depth historical K/V records (4,576 vs 65,536) (Figure 2). These results support concentrating multi-depth access on recent tokens while retaining one updated state per distant token, preserving useful context for shared processing with a smaller cache.

How does the method of even-odd training compare to actually training FLOPs matched / parameter matched (through additional MoEs in the baseline to boost parameters artificially) untied approaches? What can this tell us about the efficiency of the even-odd approach?

The completed Single-Layer models demonstrate the quality achievable with selected history and FFN reuse. At 100M/4B tokens, they give lower CE (1.9440 vs 1.9709) and a higher seven-task mean (39.94 vs 39.17) than the 105M Standard Transformer. At 200M/8B, the mean is 42.28 vs 42.37 for the 288M control with 30.7% fewer parameters. These comparisons match training tokens. Single-Layer uses two Fresh KV training passes and the controls use one exact pass.

The original equal-compute comparison uses about 3.4035e17 analytic training FLOPs. Ten-pass cross-depth reaches PPL 31.2819 at 0.400B tokens, one-pass same-depth reaches 27.5167 at 2.455B, and a fully untied dense Standard Transformer reaches 25.4879 at 2.753B. The dense control has eight independent attention/FFN sets, 19.97M parameters (1.16% fewer), width 320 and a dense FFN. The shared reference has 20.20M parameters, width 512 and MoE.

This budget comparison motivates reducing repeated state-estimation scans. An even/odd pair updates every token once, and the second half uses states refreshed by the first. Fresh KV makes shallow states fresh within each pass. At 185,057,280 tokens, its two-scan schedule gives lower CE (2.8341 vs 2.9163/2.8703) than the older four/five-scan schedules. These compare complete schedules, including their supervision distributions (Figure 3, Appendix J.2).

Progressive pass schedules, as explored in [1], are a further candidate for this architecture. The 100M one-pass source control already reaches CE 2.40 with less data (0.8B vs 2.0B tokens) at fixed shared bank and per-query key count (Figure 6). Historical-source selection thus improves data efficiency while retaining the shared FFN bank.

[1] Wang et al., Full-bandwidth Transformer, arXiv:2608.08888v2, Sections 3.1-3.2 and Appendix N.

Reviewer_ZQtoOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper studies looped transformers and aims to disentangle the benefits arising from "communication" - which past KV computations can the current step attend to, and "refinement" - which MoE experts are best for which loop step of the decoding forward pass. The paper claims that prior work makes these two changes simultaneously without ablating which category of changes is more crucial. They perform ablations using a 20M active 274M total parameter model looped 16 times with a context window of 256. The model is trained for 400-450M tokens. Their main findings are:

  1. Cross-depth KV access reduced perplexity by 2-8%.
  2. DEpth-specific choosing of MoE experts improves pplx by ~11%
  3. Cross-depth access can be restricted to just the recent tokens whereas keeping long-range tokens the same-depth KV only. This doesnt lose much in quality but reduces KV significantly.
Critical strengths

The paper studies important architectural choices around looped transformers which is a topic lot of researchers are interested in.

Critical weaknesses
  • The paper is poorly written. It is hard to understand and read.
  • The design of the experimental evaluation feels a bit haphazard and random. The ablations do not seem thorough and the context explored is only 256 tokens.
  • The main message of the paper is unclear. They set out with a goal to disentangle two sources of gains in looped models but it is unclear what the final conclusion is?
Questions for authors
  1. The write-up needs a lot of work (From humans). In its current form, it is very poorly written and hard to read.
  2. Do you have any ablations at context windows larger than 256 tokens?
3 comments
AuthorsOpenReview ↗

Response to ZQto: architectural motivation and systematic evidence

Thank you for asking us to clarify the research question, conclusions and experimental organization.

The paper is poorly written. It is hard to understand and read.

We have rewritten the Introduction, method presentation and Conclusion around choosing historical context separately from the FFN that processes the state. Cached decoding supplies the motivation. Earlier tokens have finished computing while the current token is still being updated, so their completed representations are available to a shared update.

Section 2 defines which historical states to read and which FFN to apply. Figure 1 connects these choices to the 100M prediction gain and 200M parameter savings, giving the reader a concrete method and its practical payoff before the controls and mechanisms.

The design of the experimental evaluation feels a bit haphazard and random. The ablations do not seem thorough and the context explored is only 256 tokens.

The revised evaluation separately tests historical-source selection, FFN allocation and the complete Single-Layer model. The two one-pass controls use 100M models, context 4096 and 4B training tokens (Table 2).

ChangeExact CE, before → afterSeven-task mean (%), before → after
Distant embeddings → updated states2.1884 → 2.017338.47 → 38.95
Shared bank → depth-specific banks2.0152 → 1.995638.53 → 38.98

The source comparison fixes the shared bank, recent same-depth reads and historical key count per query. Its distant source is the state after min(j,3) updates, where j is query depth. Across queries the policies retain different source-depth sets. The bank comparison fixes same-depth access and shared attention. Appendix J details the router and auxiliary-loss grouping changes.

Updated history gives lower CE while retaining the shared FFN bank. The embedding-source control also requires 2.17-2.50x as many training tokens at the same two CE targets (Section 4.2, Figure 6). The bank control then measures the benefit of dedicating FFNs under a fixed read rule.

Table 1 evaluates the complete model, including its two-pass training and loss weighting. Sections 4.3-4.4 intervene on fixed checkpoints to locate useful reads and updates. This connects each design choice to prediction changes and then to the complete model's quality.

The main message of the paper is unclear. They set out with a goal to disentangle two sources of gains in looped models but it is unclear what the final conclusion is?

Historical access and FFN reuse should be chosen separately and assessed by the final prediction. Communication specifies the token-depth states attention can read. Refinement specifies the FFN or expert bank applied to the current state.

In the two one-pass controls above, updated historical sources lower CE by 0.1711 with the shared bank fixed. Depth-specific FFNs lower CE by 0.0196 in the separate same-depth control. Each comparison supplies a design judgment under its own read rule.

Single-Layer combines selected history with shared processing. At matched data, the 100M model gives lower CE and a higher downstream mean than the 105M Standard Transformer. At 200M, it reaches a comparable seven-task mean to the 288M control with 30.7% fewer parameters. These results demonstrate useful shared computation supplied with selected historical context.

Do you have any ablations at context windows larger than 256 tokens?

Yes. Both one-pass controls and the completed 100M/200M studies use context 4096.

ModelTraining tokensExact CESeven-task mean (%)
99.84M Single-Layer4B1.944039.94
105.41M Standard Transformer4B1.970939.17
199.89M Single-Layer8B1.685042.28
288.64M Standard Transformer8B1.674642.37

Within each scale, the pairs share the data manifest, optimizer and global batch. Exact CE uses 131,072 held-out tokens. Single-Layer uses two training passes and the Standard Transformers use one. Each main run uses one seed. Appendix J specifies configurations, and Figures 2 and 4 show cache counts and learning curves.

At 100M, the first measured checkpoint below CE 2.10 uses fewer training tokens (2.2B vs 2.4B). At 200M, the seven-task mean differs by 0.09 points with 30.7% fewer parameters (Section 3, Table 1). The selected-history/shared-FFN design therefore has measured quality and resource benefits in the completed long-context studies.

Reviewer_ZQtoOpenReview ↗

Thanks for your response

Thanks to the authors for the response. I could not read the entire rewrite again carefully but the introduction seems more readable currently. I will update my score to 2.

Thank you for revisiting the manuscript and for your feedback on the revised introduction. We appreciate your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of the revision and additional evidence

Thank you for the careful reviews and follow-up discussion. Lnv1 confirms that most concerns and questions have been addressed by the clarifications and experiments. ZQto finds the revised introduction more readable. Both reviewers have raised their scores. Lnv1's follow-up ZQto's follow-up

Selecting historical context separately from FFN reuse gives measured prediction and resource benefits at 100M and 200M. Sections 1-2 explain Single-Layer, Section 3 reports the completed larger-scale results, and Section 4 tests how historical reads and shared updates help prediction. Appendix G documents the original experiments.

Core idea and completed results

During cached decoding, earlier tokens have completed their computations while the current token is still evolving. Communication chooses the historical token-depth states readable by attention. Refinement chooses the FFN or expert bank applied to the current state. Single-Layer shares attention and an expert bank, reads all recent depths and retains one updated state per distant token.

  • At 100M/4B tokens, Single-Layer achieves lower CE (1.9440 vs 1.9709) and a higher seven-task mean (39.94 vs 39.17) than the 105M Standard Transformer. Its first evaluated checkpoint below CE 2.10 uses fewer training tokens (2.2B vs 2.4B).
  • At 200M/8B tokens, it reaches a near-matching seven-task mean of 42.28 vs 42.37 for the 288M Standard Transformer with 30.7% fewer parameters. At sixteen updates and a 4096-token history, its cache retains 7.0% of full cross-depth historical K/V records.

Both scales use context 4096 and about 40 training tokens per total parameter. Each main run uses one seed. The compact 20M routing study uses seeds 42, 43 and 44. Table 1 includes seven tasks and nine public checkpoints under a fixed scoring protocol, alongside their released training budgets.

Findings that support the two design choices

  • With the shared bank, one-pass training and historical key count per query fixed, updated distant states lower CE from 2.1884 to 2.0173 at 4B tokens. At the first measured checkpoints reaching two CE targets, the embedding-source control requires 2.17-2.50x as many training tokens. Useful history improves prediction while preserving FFN reuse (Section 4.2, Figure 6).
  • At fixed same-depth access and shared attention, depth-specific FFNs lower CE from 2.0152 to 1.9956 and raise the seven-task mean from 38.53 to 38.98 in the 100M/4B control. This measures FFN allocation under a specified historical read rule (Section 4.1, Table 2).
  • Deeper historical context steers early expert selection. In matched 20M R8 cache swaps, current token ID and position are fixed. At j=1, deeper recent-history sources recover the donor choice in 66/74 divergent decisions, versus 5/74 for shallower sources at the same substituted-slot count. This identifies a path from completed historical computation to early shared processing (Appendix H.4).
  • Shared updates can prepare later prediction gains. On the completed 100M checkpoint, the third update's immediate NLL reduction averages -0.0035 nat, while skipping it raises final NLL by 0.1453 nat. The intervention measures its contribution through later computation to the final prediction (Section 4.4, Figure 8).
  • Fresh KV achieves lower Exact CE with fewer sequential scans. At 185,057,280 tokens, two scans reach CE 2.8341, versus 2.9163/2.8703 for the older four/five-scan schedules. These compare complete training schedules (Figure 3, Appendix J.2).
  • Fresh KV also gives lower prefill latency with closely matched mean continuation NLL on the current 100M/200M checkpoints. At batch 1 on one H100, 4096-token prefill takes 0.077/0.238 s. After 4032-token prompts, mean NLL changes by +0.00284/-0.00009 across 64 teacher-forced tokens and 32 windows. The replies also report same-depth/full/W8 timings and approximation errors for the earlier 20M cohort.

Additional recipe checks answer the original training concerns. At the same analytic training-FLOP budget, the original 20M one-pass same-depth and fully untied dense controls obtain PPL 27.5167/25.4879, versus 31.2819 for ten-pass cross-depth. A newly trained 274M, context-256 boundary pair improves Exact PPL from 28.3792 to 27.5258 with added anchors and graph CE gaps below 0.001. Their configurations appear in the replies.

The source and bank controls make communication and refinement separately testable. The complete models demonstrate the practical value of combining selected historical context with shared processing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.