- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies looped MoE Transformers through a unified Single-Layer framework that decouples communication (which historical token-depth states are readable via a shared attention module) from refinement (how FFN expert banks are scheduled across recurrent steps). Using an alternating even/odd parallel training scheme and exact token-major serial evaluation on FineWeb-Edu (at 274M and 20M resident params, ctx length ), the authors show that cross-depth historical access consistently lowers Dev PPL across all FFN schedules, while depth-specific expert banks still beat a single shared bank. They further test a selective memory policy (retaining full depth trajectories for the last tokens and a single shallow anchor for older tokens), cutting analytic KV state storage by 84.8% at w/o hurting PPL.
Critical strengths
- The paper cleanly separates cross-depth KV access from FFN sharing in looped MoE transformers. This makes it easy to see how historical context access and expert scheduling each affect performance.
- The selective window and anchor memory idea is very practical. It keeps full depth states only for recent tokens and one shallow state for older tokens, which cuts KV cache size a lot without hurting perplexity.
Critical weaknesses
My biggest concern is the gap between the paper's training/inference mechanics and practical efficiency, combined with the lack of wall-clock or compute-matched baselines. Exact causal execution (Sec. 3.2) is strictly token-major because shallow steps of token read deeper steps of token , which means even prompt prefill cannot be parallelized across tokens w/o falling back to the multi-pass approximation. Meanwhile, parallel training requires 10 sequential passes per step (approx. 6 full-scan equivalents in FLOPs, Sec. D.2), yet all comparisons are only token-matched (400M--450M tokens) rather than training-FLOP matched. On top of that, Tab. 2 only reports analytic matmul counts where same-depth and full cross-depth are listed at the exact same 0.061 GFLOPs/token simply because the current kernel computes dense attention before masking (line 795), with zero actual latency, throughput, or peak memory numbers.
Second, the headline result in Sec. 5.2 that retaining all tested boundaries hurts PPL by 10.5%--90.8% (Fig. 4b, Tab. 6) looks heavily confounded by the training approximation. In Tab. 6, adding makes the shared-bank model blow up from 28.28 to 53.96 PPL, whereas in Tab. 1, full cross-depth (which exposes all 16 depths) reaches 24.32 PPL on the same 274M model. Looking at App. F.2, Tab. 6 switched to a 2-sweep block-Jacobi training contract (and lacks paired parallel/exact gap checks, line 904), whereas the no-extra-anchor baseline used exact-graph training and Tab. 1 used a schedule. It seems very likely that the 53.96 PPL spike is an optimization/Jacobi discrepancy artifact from exposing unconverged deep states () under only 2 sweeps, rather than more historical information inherently hurting the architecture.
Finally, the empirical scope feels somewhat thin and incomplete in places. All main experiments use a very short ctx length (), a small token budget (400M--450M tokens, which is quite undertrained for a 274M MoE pool with 48 experts and likely exaggerates the gap between shared routing and depth-specific banks), a single seed, and Dev-only PPL with no downstream evaluations. The text also contains several odd notes about missing experimental logs (e.g., lines 613--614, Tab. 4 missing train PPL, Tab. 7 missing the endpoint, and line 904 missing parallel/exact scores for Tab. 6), which makes the experimental record feel a bit unfinished.
Questions for authors
-
In Tab. 6, why does retaining degrade shared-bank Exact Dev PPL to 53.96 when retaining all 16 depths in Tab. 1 achieves 24.32? Can you report the parallel-vs-exact discrepancy for the Tab. 6 runs, and check if that degradation disappears under the stronger alternating schedule used in Tab. 1?
-
How does the model handle prompt prefill at inference time? If exact execution is strictly serial token-by-token (Fig. 2c), doesn't prefill latency scale sequentially with ? Could you provide actual wall-clock throughput and peak GPU memory for same-depth, full cross-depth, and selective memory ()?
-
In the 274M study, each expert in the shared bank is routed from a 48-expert pool at every step, whereas only routes among 3 experts per depth. At 450M tokens, could the 11.8%--15.5% shared-bank deficit simply be router under-training or load-balancing friction in a larger action space? Do you have expert utilization stats or longer token-horizon runs?
-
Since same-depth access does not have cross-token depth dependencies, it can be trained in a single exact pass rather than 10 alternating passes (~ fewer training scans). How does same-depth compare to cross-depth when matched on total training FLOPs instead of training tokens?
5 comments
Response to Lnv1 (2/3): Fresh KV training and parallel prefill
Meanwhile, parallel training requires 10 sequential passes per step (approx. 6 full-scan equivalents in FLOPs, Sec. D.2), yet all comparisons are only token-matched (400M--450M tokens) rather than training-FLOP matched.
Fresh KV supplies updated historical states with lower Exact CE and fewer full-token scans at the common measured checkpoints. At query depth j, sources up to j come from the current pass, while deeper sources use previous-pass estimates. Later depths can use fresh shallow states within the same scan.
The 100M, R=8, context-4096 comparison uses 185,057,280 training tokens per row. Exact evaluation uses the same 32 windows and 131,072 scored tokens.
| Training schedule | Full-token scans / sequential calls | Exact CE |
|---|---|---|
| Fresh KV, two full passes | 2 / 2 | 2.8341 |
| Older alternating schedule | 4 / 6 | 2.9163 |
| Older alternating schedule | 5 / 8 | 2.8703 |
At 2,000,076,800 tokens, two/four scans reach CE 2.1233/2.1442. All schedules supervise updated positions and differentiate every update. This compares complete schedules, including their distributions of supervision.
Fresh KV reaches these results with two scans and two sequential calls, versus four/five scans and six/eight calls (Figure 3, Appendix J.2). This gives the shared block useful historical context through a training schedule with fewer repeated scans.
How does the model handle prompt prefill at inference time? If exact execution is strictly serial token-by-token [...]?
Could you provide actual wall-clock throughput and peak GPU memory for same-depth, full cross-depth, and selective memory [...]?
We understand this as asking whether cross-depth access admits fast prefill, including its execution costs and subsequent prediction quality.
The 100M/200M Single-Layer checkpoints obtain lower prefill latency with closely matched mean continuation NLL using two-pass Fresh KV. Measurements use one H100, batch 1, 4096 prompt tokens and BF16 execution/KV.
| Model | Exact / two-pass prefill (s) | Exact / two-pass peak GiB |
|---|---|---|
| 100M W32 | 56.586 / 0.077 | 0.716 / 1.079 |
| 200M W32 | 87.725 / 0.238 | 1.400 / 2.182 |
Exact prefill executes prompt tokens serially. Fresh KV uses two depth-major passes with parallel prompt positions, fresh shallow K/V and previous-pass deeper estimates. Continuation uses the second pass's cache and exact single-token steps.
With 4032-token prompts and 64 teacher-forced continuation tokens across 32 windows, exact/two-pass mean NLL is 1.80309/1.80593 at 100M and 1.53873/1.53864 at 200M. First-token NLL changes are +0.03888/-0.00824.
The requested same-depth/full/W8 comparison uses the earlier 20M cohort on the same H100, batch 1, 256 prompt tokens and BF16 execution/KV.
| Execution | Prefill ms | Prefill tok/s | Decode tok/s | Peak GiB |
|---|---|---|---|---|
| Same-depth, exact | 18.15 | 14107.1 | 89.7 | 0.162 |
| Full, exact | 2845.13 | 90.0 | 89.0 | 0.162 |
| Full, two-pass | 43.89 | 5832.4 | 89.0 | 0.162 |
| W8, exact | 2071.38 | 123.6 | 122.5 | 0.159 |
| W8, two-pass | 39.43 | 6492.4 | 121.0 | 0.159 |
Prefill includes cache construction and final logits. Medians use one warmup and three repeats for main models, five for compact models. Decode generates 32 tokens in each of three repeats. Memory is PyTorch peak allocated during prefill.
For the earlier 20M full/W8 checkpoints, two-pass mean NLL rises by +0.1070/+0.2840 after 224-token prompts (32 windows, 32 continuation tokens). The completed 100M/200M models prefill 4096 tokens in less than one second. Their continuation checks connect parallel construction of historical context to its use by subsequent shared updates.
On top of that, Tab. 2 only reports analytic matmul counts [...] with zero actual latency, throughput, or peak memory numbers.
The measurements above provide actual prefill latency, decode throughput and peak allocated memory. Peak allocated memory includes weights, activations and cache allocations in this implementation.
The original same-depth kernel computed dense attention before masking, which explains its identical analytic inference count to full cross-depth. Figure 2 separately counts historical storage. At sixteen updates and a 4096-token history, Single-Layer's W32 policy retains 7.0% of full cross-depth historical K/V records (4,576 vs 65,536).
The timings and memory quantify complete execution, while the record counts show how selecting historical context reduces storage for the shared update architecture. This makes its practical quality and resource tradeoff inspectable.
Response to Lnv1 (3/3): longer training, refinement, and downstream results
At 450M tokens, could the 11.8%--15.5% shared-bank deficit simply be router under-training or load-balancing friction [...]?
Do you have expert utilization stats or longer token-horizon runs?
We examine shared-bank quality with a longer training horizon and explicit bank/routing controls. The completed 100M/4B and 200M/8B models use context 4096, a shared bank of 24 experts and about 40 training tokens per total parameter (Section 3, Appendix J).
At 100M, Single-Layer gives lower CE (1.9440 vs 1.9709) and a higher seven-task mean (39.94 vs 39.17) than the 105M Standard Transformer. At 200M, its mean is 42.28 vs 42.37 for the 288M control with 30.7% fewer parameters.
The 4B-token control then studies FFN allocation at fixed one-pass same-depth access and shared attention. Depth-specific banks give lower CE (1.9956 vs 2.0152) and a higher seven-task mean (38.98 vs 38.53) than the shared bank (Section 4.1, Table 2, Figure 5). Appendix J specifies router projections and auxiliary-loss grouping. This quantifies dedicated FFNs under a specified historical context.
Routing also affects shared processing. In the three-seed compact 20M study, orthogonal-code routing gives lower shared-bank PPL (32.22 vs 32.67) than logarithmic-code routing. The depth-specific reference gives 32.58 (Table 9b, Appendix H).
Shared updates can prepare gains through later computation. On the completed 100M checkpoint, with historical K/V fixed, the third update's immediate NLL reduction averages -0.0035 nat, while skipping it raises final NLL by 0.1453 nat. At 18 of 64 positions, its immediate readout worsens while its final contribution is beneficial. The intervention skips a complete update and executes its suffix (Section 4.4, Figure 8, Appendix K).
The longer runs demonstrate useful shared processing. The bank/routing controls assess refinement under a fixed historical read rule, and the token intervention judges shared updates by their contribution to the final prediction. Together they support studying FFN reuse separately from the context attention supplies.
All main experiments use a very short ctx length [...] a small token budget [...] a single seed, and Dev-only PPL with no downstream evaluations.
The completed Single-Layer models now have seven-task zero-shot evaluation. The 100M prediction gain and 200M parameter benefit extend to downstream capability. The studies use context 4096 and 4B/8B training tokens (Table 1).
Every model scores the same 19,149 documents and 68,428 candidate continuations with its own tokenizer and causal graph. Appendix I specifies metrics and numerical checks. Table 1 also includes nine re-evaluated public checkpoints and their released training budgets.
Each 100M/200M run uses one seed. The compact routing study uses seeds 42, 43 and 44, with paired windows within each seed. The paired 100M results connect better language-modeling quality to a higher downstream mean. At 200M, the near-matching mean with 30.7% fewer parameters demonstrates the resource benefit of selected historical context combined with FFN reuse.
The text also contains several odd notes about missing experimental logs [...] which makes the experimental record feel a bit unfinished.
The current 100M/200M conclusions use complete, traceable evaluation records. Each archived configuration identifies the training data, checkpoint, historical read rule, FFN allocation and evaluation graph. Exact CE uses 131,072 held-out tokens. The downstream archive retains per-task scores and fixed evaluation requests, linking each reported result to its actual inputs and configuration (Appendix J).
The new boundary follow-up reports paired training-graph and exact-graph scores, as detailed in our first reply. Recovered W2/W3 evaluations and the fully untied equal-compute endpoint provide completed compact-model comparisons. Appendix G documents the original settings.
These records make both architectural choices and their predictive effects traceable, so readers can verify the comparisons supporting separate historical-source and FFN-allocation decisions.
Response to Lnv1 (1/3): stronger training and equal-compute results
Thank you for asking us to check the training schedule and the equal-compute comparison.
We understand the concern as whether added historical boundaries help under stronger state estimation and exact execution. We test this with a newly trained paired follow-up.
Under the stronger schedule, added anchors give lower Exact PPL (27.5258 vs 28.3792), with both absolute graph CE gaps below 0.001. The pair uses a 274.43M shared-bank model, R=16, context 256, W32 and 400,031,744 training tokens.
Both arms retain the segment-entry state matched to the query and are based on the all-boundary configuration. Their model/training configurations differ only in the added-anchor policy. The differentiable serial warm start and ten gradient-tracked updates cost seven full-token scans. Both use the same 128 validation windows.
The all-boundary arm finished its last 132 updates on H800 with a different PyTorch version, while z0 stayed on H100. Appendix G documents the original 53.96 endpoint and its earlier settings.
The new pair shows a prediction benefit from richer historical access under exact execution while the expert bank remains shared. It supplies a controlled design judgment about communication within shared refinement.
The current architectural payoff is that useful history can improve a shared bank even within one-pass training. At fixed bank and per-query historical key count, the 100M source control reaches CE 2.40 with less data (0.8B vs 2.0B tokens), and CE 2.30 with 1.2B vs 2.6B tokens (Figure 6).
For the original 20M recipe, the one-pass controls achieve lower PPL at equal analytic training FLOPs.
The cross-depth reference has 20,203,648 parameters, R=8 and context 256. The same-depth control shares attention, differentiates all eight updates and retains the original routing. Warmup and cosine follow the same fraction of the compute budget. Integer-step rounding changes budgets by at most 0.00052%. All endpoints use 128 context-256 windows. FLOPs count logical forward/backward matrix work.
The dense Standard Transformer has eight independent attention/FFN sets, 19.97M parameters (1.16% fewer) and width 320, compared with the width-512 MoE shared reference. The one-pass controls use the fixed budget to process 2.455B/2.753B tokens, versus 0.400B for repeated cross-depth scans.
Fresh KV reduces repeated scans in the newer 100M, context-4096 study. At 185,057,280 tokens, two scans give lower CE (2.8341 vs 2.9163/2.8703) than the older four/five-scan schedules. At 2,000,076,800 tokens, two/four scans reach 2.1233/2.1442. These compare complete schedules, including supervision distributions (Figure 3, Appendix J.2). Progressive pass schedules, as explored by Full-bandwidth Transformer, offer a further candidate for this architecture.
At matched data, the completed 100M Single-Layer model improves CE and downstream mean over the 105M Standard Transformer. At 200M, the mean is 42.28 vs 42.37 for the 288M control with 30.7% fewer parameters (Section 3.1, Table 1). The source control identifies useful communication within shared refinement, and the complete models measure the value of their combination.