acceptodds
Under review as a conference paper at ICLR 2027

Full-bandwidth Transformer

22 views

Abstract

Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal access to the past, but the vertical feedback channel between decoding steps remains narrow: only the sampled token returns to the bottom of the stack, while the top-layer hidden state is discarded. We introduce the full-bandwidth transformer, which widens this channel with latent feedback: at each decoding step, the previous top-layer hidden state is fused with the sampled token embedding through a gated linear unit and fed back as the next input. Latent feedback lets non-verbalized computation re-enter the stack with a renewed depth budget, while preserving the standard transformer architecture, KV cache, and language-modeling objective. To train full-bandwidth transformers without losing parallel teacher forcing, we use a scheduled multi-pass objective that introduces latent feedback late in pretraining and mixes a small fraction of deeper feedback passes for stability. We train 1B-parameter full-bandwidth transformers up to 400B tokens and find that latent feedback improves validation loss, 5-shot language-model evaluation, math and coding generation, and instruction-tuned performance. With negligible per-token decoding overhead, full-bandwidth transformers match or approach standard transformers trained with roughly 1.5× more tokens, and manage to produce shorter reasoning when no off-policy templates are provided. Code available at https://anonymous.4open.science/r/Full-bandwidth-transformer-1680/

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Recurrence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mdejOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper makes an observation that during inference, when generating a token at time t, the embeddings for all layers of all previous tokens are already generated, yet the standard transformer model uses at layer l only embeddings of layers earlier than l. The authors propose a method for training and inference that allows utilisation of all previously generated embeddings. Specifically, the Full-Bandwidth transformer uses the embedding of the last layer of the previous token (time t-1) combined with the true discrete token to create an embedding at time t.

The authors claim that this method of training allows reaching the performance of models trained with more training tokens and with a negligible inference cost. However, Bandwidth Transformer incurs a cost on training memory and time, and limited evaluation makes it impossible to evaluate the trade-off that comes with using the method proposed by the authors.

Critical strengths
  1. The most important strength is the significance of the method if it is shown to provide benefits. Transformer training and inference are a foundation of all LLMs and, hence, all methods built on them. Providing a benefit in this area would create a significant ripple effect in performance, cost, and other aspects of all AI methods that rely on LLMs.
  2. Another strength is that the inference cost of the method is actually negligible.
  3. The writing is pretty clear and easy to follow.
Critical weaknesses

The most critical weakness is the fact that under current evaluation it is impossible to evaluate the cost that Full-Bandwidth transformer induces on training and therefore whether the tradeoff that comes with it is worth it:

  1. The method by not detaching the gradient increases the GPU memory requirements by necessitating the save the activations as the authors state. However, the paper does not include any comparison nor analysis of the increased memory.
  2. The comparison under the same token-training is clearly unfair to the standard transformer due to significantly increased flop cost and memory requirements of the Full-Bandwidth Transformer (FBT).
  3. Because of that authors also compare the FBT with 200B training tokens to standard transformer trained with 400B training tokens. However, the paper does not include training curves, hence it is not known whether the models have already converged at that point and FBT actually allows to converge to a better local minimum, or just allows for a faster (in terms of speed per number of tokens seen) training that in the end would converge to the same value. This is especially concering that 1T model (std) still produces significant boost in comparison to all lower-token regimes showing that the model is far from convergence.
  4. The evaluation in general is limited both in terms of benchmarks (especially as performance is not consistent across them) and ablations, which are not sufficient to isolate the mechanism.

Not a critical weakness - a typo: on l175 commented line is line 8, not 7 (should say "line 7 vs commented line 8" ).

Questions for authors

I would like to see several things to understand the influence and the cost related to FBT.

  1. Memory analysis, demonstration of how much the memory increases with FBT and how much we can counterbalance it with activation recomputation for what cost (in the appendix the authors say they use it, and measure the time, but it is not said whether the activation recomputation makes the memory cost the same as for standard transformer).
  2. Fair fixed-flop and memory comparison. Currently, the authors count each additional pass as a flop-cost. However, later in the appendix the authors said that they use activation recomputation to reduce the memory. This adds flops to the equations which are not counted in the original comparison. Furthermore it gives the opportunity to provide a fair comparison, where the authors set the gradient recomputation such that the memory-usage is the same. Add total number of flops that FBT added this way (both for the forward pass and for the activation recomputations in the backward pass). This allows to have a FLOP-matched comparison with a standard transformer (FLOPs counted for the entire training). If the FBT shows performance improvements in this type of setting across multiple scales and benchmarks and does not decrease the practical cost such as memory (it shouldn't after properly set activation recomputation) and total training wall-clock time, it shows that FBT is a valuable consideration.
  3. Training curves and progression of performance with the number of tokens. Results for saturated models with given token count, to understand whether FBT can achieve better performance or just "faster" convergence in terms of tokens seen.
  4. I kindly ask the authors to evaluate the FBT on more benchmark as well as current mixed results even in the fixed token setting (which are unfair to the standard transformer) are hard to interpret and if anything do not look too promising for the FBT.
11 comments
AuthorsOpenReview ↗

Author response (1/2)

We would like to thank the reviewer for the careful review and thoughtful comments.

Please see below for our responses to the weaknesses and questions.

Training time memory analysis

During training, FBT's memory for the model weights, gradients, and optimizer states is the same as that of the standard transformer.

The main extra memory overhead comes from the extra activation, whose size is determined by the batch size and the number of passes.

  • When gradient checkpointing is used, the input of each pass needs to be stored. This costs an extra 0.4 GB- 0.8 GB per GPU under the current pre-training setting, which is negligible compared with the standard transformer.
  • When gradient checkpointing is not used, we need to store the intermediate activation memory, which costs ~44 GB per extra pass, per GPU.

In Figure 15 in the appendix, we also verified the feasibility of detaching the gradient from the hidden state. On a small-scale 10B-token experiment, the detached version still shows improvement but reduces the gain.

Broadly, we agree that the memory increase is a limitation, a general challenge for all looping-based pre-training methods, which leaves room for future pre-training infrastructure studies.

Does FBT lead to faster convergence?

hence it is not known whether the models have already converged at that point and FBT actually allows to converge to a better local minimum

Modern language models are not trained to convergence because the computational complexity is very high and it is typically superior to use available compute on larger and less converged models. This insight is the core reason why scaling laws have been of great interest.

FBT is more data-efficient compared with the standard transformer, evaluated on free-form generation tasks powered by latent feedback decoding (Soft in Figure 5 and Table 1), which we claim to be an advantage of FBT.

However, it does NOT achieve this through faster "convergence" in pre-training. Figure 17 in the appendix illustrates the training curve of full bandwidth transformer v.s. standard transformer under the same token budget, where we do NOT observe FBT's first pass, i.e. the standard NTP loss, improve over the standard transformer. The main performance gain from FBT comes from the latent feedback decoding (Soft) empowered by extra training time FLOPs. This gain is so large that it is superior to training with more tokens (and FLOPs) in the standard way, similar to the way that using larger models is superior to deeper convergence on smaller models.

Fair comparison

The comparison under the same token-training is clearly unfair to the standard transformer due to significantly increased flop cost and memory requirements of the Full-Bandwidth Transformer (FBT).

To clarify, we have three main points of comparisions:

  • Comparing FBT's standard decoding with soft decoding to illustrate the usefulness of latent feedback.
  • Comparing FBT with standard transformer trained on the same token budget to illustrate FBT's multi-pass training does NOT harm the model (in contrast to e.g. MTP prediction head).
  • Comparing FBT with a standard transformer trained on more tokens to illustrate FBT's data and FLOP efficiency.

For the last point, since the extra memory overhead is negligible when gradient checkpointing is used, we can just compare the actual end-to-end budget by looking at the wall clock time estimated using the per-step time from Appendix G

PassesStep timeIncrease from previous rowEstimated total FLOPs
10.36 s—3F3F
20.75 s+0.39 s8F8F
31.13 s+0.38 s12F12F
41.53 s+0.40 s16F16F

where the FLOPs for a checkpointed kk-pass step is given by: Ck≈kF+2kF+kF=4kF.C_k \approx kF + 2kF + kF = 4kF. The three terms correspond to forward computation, backward computation, and activation recomputation, respectively. We do not use activation recomputation when training with only one pass (first 75%) so C1=3FC_1 = 3F.

Now for each full FBT run, we have

RunToken-equivalent computeAverage step timeEstimated training time on 8 B200sTime relative to standard at the same token count
10B30B1.1300 s10.46 h (0.44 days)3.139×
100B150B0.5525 s51.16 h (2.13 days)1.535×
200B256B0.4689 s86.83 h (3.62 days)1.303×
400B (1.28×)512B0.4689 s173.67 h (7.24 days)1.303×
400B (1.5×)600B0.5525 s204.63 h (8.53 days)1.535×

FBT is more compute-efficient when looking at end-to-end training time: The 400B (1.5x) tokens run takes ~620B tokens equivalent training time, and it shows performance on par with the 1T tokens run on LM Eval (appendix H, figure (d)) and other free-form generation tasks (Table 1). The 200B tokens FBT (~260B tokens equivalent training time) also manages to exceed or match the performance of a 400B tokens standard transformer.

AuthorsOpenReview ↗

Author response (2/2)

Lack of benchmarks and ablations

Our evaluation covers several capability categories commonly reported for non-reasoning small language models (e.g. Qwen2-1.5B [1]), and extends to instruction-following and writing tasks:

  • Language understanding and knowledge: MMLU, ARC-Easy, ARC-Challenge, BoolQ, RTE, PIQA, WinoGrande, OpenBookQA, COPA, and TruthfulQA-MC2.
  • Math and code generation: GSM8K, MATH-500, HumanEval, and MBPP, evaluated before and after instruction tuning. The gain is consistent except for base-model MATH-500 results.
  • Instruction following and writing (appendix J): IFEval and the English subset of WritingBench. FBT shows no improvement in these non-reasoning tasks. The existing results are also considered "statistically significant" by reviewer WTGS.

Other free-form evaluation tasks involve large-scale models and extensive post-training to conduct (e.g. agentic benchmarks), which goes beyond the scope of the current work.

We also included the following ablations in the appendix:

  • Gradient flow: detaching versus backpropagating through hidden states between passes.
  • Feedback scheduling: when additional passes are introduced, the number of passes, and staged versus interleaved training.
  • Fusion design: gated multiplication, linear addition, and concatenation followed by projection.
  • Input regularization: training with and without hidden-state jitter noise.

We are happy to consider more benchmarks and ablations upon the reviewer's request.

[1] https://huggingface.co/Qwen/Qwen2-1.5B#qwen2-05b--qwen2-15b-performances

Typo

Not a critical weakness - a typo: on l175 commented line is line 8, not 7 (should say "line 7 vs commented line 8" ).

Thanks for pointing this out! We have fixed it in the updated manuscript

Thank you for the reply,

Let me start with the most important points.

Fair comparison

To clarify, we have three main points of comparisions:

  • Comparing FBT's standard decoding with soft decoding to illustrate the usefulness of latent feedback.
  • Comparing FBT with standard transformer trained on the same token budget to illustrate FBT's multi-pass training does NOT harm the model (in contrast to e.g. MTP prediction head).
  • Comparing FBT with a standard transformer trained on more tokens to illustrate FBT's data and FLOP efficiency.

For the last point, since the extra memory overhead is negligible when gradient checkpointing is used, we can just compare the actual end-to-end budget by looking at the wall clock time estimated using the per-step time from Appendix G

Thank you for listing these points of comparison. By far the most important of these points is the 3rd one, which should demonstrate that FBT has meaningful benefits over standard transformers in some applications.

My biggest issue with the paper is that I don't believe that the manuscript establishes that. The authors refer to the 400B 1.5x results, which give average performance on the test suite comparable to 1T standard transformers with 600B compute (700B if we count FLOPs, but I'm okay assuming it's 600 based on wall-clock time). However, there are several important problems with acknowledging this result:

  • This is a single result on a single run on benchmarks that are known to be noisy
  • The authors themselves wrote in the paper that different schedulers of the method diverged on this scale, which just increases my doubts in the robustness of this result
  • Furthermore, as the benchmark results are noisy seed-to-seed, I wanted to inspect perplexity, which, while not sufficient on its own is much more stable between seeds. However, the authors did not include the perplexity for the 1T run (Figure 4, left). Why is it not included?
  • Importantly, other results do not lead to a similar conclusion. For example, I would like to ask the authors how to interpret Figures 5 and 4 in this regard. Specifically:
    • we cannot compare 10B standard to 10B FBT (and 100B standard to 100B FBT etc), because it is unfair as FBT uses more FLOPs.
    • The fairest setup we can is to choose the most favourable way for FBT, but where the baseline **still uses at least as many FLOPs ** - so 100B FBT (150B matched FLOPs) vs 200B transformer, etc. In this comparison, FBT beats the standard transformer only 6 out of 36 times in Figure 5, and Figure 4 looks even worse.

The authors could argue that 200B has more FLOPs than 150B, so this should be somehow discounted, but these numbers were selected by the authors, and they could have been chosen more reasonably to make the comparison easier while still fair. As I said before and the authors stated as well, it should be a flop matched comparison, so comparing 100B FBT to 100B standard transformer is unfair.

What would make it clear to me that the FBT robustly beats the transformer in a fair setting is the following:

  • the list of pairs of token counts, a1, b1, a2, b2 where FBT transformer on a1 tokens is compared to standard transformer on b2 tokens, and FLOPs(FBT(a_i)) < FLOPs(standard_transformer(b_i)), but at the same time Performance(FBT(a_i)) > Performance(standard_transformer(b_i)), where FBT(a_i) is the FBT trained on a_i tokens.
  • This should include both perplexity, benchmark scores and some seed-to-seed training variance, as without it, especially on the noisy evaluation benchmark it is impossible to evaluate how much of the result is an actual signal.
  • All runs should be on the same schedule of full bandwidth, to prove that the schedule is robust and the runs are not selected post-factum.

With a single run per configuration, different looping schedules and, most importantly, results that are worse for FBT than a transformer if we look at the most favourable, but still fair comparison (100B FBT vs 200B standard), there is no evidence at all that FBT outperforms a standard transformer.

Training token size

I agree that modern language models are not trained until strict convergence, but I would also like to acknowledge that the models typically use much more tokens than 10B or 100B used in the experiments. Open-source models typically use 3T-10T in that range. Furthermore, the authors already have a checkpoint of 1T training tokens with the standard transformer, and the additional FBT part comes only at the end of the training. Why not include the performance at 1T as well? This is important to demonstrate that:

  1. stability issues shown in the smaller scale (400B) do not make it impossible to use FBT in a slightly larger scale.
  2. the results hold at this scale, which is often an issue with architecture designs.
Reviewer_GwvxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors feed the previous position's top-layer hidden state back into the input of the next position, fused with the token embedding through a gated linear unit. Training stays parallel by approximating this recurrence with a few extra forward passes over right-shifted hidden states, added only in the last 25% of pretraining. 1B models trained on 10B to 400B tokens are evaluated on validation loss and downstream tasks before and after instruction tuning. The authors claim that this matches standard transformers trained on 1.5x to 2x more tokens at about 2% extra decoding cost. The results look promising, but overall the contribution does not yet strike me as a big step, and I am currently not convinced it is significant enough for acceptance.

Critical strengths

The method is simple, leaves the KV cache and serving stack intact, and is tested at an unusual scale for this line of work. The authors report token-equivalent compute for every run, and on validation loss the 200B run reaching the 400B baseline is a compute-favourable result.

Critical weaknesses

As far as I understood, no baseline spends the same extra passes without the feedback (e.g. a 2-pass looped model), so you can't seperate the gain from the extra training compute. Everything is a single run, and many downstream gaps, such as 1 to 3 points on MATH-500, are within benchmark noise. More importantly, the main 400B results use the 1.28x schedule, which Appendix H says "failed to extrapolate at 400B tokens". The model is trained on feedback chains of two or three steps but decoded for hundreds, so stability beyond the trained depth is the central risk and belongs in the main text. A compute-matched baseline and seeds for Table 1 would make me more confident.

Questions for authors

Clarification: Why are the main 400B results reported for the schedule that Appendix H says is unstable, and how far into a rollout does it diverge?

Suggestion: Could you add a looped baseline that uses the same extra passes without latent feedback, started from the same 75% checkpoint?

Clarification: Do gradients flow through shift_right(h) into earlier passes, or is it detached?

Please provide short answers and skip unnecessary details.

4 comments
AuthorsOpenReview ↗

Author response (1/2)

We would like to thank the reviewer for the helpful and insightful feedback!

Please see below for our response to the weaknesses and questions

Looped baseline and usefulness of latent feedback

We do not foresee obvious reasons for an FBT to beat a transformer looped twice under the same training compute: Looped transformers double the inference overhead of generating each token and potentially the KV cache size; FBT has the same cost and KV cache size. The extra inference time compute from looping should provide extra gain if implemented properly.

However, hypothetically speaking, latent feedback in FBT can be combined with looping and boost it: If an adaptive regime loops extensively at one position, feeding its final hidden state to the next position's input enables us to reuse it rather than just disposing of it after decoding. We leave the combination for future studies.

The latent feedback is still useful, as supported by the current results : Our Soft-versus-Std. comparison in Figure 5 and Table 1 holds the trained weights fixed (i.e. same training compute) and demonstrates gains from enabling latent feedback during decoding. This supports the usefulness of the feedback channel at essentially unchanged decoding cost. Of course, FBT's gain from Soft is enabled by the extra training time compute; that's why we also compare it with standard transformer trained under more tokens.

Could you add a looped baseline that uses the same extra passes without latent feedback,

We additionally include a baseline similiar to looping using standard decoding without latent feedback based on an existing FBT checkpoint, but we use Fused to loop the model twice for both prefilling and decoding. The results are presented in the bolded row below: It does improve over Standard decoding, but the performance is on par with Soft, despite doubling the overhead and KV cache size.

Decoding methodGSM8KMATH-500
Standard49.36%35.20%
Two passes for prefill and decoding51.93%37.80%
Soft51.63%38.60%
Fused (two passes for prefill, Soft for decoding)52.39%39.60%

Note that the looping design here may not be the state-of-the-art version of loop transformer, and the results are based on a smaller model size and run provided in the attached codebase, rather than the manuscript's original setting, whose assets are currently unavailable due to logistical constraints. Therefore, the results may not reflect a strong looped transformer baseline v.s. FBT at large-scale experiments.

Stability of rollout

We have also included the discussion of the stability in the main text per your suggestion for better clarification.

Why are the main 400B results reported for the schedule that Appendix H says is unstable, and how far into a rollout does it diverge?

We would like to clarify that during rollout, we do not observe divergence; otherwise, we would see significantly degraded performance in free-form generation tasks, whose rollout length goes far beyond 4 tokens.

The extrapolation divergence of the 400B token 1.28x recipe refers to when the number of forward passes used for prefilling goes beyond 4 (appendix H, figure b and d, blue lines), which serves as a predictive proxy for the inference time recurrence stability.

This suggests that instability of extrapolation in prefilling over 4 passes does NOT imply instability in rollout: We hypothesize that, because a full-bandwidth transformer can access past information through attention (unlike an RNN), its hidden state need not carry all historical information through an uninterrupted chain of recurrent updates. Therefore, the influence of distant states along the direct feedback pathway may diminish in a short window size (e.g. 4), while historical information represented in the KV cache remains accessible. We also empirically verify the window size with a hidden state replay experiment, where three parallel passes over the rollout recover 99.88% next-token argmax agreement (Figure 20, appendix S.2).

We keep the 1.28x results in the main text and openly admit the instability for transparency and consistency purposes: We also present the results of the 1.3x and 1.5x schedule recipe, costing ~2 and ~20 percent more compute but providing better performance and stability (red and yellow lines in appendix H).

AuthorsOpenReview ↗

Author response (2/2)

Baseline and seeds

Everything is a single run, and many downstream gaps, such as 1 to 3 points on MATH-500, are within benchmark noise.

Each experiment run takes at least a week on 8xB200. Running the experiments on multiple seeds goes beyond the resources we have access to. However, we observe consistent improvements across multiple runs under different budgets (100B, 200B, 400B) and feedback pass schedules (Appendix H), so we do not think the performance gain comes from random noise.

The effectiveness of the method is also cross-confirmed by other concurrent and independent works, e.g., Recycler from ByteDance [1] (released Oct 2026) adopts an almost identical approach and training recipe and observed performance improvement in a wider range of coding and agentic benchmarks.

A compute-matched baseline and seeds for Table 1 would make me more confident.

The 1T token run is a compute-matched (and exceeds) baseline for the 400B 1.5x run: The 400B FBT's Soft shows performance on par with the 1T run (except for HumanEval) under almost identical inference overhead (up to 0.5 - 1 percent more).

[1] https://seed.bytedance.com/en/public_papers/recycler-cost-free-latent-reasoning-via-hidden-state-recycling?order_desc=true&offset=0

Detaching gradient

Do gradients flow through shift_right(h) into earlier passes, or is it detached?

As we state in line 215, we did NOT detach the gradient from the hidden state. In Figure 15 in the appendix, we ablated detaching the gradient in small scale 10B tokens run, where we still observe a performance gain when including feedback in prefilling, the improvement is reduced compared with the non-detached version, though.

Reviewer_GwvxOpenReview ↗

three quick questions

Apologies for the late reply. I have been carefully following the discussion with Reviewer mdej, and I have three short questions.

  1. What is the validation loss of the 1T standard model? It is listed in the legend of Figure 4 (left), but I can't see the line.

  2. As far as I understand, the full-bandwidth losses in Figure 4 (left) at one or more feedback passes use at least twice the inference compute of the baseline, since the whole text is processed again. What is the validation loss with Soft decoding, i.e. with the token-by-token feedback at the same inference cost as the baseline?

  3. In Table 1, the full-bandwidth models are trained with three passes during long-context extension and instruction tuning, while the baselines, I think, use one. Do you have a baseline with matched post-training compute, or a full-bandwidth model tuned with a single pass?

Thank you for your short answers.

AuthorsOpenReview ↗

Author response

Hi

Thank you for your response, due to the short time window, we apologize that we cannot present complete results

First of all, hope you find our response to the loop transformer v.s. FBT convincing! I believe that will be a commonly asked question, and we will introduce further clarifications into the draft.

What is the validation loss of the 1T standard model?

We currently do not have access to exact number at this moment but it's better than 400B FBT. However, validation loss comparision does not showcase the main benefit of FBT (as we will argue in below)

As far as I understand, the full-bandwidth losses in Figure 4 (left) at one or more feedback passes use at least twice the inference compute of the baseline, since the whole text is processed again.

That is correct.

What is the validation loss with Soft decoding, i.e. with the token-by-token feedback at the same inference cost as the baseline?

However, note that Soft decoding is NOT meant to benefit validation loss or LM eval accuracy and CANNOT be applied for prefilling.: It only comes into effect when decoding is involved. We also never claimed or suggested that FBT improves prefilling for free. The goal of Figure 4 is to demonstrate that: We train the model like a loop transformer and we can use it like one for optional "test time scaling" for prefilling.

The main benefit and difference of FBT (compared with standard or loop transformer) lies in a free performance boost (via Soft decoding) on open-ended generation tasks (in constrat to prefilling only tasks) (as shown in Figure 5 and Table 1).

Lastly, if better prefilling via multi-forward pass is the goal for optimization (e.g. better Figure 4 results), then we totally agree there are lots of strong loop transformer baselines out there, but that is not our main benefit.

In Table 1, the full-bandwidth models are trained with three passes during long-context extension and instruction tuning, while the baselines, I think, use one. Do you have a baseline with matched post-training compute, or a full-bandwidth model tuned with a single pass?

Yes that's correct and good point. Note that the long context and instruction tuning data are often limited in quantity, therefore a fair baseline can repeat the data multiple times for a standard transformer to match the compute of an FBT, however, this leads to overfitting and severe catastrophic forgetting. One possible solution is to reduce the number of repetition as well as introducing regularization (e.g. weight decay), however our preliminary experiments show that in this regime that favors standard transformer, we can further boost performance by training with multiple passes (without inducing more overfitting and forgetting).

So in summary, when data is constrained, FBT enables us to spend more flops (via multiple pass training) to build a stronger model with same inference cost (via soft decoding) without the risk of overfitting. So we believe FBT is a clear win from the data efficient perspective.

Reviewer_WTGSOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a variant of the standard transformer called the Full-Bandwidth transformer where during decoding token t+1, instead of using only the low-bandwidth discrete token embedding of token t, it also uses the "full"-bandwidth richer embedding of token t in the penultimate layer of token t's forward pass. This information is fused with the embedding of token t and presented to the first layer of token t+1's generation. This enables passing richer representations from higher layers to lower layers. The main challenge in getting this to work is training: which needs to be run in a multi-pass manner now. Pass i+1 shifts the tokens one step forward in time compared to pass i. To control the added cost of multiple passes in training, the authors introduce extra passes only in later stages of training - 1 pass until 75%, 2 passes after that. However, to truly match what happens in inference, one needs to run sequence_length number of passes in training which is infeasible. The authors claim that introducing a small 3% stage at the end of training largely removes the need for this and stabilizes the training-inference gap. They present as evidence for this a study showing how much the magnitude of hidden states changes when more passes are done on the final checkpoint is very minimal. The authors observe that allowing such latent feedback has unexpected benefits such as the latent feedback acting as a scratchpad and thereby helping reduce thought lengths during decoding.

Critical strengths
  1. The paper expands on a recent novel direction of using higher layer embeddings in lower layers in a recurrent manner by using mutli-pass training.

  2. The benefits provided from the method are statistically significant.

  3. The paper is well written and nice to read.

Critical weaknesses
  1. The convergence of the training with k passes to true recurrence only gets tested when decoding very long sequences. The chosen datasets typically have answers shortern than 1k tokens. This leaves open the question of what happens for truly long decoding sequences.

  2. While the method is impressive and shows promising results for pre-training, I see a number of issues that might arise during post-training (especially RL where sampler trainer mismatches can cause instabilities). This is largely unaddressed in this paper.

  3. Not a critical weakness but a minor one: experiments could be a bit more comprehensive along two axes: a) evaluate more benchmarks outside domains of math and coding, b) train models at different scales to see if the gains are consistent across scales.

Questions for authors
  1. The paper on StagFormer (StagFormer: Time Staggering Transformer Decoding for RunningLayers In Parallel) is very related and it would be nice to see a discussion on how your design compares with it in the related work section.
  2. As we decode more we are going further and further away from train setup, I can imagine having to re-prefill after a block of M tokens generated. How are you able to circumvent this? Is it purely the 3% 3-pass training that removes the need for periodic re-prefill?
  3. How would post-training work? especially with trainer sampler divergence in RL
  4. Any examples of decode longer than 1K tokens?
  5. In Table 1, why are the gains of Fused only marginal over Soft?
3 comments
AuthorsOpenReview ↗

Author response (1/2)

We would like to thank the reviewer for the detailed response and the acknowledgment of our work's timeliness, statistically significant benefits, and writing clarity.

Please see below for our response to the weaknesses and the questions

Post-training recipe

Yes, post-training is a big challenge. In Appendix N of the manuscript, we discussed the challenge and the potential solutions.

To summarize Appendix N,

  • The challenge lies in the evaluation of rollout likelihood from an off-policy rollout generated under latent feedback decoding, which requires recursive re-prefilling position by position by the trainer if evaluated exactly.
  • The solution is to approximate the rollout likelihood by either reusing the sampler's hidden state or with temporal parallelism, i.e. multiple forward passes.

We have also included some preliminary RL results assuming a mild amount of sampler-trainer mismatch, where the sampler and trainer are implemented with vLLM and Hugging Face, respectively. The "Online RL post-training" section in the README file and results in "online_rl_experiments" folder in the attached code repo show that the reward manages to climb under the two approximations.

However, we have not yet concluded whether the recipe works under more severe sampler-trainer mismatch, e.g., in off-policy / async RL or under larger models, where the bias in policy gradient could cause instability more easily. We view this as a very important future direction.

Convergence of recursion, examples of long CoT and periodic re-prefill.

These are all very good points!

Convergence or recursion In Appendix H (FEEDBACK PASS SCHEDULING ABLATION), the subfigure (c) tested extrapolation for 1000 passes, and it remains stable.

Examples of long CoT over 1000 tokens Certain questions e.g. in Math500 would have rollouts longer than 1K tokens (since we cap the token budget at 2048) and we do not observe signs of collapsing.

In addition, the current results already confirm that the model extrapolates to horizons well beyond training time passes (median of 600 tokens for math500 v.s. 3 passes during training). However, we currently do not have datasets where the model is capable of producing a median generation length beyond 1K tokens, and we believe the model needs to be post-trained extensively in order to generate very long CoT in one round. A larger model may also be helpful here.

Periodic re-prefill If the model manages to keep generating to infinite length (hypothetically), we do NOT foresee the necessity of periodic re-prefilling: Longer rollout does not necessarily imply more inference train-inference mismatch. In particular, since a full-bandwidth transformer can access past information through attention (unlike an RNN), its hidden state need not carry all historical information through an uninterrupted chain of recurrent updates. The influence of distant states along the direct feedback pathway may diminish in a short window size, while historical information represented in the KV cache remains. accessible.

Overall, as we acknowledged in Line 711 in appendix A, a clearer picture on how, when, and why does the 3 or 4 Jacobi iterations extrapolate to long horizons is valuable for future studies. Additionally, such observations are not just limited to our work; they are also made in other concurrent works [1, 2]

[1] https://seed.bytedance.com/en/public_papers/recycler-cost-free-latent-reasoning-via-hidden-state-recycling?order_desc=true&offset=0

[2] https://arxiv.org/abs/2610.25076

In Table 1, why are the gains of Fused only marginal over Soft?

Fused only uses two forward passes for the prompt part, which is fairly short compared with full rollout; therefore, sometimes it may not introduce a significant improvement (it still does in many cases, e.g. on FBT 400B, 1.5x, GSM8K's fused improves 3 point of accuracy than soft).

On prefill-only tasks, fused prefilling consistently shows more pronounced performance gain (Fig.4 right).

AuthorsOpenReview ↗

Author response (2/2)

More comprehensive experimental settings

More benchmarks Beyond the four free-form tasks in the main text, in Table 3 in the appendix, we evaluated full-bandwidth transformer on IFEval and WritingBench, where we do NOT observe clear performance gain in these two datasets. Due to the small capacity of our model (1B parameter) and the RL pipeline still in construction, more sophisticated tasks such as LiveCodeBench or SWE Verified are unavailable for now; We will certainly keep updating manuscripts with new settings once we have them.

Model at different scales In the code repo we attached, we trained a smaller 900M parameter NanoChat model, and the performance gain remains; models of even smaller sizes cannot accomplish free-form generation tasks, so it's hard to verify the effectiveness. We are unable to experiment with models of larger scale ourselves due to resource constraints; however, a recent concurrent work from ByteDance [1] (which proposed an almost identical training and inference pipeline) suggested that the approach remains effective for models of tens of billions of parameters (inferred from the benchmark number, exact parameter scale undisclosed).

[1] https://seed.bytedance.com/en/public_papers/recycler-cost-free-latent-reasoning-via-hidden-state-recycling

Discussion of StagFormer

Thanks for pointing this out! StagFormer is indeed highly relevant, particularly the recurrent version discussed in Appendix A. There are two major differences

Motivation StagFormer's motivation is efficiency: The main text aims at parallelizing the context representation generation and next token prediction by splitting the model vertically into two chunks (similiar to prefiller and decoder in YOCO) that can be executed in parallel. Our work is motivated by improving the information visibility in the decoding process.

Training recipe StagFormer's Appendix A tested having shallow layers cross-attend to previous positions' top-layer hidden states; however, the authors observed degraded performance. We hypothesize the cause is that their two-pass training recipe fails to teach the model to read recurrently generated hidden states, similiar the extrapolation failure we observed when training with two passes (Figure 3, green line).

We have included a discussion of StagFormer in the related work section of the updated manuscript (highlighted in red).

Reviewer_WTGSOpenReview ↗

Thanks for response

Thank the authors for their response. I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.