- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper makes an observation that during inference, when generating a token at time t, the embeddings for all layers of all previous tokens are already generated, yet the standard transformer model uses at layer l only embeddings of layers earlier than l. The authors propose a method for training and inference that allows utilisation of all previously generated embeddings. Specifically, the Full-Bandwidth transformer uses the embedding of the last layer of the previous token (time t-1) combined with the true discrete token to create an embedding at time t.
The authors claim that this method of training allows reaching the performance of models trained with more training tokens and with a negligible inference cost. However, Bandwidth Transformer incurs a cost on training memory and time, and limited evaluation makes it impossible to evaluate the trade-off that comes with using the method proposed by the authors.
Critical strengths
- The most important strength is the significance of the method if it is shown to provide benefits. Transformer training and inference are a foundation of all LLMs and, hence, all methods built on them. Providing a benefit in this area would create a significant ripple effect in performance, cost, and other aspects of all AI methods that rely on LLMs.
- Another strength is that the inference cost of the method is actually negligible.
- The writing is pretty clear and easy to follow.
Critical weaknesses
The most critical weakness is the fact that under current evaluation it is impossible to evaluate the cost that Full-Bandwidth transformer induces on training and therefore whether the tradeoff that comes with it is worth it:
- The method by not detaching the gradient increases the GPU memory requirements by necessitating the save the activations as the authors state. However, the paper does not include any comparison nor analysis of the increased memory.
- The comparison under the same token-training is clearly unfair to the standard transformer due to significantly increased flop cost and memory requirements of the Full-Bandwidth Transformer (FBT).
- Because of that authors also compare the FBT with 200B training tokens to standard transformer trained with 400B training tokens. However, the paper does not include training curves, hence it is not known whether the models have already converged at that point and FBT actually allows to converge to a better local minimum, or just allows for a faster (in terms of speed per number of tokens seen) training that in the end would converge to the same value. This is especially concering that 1T model (std) still produces significant boost in comparison to all lower-token regimes showing that the model is far from convergence.
- The evaluation in general is limited both in terms of benchmarks (especially as performance is not consistent across them) and ablations, which are not sufficient to isolate the mechanism.
Not a critical weakness - a typo: on l175 commented line is line 8, not 7 (should say "line 7 vs commented line 8" ).
Questions for authors
I would like to see several things to understand the influence and the cost related to FBT.
- Memory analysis, demonstration of how much the memory increases with FBT and how much we can counterbalance it with activation recomputation for what cost (in the appendix the authors say they use it, and measure the time, but it is not said whether the activation recomputation makes the memory cost the same as for standard transformer).
- Fair fixed-flop and memory comparison. Currently, the authors count each additional pass as a flop-cost. However, later in the appendix the authors said that they use activation recomputation to reduce the memory. This adds flops to the equations which are not counted in the original comparison. Furthermore it gives the opportunity to provide a fair comparison, where the authors set the gradient recomputation such that the memory-usage is the same. Add total number of flops that FBT added this way (both for the forward pass and for the activation recomputations in the backward pass). This allows to have a FLOP-matched comparison with a standard transformer (FLOPs counted for the entire training). If the FBT shows performance improvements in this type of setting across multiple scales and benchmarks and does not decrease the practical cost such as memory (it shouldn't after properly set activation recomputation) and total training wall-clock time, it shows that FBT is a valuable consideration.
- Training curves and progression of performance with the number of tokens. Results for saturated models with given token count, to understand whether FBT can achieve better performance or just "faster" convergence in terms of tokens seen.
- I kindly ask the authors to evaluate the FBT on more benchmark as well as current mixed results even in the fixed token setting (which are unfair to the standard transformer) are hard to interpret and if anything do not look too promising for the FBT.
11 comments
Author response (2/2)
Lack of benchmarks and ablations
Our evaluation covers several capability categories commonly reported for non-reasoning small language models (e.g. Qwen2-1.5B [1]), and extends to instruction-following and writing tasks:
- Language understanding and knowledge: MMLU, ARC-Easy, ARC-Challenge, BoolQ, RTE, PIQA, WinoGrande, OpenBookQA, COPA, and TruthfulQA-MC2.
- Math and code generation: GSM8K, MATH-500, HumanEval, and MBPP, evaluated before and after instruction tuning. The gain is consistent except for base-model MATH-500 results.
- Instruction following and writing (appendix J): IFEval and the English subset of WritingBench. FBT shows no improvement in these non-reasoning tasks. The existing results are also considered "statistically significant" by reviewer WTGS.
Other free-form evaluation tasks involve large-scale models and extensive post-training to conduct (e.g. agentic benchmarks), which goes beyond the scope of the current work.
We also included the following ablations in the appendix:
- Gradient flow: detaching versus backpropagating through hidden states between passes.
- Feedback scheduling: when additional passes are introduced, the number of passes, and staged versus interleaved training.
- Fusion design: gated multiplication, linear addition, and concatenation followed by projection.
- Input regularization: training with and without hidden-state jitter noise.
We are happy to consider more benchmarks and ablations upon the reviewer's request.
[1] https://huggingface.co/Qwen/Qwen2-1.5B#qwen2-05b--qwen2-15b-performances
Typo
Not a critical weakness - a typo: on l175 commented line is line 8, not 7 (should say "line 7 vs commented line 8" ).
Thanks for pointing this out! We have fixed it in the updated manuscript
Thank you for the reply,
Let me start with the most important points.
Fair comparison
To clarify, we have three main points of comparisions:
- Comparing FBT's standard decoding with soft decoding to illustrate the usefulness of latent feedback.
- Comparing FBT with standard transformer trained on the same token budget to illustrate FBT's multi-pass training does NOT harm the model (in contrast to e.g. MTP prediction head).
- Comparing FBT with a standard transformer trained on more tokens to illustrate FBT's data and FLOP efficiency.
For the last point, since the extra memory overhead is negligible when gradient checkpointing is used, we can just compare the actual end-to-end budget by looking at the wall clock time estimated using the per-step time from Appendix G
Thank you for listing these points of comparison. By far the most important of these points is the 3rd one, which should demonstrate that FBT has meaningful benefits over standard transformers in some applications.
My biggest issue with the paper is that I don't believe that the manuscript establishes that. The authors refer to the 400B 1.5x results, which give average performance on the test suite comparable to 1T standard transformers with 600B compute (700B if we count FLOPs, but I'm okay assuming it's 600 based on wall-clock time). However, there are several important problems with acknowledging this result:
- This is a single result on a single run on benchmarks that are known to be noisy
- The authors themselves wrote in the paper that different schedulers of the method diverged on this scale, which just increases my doubts in the robustness of this result
- Furthermore, as the benchmark results are noisy seed-to-seed, I wanted to inspect perplexity, which, while not sufficient on its own is much more stable between seeds. However, the authors did not include the perplexity for the 1T run (Figure 4, left). Why is it not included?
- Importantly, other results do not lead to a similar conclusion. For example, I would like to ask the authors how to interpret Figures 5 and 4 in this regard. Specifically:
- we cannot compare 10B standard to 10B FBT (and 100B standard to 100B FBT etc), because it is unfair as FBT uses more FLOPs.
- The fairest setup we can is to choose the most favourable way for FBT, but where the baseline **still uses at least as many FLOPs ** - so 100B FBT (150B matched FLOPs) vs 200B transformer, etc. In this comparison, FBT beats the standard transformer only 6 out of 36 times in Figure 5, and Figure 4 looks even worse.
The authors could argue that 200B has more FLOPs than 150B, so this should be somehow discounted, but these numbers were selected by the authors, and they could have been chosen more reasonably to make the comparison easier while still fair. As I said before and the authors stated as well, it should be a flop matched comparison, so comparing 100B FBT to 100B standard transformer is unfair.
What would make it clear to me that the FBT robustly beats the transformer in a fair setting is the following:
- the list of pairs of token counts, a1, b1, a2, b2 where FBT transformer on a1 tokens is compared to standard transformer on b2 tokens, and FLOPs(FBT(a_i)) < FLOPs(standard_transformer(b_i)), but at the same time Performance(FBT(a_i)) > Performance(standard_transformer(b_i)), where FBT(a_i) is the FBT trained on a_i tokens.
- This should include both perplexity, benchmark scores and some seed-to-seed training variance, as without it, especially on the noisy evaluation benchmark it is impossible to evaluate how much of the result is an actual signal.
- All runs should be on the same schedule of full bandwidth, to prove that the schedule is robust and the runs are not selected post-factum.
With a single run per configuration, different looping schedules and, most importantly, results that are worse for FBT than a transformer if we look at the most favourable, but still fair comparison (100B FBT vs 200B standard), there is no evidence at all that FBT outperforms a standard transformer.
Training token size
I agree that modern language models are not trained until strict convergence, but I would also like to acknowledge that the models typically use much more tokens than 10B or 100B used in the experiments. Open-source models typically use 3T-10T in that range. Furthermore, the authors already have a checkpoint of 1T training tokens with the standard transformer, and the additional FBT part comes only at the end of the training. Why not include the performance at 1T as well? This is important to demonstrate that:
- stability issues shown in the smaller scale (400B) do not make it impossible to use FBT in a slightly larger scale.
- the results hold at this scale, which is often an issue with architecture designs.
Author response (1/2)
We would like to thank the reviewer for the careful review and thoughtful comments.
Please see below for our responses to the weaknesses and questions.
Training time memory analysis
During training, FBT's memory for the model weights, gradients, and optimizer states is the same as that of the standard transformer.
The main extra memory overhead comes from the extra activation, whose size is determined by the batch size and the number of passes.
In Figure 15 in the appendix, we also verified the feasibility of detaching the gradient from the hidden state. On a small-scale 10B-token experiment, the detached version still shows improvement but reduces the gain.
Broadly, we agree that the memory increase is a limitation, a general challenge for all looping-based pre-training methods, which leaves room for future pre-training infrastructure studies.
Does FBT lead to faster convergence?
Modern language models are not trained to convergence because the computational complexity is very high and it is typically superior to use available compute on larger and less converged models. This insight is the core reason why scaling laws have been of great interest.
FBT is more data-efficient compared with the standard transformer, evaluated on free-form generation tasks powered by latent feedback decoding (
Softin Figure 5 and Table 1), which we claim to be an advantage of FBT.However, it does NOT achieve this through faster "convergence" in pre-training. Figure 17 in the appendix illustrates the training curve of full bandwidth transformer v.s. standard transformer under the same token budget, where we do NOT observe FBT's first pass, i.e. the standard NTP loss, improve over the standard transformer. The main performance gain from FBT comes from the latent feedback decoding (
Soft) empowered by extra training time FLOPs. This gain is so large that it is superior to training with more tokens (and FLOPs) in the standard way, similar to the way that using larger models is superior to deeper convergence on smaller models.Fair comparison
To clarify, we have three main points of comparisions:
For the last point, since the extra memory overhead is negligible when gradient checkpointing is used, we can just compare the actual end-to-end budget by looking at the wall clock time estimated using the per-step time from Appendix G
where the FLOPs for a checkpointed k-pass step is given by: Ck≈kF+2kF+kF=4kF. The three terms correspond to forward computation, backward computation, and activation recomputation, respectively. We do not use activation recomputation when training with only one pass (first 75%) so C1=3F.
Now for each full FBT run, we have
FBT is more compute-efficient when looking at end-to-end training time: The 400B (1.5x) tokens run takes ~620B tokens equivalent training time, and it shows performance on par with the 1T tokens run on LM Eval (appendix H, figure (d)) and other free-form generation tasks (Table 1). The 200B tokens FBT (~260B tokens equivalent training time) also manages to exceed or match the performance of a 400B tokens standard transformer.