acceptodds
Under review as a conference paper at ICLR 2027

Token Averaging for Compute-Efficient Language Model Pretraining

Abstract

Training language models on more text ordinarily requires proportionally more compute. We introduce token averaging, a parameter-free method that replaces each group of consecutive token embeddings with their mean before the first attention layer. The transformer therefore processes times fewer positions while predicting the first token of the next group. This allows a fixed training budget to cover approximately times more raw text; at , the model trains on twice as much text using - fewer FLOPs than the baseline (without token averaging). Since token embeddings are averaged, we randomize group boundaries during training and evaluate using different boundary offsets, ensuring that the token-averaged model and the baseline are scored on identical target positions. In each offset, a sequence times shorter is processed, so the cost of all offset evaluations is approximately equal to one forward pass for the baseline. At inference, each pass stores as many KV-cache entries and incurs of the attention cost compared with the baseline, i.e., half the cache and one-quarter of the attention cost at . Across models from 50M to 500M parameters, our experiments show that the baseline requires , , , and more compute, respectively, to reach the same loss as the model. Full-position evaluation shows lower perplexity at 125M with fewer training FLOPs and comparable loss at 500M with approximately fewer FLOPs. Additional experiments show that learned pooling over token content and within-group position outperforms mean pooling by nats at and nats at . Together, these results show that token averaging increases both training-data throughput and inference efficiency without compromising model quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model pretraining, compute efficiency, token averaging, sequence compression, data efficiency, efficient inference, scaling laws

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_3KvGOpenReview ↗
Rating
1
Confidence
2(−1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies a simple trade-off: average adjacent token embeddings, process fewer positions, and spend the saved compute on more training text. Unlike patch-level methods that later return to token-level training, it keeps the compressed representation throughout. Random offsets vary the prediction targets during training, and evaluation combines offsets to score the same target positions as the baseline. Experiments cover 50M–500M models, with small improvements in endpoint NLL at the smaller scales and similar results at 500M.

Critical strengths
  1. I like the simplicity of the setup. The tokenizer and transformer body stay the same, so the main architectural change is easy to understand. The full-position endpoint results, especially at 50M and 125M, suggest this trade-off is worth studying.

  2. The pooling ablation gives a useful result: at larger compression ratios, position-aware weighting helps substantially, while content-only weighting barely helps.

Critical weaknesses
  1. The equal-loss savings are my main concern. Section 3.3 correctly explains why native losses across k are not directly comparable, but Table 4 uses exactly those losses to obtain the 1.28× and 1.17× savings. Rescoring the final checkpoints does not fix the earlier curves. At 500M, the target is 3.073, whereas Table 1 gives a final baseline full-position NLL of 3.068. I cannot tell what saving holds at the latter target.

  2. I could not reconcile the training setup. Section 5.2 and Table 2 say both raw tokens per update and update counts are fixed, yet the averaged runs process twice as many tokens. These cannot all be true. If the raw-token batch is fixed, averaging also changes the prediction batch and number of updates, so the improvement is not cleanly attributable to more data. The 50M mean-pooling result also changes from 4.275 in Table 1 to 4.358 in Table 3; this gap is larger than the reported main improvement.

Questions for authors
  1. Can you rescore checkpoints around the loss crossings with matched full-position evaluation and recompute the savings, including the 500M target of 3.068?

  2. What are the actual update counts and prediction batch sizes? What explains the difference between the mean-pooling results in Tables 1 and 3?

  3. With all offset caches included, what are the measured peak memory and decoding throughput?

2 comments

Thank you for the careful reading; your questions identified real problems, which we have corrected.

Q1. Equal-loss savings with matched full-position evaluation. We rescored all available intermediate and final checkpoints with full-position NLL on one set of 3,339 sequences, using the k=1k{=}1 final loss on that set as the target. Revised ratios: 1.29× (50M, 5 seeds), ≥1.07× (125M; lower bound, no intermediate checkpoints), 1.13× (250M; previously 1.17×), 1.08× (500M). The 500M target. 3.073 and 3.068 came from two different evaluation subsets (training-loop evaluation vs. final rescore). A crossing is only meaningful when the target and the curve use the same set. On the 3,339-sequence set, the baseline ends at 3.0706; k=2k{=}2 reaches this value at 1.08× less compute and ends at 3.0688 with 0.95× the FLOPs.

Q2. Update counts and prediction batch. You are right that the text was inconsistent. Raw tokens per update are fixed; k=2k{=}2 performs twice as many updates, each with half as many predictions. Total predictions are equal.

ScaleRaw tokens/updatek=1k{=}1 updatesk=2k{=}2 updates
50M16,38461,036122,071
125M98,30425,43250,863
250M32,768152,588305,176
500M32,768305,176610,352

Predictions per update are equal to the raw tokens per update for k=1k{=}1 and half of that for k=2k{=}2. To separate data from update count, we ran controls at 50M (3–5 seeds each). At a fixed 8K raw tokens per update, k=2k{=}2 is 0.095 lower (4.041 vs. 4.136); at 16K, 0.048 lower. At an equal number of updates, k=1k{=}1 is lower, so part of the gain comes from the additional, cheaper updates that averaging makes possible within the same FLOPs. This is now stated in Section 5.1, with the controls in Appendix H.

Tables 1 vs. 3. The Table 3 runs were trained earlier with a previous code version and a different seed. On the same evaluation set the two checkpoints still differ (4.358 vs. 4.287), so the difference comes from the training runs, not the evaluation. With the final code, 5 seeds of k=2k{=}2 give 4.295 ± 0.003 vs. 4.342 ± 0.002 for k=1k{=}1. Table 3 now includes same-code runs next to this baseline, in a separate block.

Q3. Peak memory and decoding throughput with all offset caches. With kk rotating caches, total KV memory matches the baseline, and we withdraw the KV-saving claim. Measured at 500M (RTX A6000, bf16, 256 new tokens):

ContextBatchKV GiB (k=1k{=}1 / k=2k{=}2)Peak GiB (k=1k{=}1 / k=2k{=}2)Throughput (k=2k{=}2 vs. k=1k{=}1)
204810.24 / 0.241.33 / 1.291.03×
2048327.73 / 7.7411.39 / 10.151.44×
819287.09 / 7.0910.75 / 9.501.24×

A single-cache variant reduces KV memory by k×k\times; with learned pooling it costs 0.015–0.040 nats and remains below the k=1k{=}1 baseline.

Presentation. Section 4 now states the evaluation protocol and the training setup (update counts, prediction batch) precisely, and Section 5.2 is corrected.

We hope these corrections address your concerns.

Thanks for the revision. However, due to the inconsistency between the revised results and the original version, I decided to keep my score. I hope the authors carefully check the paper before the next submission to a conference.

Reviewer_iLBROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies token averaging for efficient language model pretraining, merging groups of consecutive token embeddings and predicting the first token of the next group. Although compression introduces information loss, authors suggest that training on more tokens can compensate this degradation and improve NLL while reducing estimated training FLOPs.

However, I have some concerns:

  1. The diminishing NLL gains as model size increases. For 0.5 B model, this method cannot bring enough benefits. As the smaller model do not need data as much as super large model, this work may not be practical in real world settings.

  2. The lack of comparisons with the closest prior methods, particularly Patch-Level Training and TST, under matched total training FLOP budgets.

  3. Whether the reported FLOP savings translate into actual wall-clock training speedups.

Critical strengths

The paper is clearly written, with its core method easy to follow. The paper presents a simple, parameter-free token-averaging method requiring minimal changes to a standard Transformer. At the tested scales, (k=2) enables training on twice as many raw tokens with 5–7% fewer estimated training FLOPs, achieving lower NLL at 50M–250M and comparable NLL at 500M. Random-offset training and full-position evaluation address the mismatch in prediction positions introduced by grouping. The context-length and pooling ablations also provide useful insights into the loss–compute trade-off and the importance of within-group positional information.

Critical weaknesses
  1. The authors aim to improve pretraining efficiency. However, pretraining small models requires relatively little compute and data, making the motivation for focusing on this setting unclear. Furthermore, the experimental in table 1 results show diminishing gains as model size increases, raising concerns about whether these benefits would persist at larger scales.

  2. Missing comparisons with the closest prior methods. Patch-Level Training and TST are discussed but not evaluated under matched total training FLOP budgets, including their recovery stages. This leaves unclear whether the proposed boundary-token prediction and permanent compression offer a better loss–compute trade-off than these closely related two-stage approaches, since this paper just compare with k=1 baseline.

  3. Missing wall-clock training measurements. The reported efficiency gains rely on analytically estimated FLOPs, which do not necessarily translate into shorter training time. Do fewer FLOPs but more training batches actually reduce wall-clock training time?

Questions for authors
  1. Could you evaluate the method on larger models? If resources are limited, maybe you provide additional cross-scale analysis explaining the diminishing NLL improvements and discussing whether the benefits may persist at larger scales?

  2. How does the proposed method compare with the most closely related baselines?

  3. Do fewer FLOPs but more training batches actually reduce wall-clock training time?

The first question is the most important one. If authors could answer this question, I will give a weak reject. If other questions could be solved, I'd like to improve the rating to weak accept.

2 comments

Thank you for the clear review and for stating which results would change your assessment.

Q1. Larger models / cross-scale analysis. We are scaling the method to 1B parameters and beyond; these runs are in progress and will be shared as soon as they are available (see the general response). In the meantime, we provide the cross-scale analysis you suggested, with all numbers on one evaluation set and full-position NLL:

50M125M250M500M
Δ NLL (k=2k{=}2 − k=1k{=}1)−0.048−0.037−0.014−0.002
Equal-loss compute ratio1.29×≥1.07×1.13×1.08×
FLOPs (k=2k{=}2 / k=1k{=}1)0.930.930.940.95
  • k=2k{=}2 is never worse than k=1k{=}1 at equal or lower FLOPs.
  • The diminishing gain is specific to mean pooling. At 50M, pooling that keeps positional information doubles the k=2k{=}2 gain (−0.099 ± 0.008 over 3 seeds) and reaches −0.21 and −0.24 at k=4k{=}4 and k=8k{=}8. Improving the pooling function is therefore the most promising route to larger gains at scale.
  • Zero-shot accuracy is preserved at all scales (average within 0.006 of the baseline), and at 500M k=2k{=}2 is higher on HellaSwag, PIQA and ARC-Easy.
  • The inference benefit increases with context length and batch size (decoding up to 1.44× faster at 500M).

Motivation for small models. The mechanism does not depend on model size. We study 50M–500M to allow controlled, multi-seed comparisons, and Section 6 now scopes our claims to the compute-limited, data-rich regime.

Q2. Comparison with PLT and TST. At 50M, FLOPs-matched including their recovery stages (2 seeds each):

MethodNLLΔ vs. k=1k{=}1
TST, s=4s{=}4 / s=8s{=}84.385 / 4.410+0.043 / +0.068
PLT, K=2K{=}2 / K=4K{=}4 (setup of Shao et al.)4.198 / 4.110−0.144 / −0.232
Ours, k=2k{=}2 mean / learned4.295 / 4.243−0.048 / −0.099

PLT reaches lower training loss at this budget; TST does not reach the baseline. Token averaging differs in that it keeps the compressed sequence after training: it is single-stage, has no stage schedule to tune, and is the only one of these methods with decoding gains.

Q3. Wall-clock. Yes. Measured on one RTX A6000 (bf16, compiled), k=2k{=}2 processes raw tokens 2.49× (50M), 1.97× (250M) and 1.96× (500M) faster than k=1k{=}1. It therefore trains on twice the data in 0.79×, 1.01× and 1.02× the wall-clock time of the baseline. At 250M–500M the measured time is within 8% of the FLOPs estimate; the difference comes mainly from the additional optimizer steps.

We hope this addresses your questions.

Thanks for the response. I decide to keep my current rating.

Reviewer_p9juOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Token Averaging, a simple method for improving the compute efficiency of language model pretraining. The method averages every (k) consecutive token embeddings before the Transformer, reducing the number of processed positions by a factor of (k). To address the mismatch in prediction positions, the authors introduce random-offset training and full-position evaluation using multiple offsets.

Experiments on FineWeb with models ranging from 50M to 500M parameters show that (k=2) can process twice as many raw tokens with approximately 5–7% fewer estimated training FLOPs. The method achieves lower perplexity at smaller scales and comparable performance at 500M. Additional ablations investigate the effects of context length and alternative pooling strategies, showing that position-aware learned pooling performs better than uniform averaging, particularly at larger compression ratios.

Critical strengths
  1. Simple and computationally motivated approach.

The proposed method is straightforward and requires minimal architectural modifications. Mean pooling introduces no additional parameters while reducing the number of positions processed by the Transformer. This simplicity makes the approach potentially practical and easy to integrate into existing language model pretraining pipelines.

  1. Meaningful evaluation design and empirical analysis.

The random-offset training and full-position evaluation protocols address an important issue in comparing models trained with different prediction positions. The experiments across four model scales, together with the context-length and pooling ablations, provide useful insights into the trade-offs between compression, training data utilization, and model performance.

Critical weaknesses
  1. Limited scalability and experimental robustness.

The experiments are limited to models up to 500M parameters, with improvements diminishing as model size increases. Moreover, most results are based on single runs, leaving the scalability and statistical robustness of the proposed method insufficiently validated.

  1. Inconsistencies in evaluation results.

Appendix B uses native losses instead of full-position NLL for some equal-loss comparisons. Additionally, the 50M mean-pooling results differ between Tables 1 and 3 (4.275 vs. 4.358). These discrepancies require clarification to ensure consistent and reliable evaluation.

  1. Insufficient validation of practical efficiency.

The reported efficiency gains are primarily based on theoretical FLOPs estimates, without actual training throughput or wall-clock measurements. Furthermore, the need for multiple offset-specific KV caches during generation makes the practical inference benefits unclear.

Questions for authors

Q1. Scalability: Could the authors provide additional evidence of scalability beyond 500M parameters and evaluate the stability of the improvements across different random seeds?

Q2. Pooling strategies: The position-aware pooling method achieves substantial improvements at larger compression ratios. Could the authors clarify whether these improvements remain consistent when different pooling methods use the same initialization strategy?

Q3. Practical efficiency: Could the authors provide actual throughput measurements and clarify the total KV-cache memory requirements during autoregressive generation?

2 comments

Thank you for the positive assessment and the specific questions.

Q1. Scalability and seeds. We agree that larger scales are the most important next test. Runs at 1B parameters and beyond are in progress; because a 1B pair requires roughly four times the compute of our 500M pair, they will not complete within the discussion period, and we will share them as soon as they are available (see the general response). On one evaluation set, k=2k{=}2 is lower than k=1k{=}1 at every scale with 5–7% fewer FLOPs (−0.048, −0.037, −0.014, −0.002 from 50M to 500M), and full-position equal-loss ratios are 1.29×, ≥1.07×, 1.13× and 1.08×. At 50M we ran 5 seeds each: Δ = −0.048 ± 0.004 (about 12 SE), with every k=2k{=}2 seed below every k=1k{=}1 seed. With learned content+position pooling (3 seeds), Δ = −0.099 ± 0.008. The shrinking gain is specific to mean pooling; pooling that keeps positional information doubles the gain at k=2k{=}2 and reaches −0.21/−0.24 at k=4k{=}4/88.

Q2. Pooling with the same initialization. We retrained content+position pooling at k=4k{=}4 and k=8k{=}8 with the same uniform (mean) initialization as content-only pooling:

Pooling (initialization)k=4k{=}4k=8k{=}8
Mean (no parameters)4.4404.814
Content-only (uniform)4.4354.811
Content+position (recency, paper)4.1544.093
Content+position (uniform, new)4.1334.100
k=1k{=}1 baseline (5 seeds)4.3424.342

Initialization changes content+position by at most 0.021. Under the same initialization it remains 0.30 (k=4k{=}4) and 0.71 (k=8k{=}8) below content-only, so the improvement comes from positional information rather than initialization.

Q3. Practical efficiency. Training wall-clock (one RTX A6000, bf16, compiled): k=2k{=}2 processes raw tokens 2.49× (50M), 1.97× (250M) and 1.96× (500M) faster than k=1k{=}1, so it trains on twice the data in 0.79×, 1.01× and 1.02× the time of the baseline. KV memory: we confirm that generation with kk rotating caches uses the same total KV memory as the baseline, and we withdraw the KV-saving claim. Decoding is faster because each step attends to T/kT/k entries: up to 1.44× at 500M (context 2048, batch 32) and 1.24× at context 8192, batch 8. A single-cache variant reduces KV memory by k×k\times and, with learned pooling, stays within 0.015–0.040 nats of the full method and below the k=1k{=}1 baseline.

Inconsistencies. (i) All equal-loss ratios are now computed on full-position curves from rescored checkpoints. (ii) The 4.275 vs. 4.358 difference comes from two different training runs (earlier code version and a different seed), not from the evaluation: on the same evaluation set they still differ (4.287 vs. 4.358). Table 3 now includes runs with the final code next to the 5-seed baseline, in a separate block.

We hope these results strengthen your assessment.

Thank you for the detailed response and additional experiments addressing my concerns. I will maintain my original rating of 3.

Reviewer_7RtXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

With compute-optimal scaling, the data budget grows at least as quickly as the model. However, each raw token still costs one full transformer position. Therefore, more data means proportionally more compute is required. This paper addresses this issue by investigating token averaging. This method pools every k consecutive input embeddings before the first transformer layer. Therefore, the model learns and runs on fewer positions, predicting the first token of the next pooled group. Random boundary offsets during training and an offset ensemble during evaluation are employed to ensure that the losses are comparable to those of a standard model. The paper demonstrates that with equal compute, a k=2 model trained on double the training data achieves lower loss at smaller model scales and equivalent loss at the largest tested scale. Furthermore, the authors claim that KV cache requirements are reduced during inference. They conclude that the observed language modeling gains stem from more data rather than longer context and that loss can be further reduced by maintaining positional information within the pool.

Critical strengths
  1. It is a clean, parameter-free method that integrates quickly. It is evaluated using first-principles FLOP accounting that includes the output head, making every ratio more conservative and accurate.

  2. The offset-ensemble evaluation is well motivated and makes losses comparable across pool size k and baselines. The low per-offset difference across model sizes (Tab 5) is an interesting finding.

  3. A clean and controlled experiment that decomposes the method into a 2x2 data versus context grid at 50M, showing that data is the main driver of the method's advantages.

Critical weaknesses
  1. Two of the four headline results, the equal-loss ratios for the 125M (1.28x) and 250M (1.17x) models, can be found in Table 4 and are computed from the native losses. However, Section 3.3 states that the native losses of the baseline and a k-averaged model 'average different prediction sets and are not directly comparable.' This is precisely the comparison that Table 4 makes at those two scales. Table 1 further illustrates the extent of the issue: at 250M, the full-position loss is 0.039 (k=2) and 0.032 (k=1) higher than the native loss, which is greater than the measured effect of k=2 over k=1 (0.011, as stated in L323). For k = 1, the full-position value should be, if even, slightly lower than the native value, because it drops the first position. Instead, it differs by a varying amount across scales (see questions).

  2. The abstract claims 'half the KV cache,' but performing inference with a k=2 model implies switching between two KV caches. Each cache is indeed 1/k in size, but both are needed. In App. C, the authors state that generation keeps k caches of length 1/k, i.e., the same memory footprint as the baseline. To me, this makes the abstract misleading. Further, a metric that captures the practical cost, such as decoding throughput or memory at generation was never measured

  3. The gain shrinks with scale (1.30, 1.28, 1.17, 1.08, Tab.1), while the method requires twice the data, which is clearly limiting in a data-limited regime and should be stated.

  4. Evaluations: Single-seed experiments per condition. Currently, at the 250M and 500M scales, the differences could be within the margin of error. The only baseline is k=1. Patch-level training, token superposition training, and a coarser tokenizer are discussed in Section 2 but never run and adding some of them would ground this method much better. Lastly, there are no downstream evaluations. Recomputing the ratios on full-position curves, adding seeds, and correcting the KV-cache claim above would address my main concerns, and I would adjust my score accordingly.

  5. Minor: Maintext Tables 2 and 3 are never referenced from the main text.

Questions for authors
  1. The pooling runs in Table 3 appear to be from a different batch than those in Table 1. The shared k=2 mean-pooling configuration differs by 0.083 nats between the two tables. This delta is larger than the 50M headline gain of 0.049 (Table 1, Full Position column). What is different between the two batches? Without an k=1 baseline that is trained or evaluated on the same batch, how should the k=8 content+position result be interpreted? (k=8 with 4.093 versus k=1 baseline with 4.324).

  2. Why do the native and full-position losses of the k=1 baseline differ in sign and magnitude across scales (0.0, 0.009, 0.032, -0.005, Tab1: Loss columns, k=1 rows)? Did you use the same checkpoints and evaluation sets?

  3. Is it possible to recompute the 125M/250M equal-loss ratios offline on full-position curves? This would make the numbers in Table 4 comparable and make the headline results more trustworthy.

  4. What is the KV memory and decoding throughput with the same prefill and k rotating caches?

2 comments

Thank you for the detailed review and for stating clear conditions. We addressed all three: full-position ratios, seeds, and the KV-cache claim.

Equal-loss ratios from native curves (W1, Q3). We rescored all available intermediate and final checkpoints with full-position NLL on one set of 3,339 sequences. The revised ratios are 1.29× (50M, 5 seeds), ≥1.07× (125M, lower bound, no intermediate checkpoints), 1.13× (250M; previously 1.17×), and 1.08× (500M). The saving holds at every scale under the matched metric (table in the general response).

Native vs. full-position for k=1k{=}1 (Q2, and the 250M gap). Same checkpoints, different evaluation subsets: "native" was the last periodic training-loop evaluation (a fixed number of batches, so the subset depends on the batch size), and "full-position" was a final rescore on all 3,339 sequences. For k=1k{=}1 the two metrics are identical by definition, and our rescoring confirms equality at all scales. On one set, k=2k{=}2 minus k=1k{=}1 is −0.048, −0.037, −0.014 and −0.002 from 50M to 500M. All numbers in the revision use this set.

Seeds (W4). At 50M, 5 seeds each: k=1k{=}1 4.342 ± 0.002, k=2k{=}2 mean pooling 4.295 ± 0.003 (Δ = −0.048 ± 0.004, about 12 SE). Every k=2k{=}2 seed is below every k=1k{=}1 seed. Learned content+position pooling (3 seeds): 4.243 ± 0.008 (Δ = −0.099). Seed s.d. is 0.005–0.007, smaller than the 50M–250M differences; at 500M (−0.002) we describe the result as on par.

KV-cache claim (W2, Q4). You are right, and we apologize for the misleading wording; we withdraw the claim. With kk rotating caches and the same prefill, KV memory equals the baseline (500M, context 2048, batch 32: 7.73 vs. 7.74 GiB). Decoding throughput, 500M, RTX A6000, bf16:

ContextBatchk=1k{=}1 tok/sk=2k{=}2 tok/sSpeedup
2048145461.03×
204832101814681.44×
819282923631.24×

The gain appears where attention dominates (long context, larger batches). A single-cache variant with k×k\times less KV memory is described in the general response.

Diminishing gain and 2× data (W3). Section 6 now states this explicitly. For the data-limited regime, we trained k=2k{=}2 for 2 epochs on the baseline's 1B unique tokens at equal FLOPs: 4.300 vs. 4.342 for k=1k{=}1 (−0.042; 1.31× equal-loss saving). Repetition costs only 0.005 compared with fresh data, so the method does not require additional unique data at this scale.

Baselines (W4). At 50M, 2 seeds each, FLOPs-matched including the recovery stage:

MethodNLLΔ vs. k=1k{=}1
TST, s=4s{=}4 / s=8s{=}84.385 / 4.410+0.043 / +0.068
PLT, K=2K{=}2 / K=4K{=}4 (setup of Shao et al.)4.198 / 4.110−0.144 / −0.232
Ours, k=2k{=}2 mean / learned4.295 / 4.243−0.048 / −0.099

PLT reaches lower training loss; TST does not reach the baseline. Unlike both, token averaging keeps the compressed sequence after training, which is what enables the decoding gains above, and it is single-stage.

Table 1 vs. Table 3 (Q1). The Table 3 runs were trained earlier with a previous code version and a different seed. On the same evaluation set the two checkpoints still differ (4.358 vs. 4.287), so the gap comes from the training runs, not the evaluation. We retrained the pooling variants with the final code next to the 5-seed baseline: k=1k{=}1 4.342, k=4k{=}4 content+position 4.133, k=8k{=}8 content+position 4.100. Against a baseline trained and evaluated identically, k=8k{=}8 content+position is 0.24 lower. Table 3 now includes these runs in a separate block, and its caption and Section 5.3 state that the earlier runs are compared only within that block.

Downstream. Zero-shot average accuracy over five tasks is within 0.006 of the baseline at every scale (details in our response to dUat).

Minor. Tables 2 and 3 are now referenced in Sections 5.2 and 5.3.

We hope these results address your main concerns.

Thank you for the detailed response and revisions. They address my initial concerns and I appreciate how openly the new controls are reported.

However, these controls change my view of the contribution. With the same number of updates and similar FLOPs, the k=1 baseline achieves a lower loss than the main k=2 model (see Table 12). As you noted, part of the reported gain comes from the additional updates. Patch-level training achieves a lower loss with the same budget. The remaining benefit at inference depends on the batch size. Therefore I maintain my rating.

Reviewer_dUatOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Training standard language models on more text requires proportionally more compute because the Transformer processes one sequence position and output prediction per raw token. To process more training data within a fixed compute budget, the authors propose Token Averaging, a parameter-free method that replaces each non-overlapping group of kk consecutive token embeddings with their mean before the first Transformer layer and predicts only the first token of the next group. To align training and evaluation across all token positions despite the kk-fold sequence compression, the paper introduces random-offset training and an offset-ensemble evaluation protocol, alongside alternative pooling functions including a learned content-and-position variant. Experiments with GPT-NeoX-style models from 50M to 500M parameters on FineWeb show that Token Averaging (k=2k=2) trains on twice as many raw tokens using 5–7% fewer training FLOPs and achieves lower full-position negative log-likelihood (NLL) at 50M–250M while reaching comparable loss at 500M (Table 1).

Critical strengths
  • Simple Parameter-Free Formulation and Well-Designed Evaluation Protocol Unlike hierarchical sequence-compression models that require separate local decoders (Lines 118–123) or patch-based methods that rely on a multi-stage schedule returning to token-level training (Lines 107–111), Token Averaging keeps the tokenizer, vocabulary, and Transformer backbone completely unchanged (Lines 044–048, 135–136, 149–152). For final-checkpoint comparisons, random-offset training and kk-pass offset-ensemble evaluation (Sections 3.2 and 3.3, Lines 182–207) align the scored target positions across different compression ratios kk, allowing full-position likelihood comparisons against the uncompressed baseline at roughly the cost of a single baseline forward pass. This comparability does not extend to every reported efficiency estimate: the equal-loss ratios at 125M and 250M are computed from native loss curves (Table 4, Lines 594–603).

  • Informative Data and Context Comparisons and Transparent FLOPs Accounting The 2×22 \times 2 experiment in Section 5.2 (Table 2) compares configurations that vary compression and raw context length. Switching from the baseline to k=2k=2 with twice the training data at the same raw context length lowers loss by 0.049 nats at 0.928×0.928\times compute; doubling the raw context within the k=2k=2 configuration lowers loss by a further 0.026 nats while raising compute to the baseline's level. These contrasts are informative, although the first changes both averaging and data volume, and the stated update counts require clarification (Q5). In addition, the authors explicitly include the dense vocabulary projection (2dV2dV) in their FLOPs accounting (Eq. 2, Section 3.5, Lines 225–239), transparently avoiding the inflated efficiency ratios that would arise from counting only Transformer body operations at small scales.

  • Clear Presentation and Informative Ablations on Pooling Mechanics The paper is well-structured and easy to follow, illustrating the compression pipeline intuitively in Figure 1 and summarizing the experimental comparisons clearly in Tables 1–3. The systematic evaluation of pooling variants across k∈{2,4,8}k \in \{2, 4, 8\} (Sections 3.6 and 5.3, Table 3) suggests that preserving within-group positional information becomes increasingly important at higher compression ratios, although each entry is a single run (Line 377).

Critical weaknesses

W1. Diminishing Efficiency Gains with Scale and Data Efficiency Concerns Averaging is lossy (Lines 056–057), and the reported equal-loss compute saving decreases across the tested scales. At 500M, the k=2k=2 model uses twice the raw data while reaching only comparable endpoint loss (Table 1). This raises concerns about data efficiency, while performance under a fixed unique-data budget with repeated training remains untested.

W2. Lack of Comparison with Alternative Efficiency Methods At the 2×2\times-data endpoints in Table 1, the remaining 5–7% reduction in total training FLOPs comes from shorter attention sequences (Lines 244–247). However, the paper does not compare against the closest alternatives, such as patch-level training (Lines 107–111) or a coarser tokenizer (Lines 420–421). It also does not discuss how its inference savings relate to, or combine with, standard KV-cache reduction methods such as GQA/MQA.

W3. Gap Between Single-Pass KV-Cache Claims and Autoregressive Decoding The 1/k1/k KV-cache reduction holds per pass (Lines 021–024), but token-by-token generation keeps kk offset caches (Appendix C), so the total decoding memory roughly matches the baseline. No decoding throughput or latency is reported.

W4. Evaluation Restricted to Pretraining Perplexity All results are language-modeling losses on FineWeb (Tables 1–3), with no downstream evaluation (Lines 420–421). It remains unclear whether comparable NLL translates into preserved zero-shot accuracy.

Questions for authors

Q1. Scalability Beyond 500M and Data-Constrained Performance Does Token Averaging maintain any compute advantage at 1B+ scales, or in data-constrained regimes where the unique-token budget is fixed and data must be repeated over multiple epochs?

Q2. Comparison with Alternative Efficiency Methods (e.g., Patch-Level Training, GQA) How does Token Averaging compare with patch-level training or a coarser-tokenizer baseline at matched compute, and how do its inference savings compare to or combine with head-level KV-cache reduction methods like GQA that avoid token-level information loss?

Q3. Memory Footprint and Throughput in Autoregressive Decoding Appendix C states that token-by-token generation keeps kk caches of length T/kT/k each (Lines 629–632). Could you confirm that the total KV memory during decoding therefore approximately matches the baseline's, and report actual decoding throughput and latency?

Q4. Zero-Shot Downstream Benchmark Evaluation Could you report zero-shot performance on standard downstream tasks (e.g., HellaSwag, ARC, PIQA, or LAMBADA) for the models in Table 1 to confirm whether task accuracy is preserved despite input averaging?

2 comments

Thank you for the thorough review.

W1/Q1. Scale and data-constrained regime. We are scaling the method to 1B parameters and beyond; these runs are in progress and will be shared as soon as they are available (see the general response). Across 50M–500M, k=2k{=}2 is lower than k=1k{=}1 at every scale with fewer FLOPs, and the full-position equal-loss ratios are 1.29×, ≥1.07×, 1.13× and 1.08×. For the data-constrained regime, we trained k=2k{=}2 for 2 epochs on the baseline's 1B unique tokens at equal FLOPs: 4.300 vs. 4.342 for k=1k{=}1 (−0.042; 1.31× equal-loss saving). Fresh data gives 4.295, so repetition costs only 0.005. Section 6 now states the data requirement and this result explicitly.

W2/Q2. Alternative methods. At 50M, FLOPs-matched including recovery stages (2 seeds each): PLT (setup of Shao et al., K=2K{=}2/K=4K{=}4) 4.198/4.110; TST (s=4s{=}4/s=8s{=}8) 4.385/4.410; ours 4.295 (mean) and 4.243 (learned) vs. 4.342 for k=1k{=}1. PLT reaches lower training loss, and TST does not reach the baseline. Token averaging is the only one of these methods whose final model operates on the compressed sequence, which gives decoding gains. GQA/MQA: the two are complementary. GQA/MQA reduce the KV size per position (fewer KV heads), while token averaging reduces the number of positions attended per step (rotating caches) or stored (single cache). The savings multiply: e.g., 4× fewer KV heads and a k=2k{=}2 single cache give 8× less KV memory than a multi-head k=1k{=}1 model. This discussion is added to Section 2.

W3/Q3. KV memory and throughput. We confirm that total KV memory during decoding matches the baseline (500M, context 2048, batch 32: 7.73 vs. 7.74 GiB), and we withdraw the KV-saving claim with apologies for the wording. Measured decoding throughput (500M, RTX A6000, bf16, 256 new tokens): 1.03× at batch 1, 1.44× at batch 32 (context 2048), and 1.24× at context 8192, batch 8. A single-cache variant reduces KV memory by k×k\times; with learned pooling it costs 0.040/0.028/0.015 nats at k=2/4/8k{=}2/4/8 and remains below the k=1k{=}1 baseline.

W4/Q4. Zero-shot downstream (50M: seed means; k=1k{=}1 / k=2k{=}2):

ScaleLAMBADAHellaSwagPIQAARC-EWinograndeAverage
50M.355 / .350.248 / .247.555 / .568.267 / .280.505 / .508.386 / .391
125M.377 / .367.256 / .254.566 / .575.274 / .278.517 / .500.398 / .395
250M.482 / .453.288 / .287.635 / .629.327 / .348.523 / .508.451 / .445
500M.545 / .515.336 / .346.647 / .662.377 / .403.519 / .517.485 / .489

Average accuracy is within 0.006 of the baseline at every scale. ARC-Easy improves at all scales; LAMBADA, which requires the exact next word, is 0.005–0.030 lower.

Q5. Update counts. Raw tokens per update are fixed, so k=2k{=}2 performs twice as many updates, each with half the predictions (50M: 61,036 vs. 122,071 updates). Section 5.2 is corrected. In an update-count control at 50M, k=2k{=}2 is also lower at a smaller fixed batch (8K raw tokens per update: 4.041 vs. 4.136).

Equal-loss ratios at 125M/250M. Recomputed on full-position curves: ≥1.07× (125M, lower bound) and 1.13× (250M).

We hope these additions address your concerns.

Thank you for the thorough response and additional experiments.

  • W3/Q3 & W4/Q4: I sincerely appreciate your transparent correction regarding the KV-cache memory claim and the new decoding benchmarks. The zero-shot downstream results also show meaningful performance above random chance across scales, which resolves our concern on downstream evaluation.
  • W1/Q1 (Scale): While I fully understand that completing 1B+ runs within the rebuttal period is practically difficult, the diminishing gains from 50M to 500M make it hard to judge the practical utility of the method for language model pretraining without 1B+ validation.
  • W2/Q2 (Baselines): Thank you for the new comparisons and the GQA/MQA discussion. However, the method does not show a clear advantage over PLT at matched FLOPs, and a comparison with a coarser tokenizer still appears necessary.

I truly appreciate the significant effort you put into this rebuttal.
Unfortunately,

  • mainly because the practical utility at 1B+ scales remains unverified
    and
  • the advantage over PLT is not clear,

I regrettably decided to maintain our current score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

General response to all reviewers

We thank all reviewers for their careful reading and constructive feedback. Several concerns were shared, so we summarize the new experiments and corrections here and answer each reviewer individually below.

1. One evaluation set and full-position curves. All endpoints are now rescored with full-position NLL on the same 3,339 held-out sequences, and the equal-loss ratios are recomputed from full-position curves (rescored intermediate checkpoints at 250M and 500M; full-position training-loop curves at 50M):

Scalek=1k{=}1k=2k{=}2Δ\DeltaFLOPs (k=2k{=}2/k=1k{=}1)Equal-loss ratio (paper → revised)
50M (5 seeds)4.3424.295−0.048 ± 0.0040.931.30× → 1.29×
125M4.1794.141−0.0370.931.28× → ≥1.07×*
250M3.4153.401−0.0140.941.17× → 1.13×
500M3.0713.069−0.0020.951.08× → 1.08×

*No intermediate 125M checkpoints exist; since the k=2k{=}2 final loss is already below the baseline's final loss at 0.93× the FLOPs, the ratio is at least 1/0.93.

2. KV-cache claim (correction). We apologize for the wording and withdraw the claim of KV-cache savings during generation. As several reviewers pointed out, token-by-token decoding keeps kk caches of T/kT/k entries, so total KV memory matches the baseline (measured: 7.73 vs. 7.74 GiB). What token averaging does reduce is the attention work per generated token, since each step attends to one cache of T/kT/k entries. Measured decoding throughput at 500M improves by up to 1.44× (context 2048, batch 32). In addition, a single-cache variant that reduces KV memory by k×k\times stays within 0.015–0.040 nats of the full method with learned pooling, and remains below the k=1k{=}1 baseline. The abstract and all related statements have been rewritten.

3. New experiments.

  • Seeds: 5 seeds each for k=1k{=}1 and k=2k{=}2 at 50M; every k=2k{=}2 seed is below every k=1k{=}1 seed.
  • Update-count controls and corrected update counts (Sections 4, 5.1, 5.2; Appendix H).
  • Data-constrained regime: k=2k{=}2 trained for 2 epochs on the baseline's unique data keeps a −0.042 gain.
  • FLOPs-matched baselines: patch-level training (PLT) and token superposition training (TST) (Section 5.4).
  • Measured training throughput and decoding benchmarks (Section 5.5; Appendices J, L).
  • Zero-shot downstream evaluation (LAMBADA, HellaSwag, PIQA, ARC-Easy, Winogrande) at all four scales (Section 5.5; Appendix K).
  • Pooling with the same initialization at k=4k{=}4 and k=8k{=}8.
  • Larger scales: we are scaling the method to 1B parameters and beyond, and these runs are in progress. A 1B pair at our token budget requires roughly four times the compute of the 500M pair, so it will not finish before the discussion period ends. We will post the results in this forum if they become available while it is open, and they will be included in the final version.

4. Scope. Section 6 now states explicitly that the method targets the compute-limited, data-rich regime, that the gain of mean pooling decreases with scale, and that k=2k{=}2 requires twice the raw data (or repeated data, see item 3).

All new results and corrections are included in the revised PDF; changes are marked in blue, and the main text remains within 10 pages.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.