- Rating
- 1
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies a simple trade-off: average adjacent token embeddings, process fewer positions, and spend the saved compute on more training text. Unlike patch-level methods that later return to token-level training, it keeps the compressed representation throughout. Random offsets vary the prediction targets during training, and evaluation combines offsets to score the same target positions as the baseline. Experiments cover 50M–500M models, with small improvements in endpoint NLL at the smaller scales and similar results at 500M.
Critical strengths
-
I like the simplicity of the setup. The tokenizer and transformer body stay the same, so the main architectural change is easy to understand. The full-position endpoint results, especially at 50M and 125M, suggest this trade-off is worth studying.
-
The pooling ablation gives a useful result: at larger compression ratios, position-aware weighting helps substantially, while content-only weighting barely helps.
Critical weaknesses
-
The equal-loss savings are my main concern. Section 3.3 correctly explains why native losses across k are not directly comparable, but Table 4 uses exactly those losses to obtain the 1.28× and 1.17× savings. Rescoring the final checkpoints does not fix the earlier curves. At 500M, the target is 3.073, whereas Table 1 gives a final baseline full-position NLL of 3.068. I cannot tell what saving holds at the latter target.
-
I could not reconcile the training setup. Section 5.2 and Table 2 say both raw tokens per update and update counts are fixed, yet the averaged runs process twice as many tokens. These cannot all be true. If the raw-token batch is fixed, averaging also changes the prediction batch and number of updates, so the improvement is not cleanly attributable to more data. The 50M mean-pooling result also changes from 4.275 in Table 1 to 4.358 in Table 3; this gap is larger than the reported main improvement.
Questions for authors
-
Can you rescore checkpoints around the loss crossings with matched full-position evaluation and recompute the savings, including the 500M target of 3.068?
-
What are the actual update counts and prediction batch sizes? What explains the difference between the mean-pooling results in Tables 1 and 3?
-
With all offset caches included, what are the measured peak memory and decoding throughput?
2 comments
Thanks for the revision. However, due to the inconsistency between the revised results and the original version, I decided to keep my score. I hope the authors carefully check the paper before the next submission to a conference.
Thank you for the careful reading; your questions identified real problems, which we have corrected.
Q1. Equal-loss savings with matched full-position evaluation. We rescored all available intermediate and final checkpoints with full-position NLL on one set of 3,339 sequences, using the k=1 final loss on that set as the target. Revised ratios: 1.29× (50M, 5 seeds), ≥1.07× (125M; lower bound, no intermediate checkpoints), 1.13× (250M; previously 1.17×), 1.08× (500M). The 500M target. 3.073 and 3.068 came from two different evaluation subsets (training-loop evaluation vs. final rescore). A crossing is only meaningful when the target and the curve use the same set. On the 3,339-sequence set, the baseline ends at 3.0706; k=2 reaches this value at 1.08× less compute and ends at 3.0688 with 0.95× the FLOPs.
Q2. Update counts and prediction batch. You are right that the text was inconsistent. Raw tokens per update are fixed; k=2 performs twice as many updates, each with half as many predictions. Total predictions are equal.
Predictions per update are equal to the raw tokens per update for k=1 and half of that for k=2. To separate data from update count, we ran controls at 50M (3–5 seeds each). At a fixed 8K raw tokens per update, k=2 is 0.095 lower (4.041 vs. 4.136); at 16K, 0.048 lower. At an equal number of updates, k=1 is lower, so part of the gain comes from the additional, cheaper updates that averaging makes possible within the same FLOPs. This is now stated in Section 5.1, with the controls in Appendix H.
Tables 1 vs. 3. The Table 3 runs were trained earlier with a previous code version and a different seed. On the same evaluation set the two checkpoints still differ (4.358 vs. 4.287), so the difference comes from the training runs, not the evaluation. With the final code, 5 seeds of k=2 give 4.295 ± 0.003 vs. 4.342 ± 0.002 for k=1. Table 3 now includes same-code runs next to this baseline, in a separate block.
Q3. Peak memory and decoding throughput with all offset caches. With k rotating caches, total KV memory matches the baseline, and we withdraw the KV-saving claim. Measured at 500M (RTX A6000, bf16, 256 new tokens):
A single-cache variant reduces KV memory by k×; with learned pooling it costs 0.015–0.040 nats and remains below the k=1 baseline.
Presentation. Section 4 now states the evaluation protocol and the training setup (update counts, prediction batch) precisely, and Section 5.2 is corrected.
We hope these corrections address your concerns.