- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
PipeT is an autoregressive architecture that pipelines computation across token positions to accelerate decoding and reduce KV-cache memory. Mixed-layer attention enables interactions across layers, extending computation before the input layer and after the output layer. Within-layer recurrence increases formal expressivity, enabling state tracking on tasks where fixed-depth Transformers fail.
Critical strengths
PipeT is a creative architecture motivated by inference efficiency. The reported inference gains are impressive and appear to increase with model size. The authors conduct extensive experimental sweeps to determine compute optimal scaling. Beyond efficiency, the circuit-complexity analysis and supporting state-tracking experiments provide a compelling connection between theoretical expressivity and empirical capabilities. The architecture opens several promising directions for future work.
Critical weaknesses
The training cost remains insufficiently characterized. The authors acknowledge slower training and prefill, informally reporting a 3-10× training slowdown (lines 427–428), but this tradeoff is essential to interpreting Figure 2 and assessing the feasibility of the architecture. The appendix’s token counts and (6ND) compute estimates do not capture the wall-clock implications of sequential dependencies across tokens rather than layers. Even without an optimized implementation, performance modeling or basic roofline estimates would help quantify the overhead.
Held-out loss alone provides an incomplete evaluation of generation quality. The paper draws inspiration from diffusion models, where aggressively parallel sampling from predicted marginals can lead to poor, incoherent samples. In a similar way, PipeT aims for greater parallelism when sampling autoregressively. In addition to the current likelihood based evaluation, a more thorough investigation of the samples generated from PipeT is warranted. A direct comparison of sampled text against standard Transformer++, measuring generative perplexity and entropy, would strengthen the evaluation.
Questions for authors
-
What are PipeT’s training and prefill costs relative to Transformer++, both empirically and under a reasonably optimized implementation or a theoretical performance model?
-
How does inference performance vary with batch size, and how does its interactivity-throughput curve compare with Transformer++ when sweeping batch sizes?
-
How competitive is PipeT against stronger inference baselines? Grouped-query attention, latent attention, and sliding-window attention offer alternative ways to reduce KV-cache pressure.
6 comments
complementary methods for KV cache reduction
How competitive is PipeT against stronger inference baselines? Grouped-query attention, latent attention, and sliding-window attention offer alternative ways to reduce KV-cache pressure.
Recall that our primary goal in this paper -- and its primary novelty -- was to widen the serialization bottleneck of decoding. PIPET accomplishes this, with negligible increase in loss. The alternatives you mention don't solve our problem, so we didn’t compare to their loss increases.
The fact that PIPET reduces KV cache size is just a nice side benefit, which appears “naturally” as a consequence of our architecture design. To review:
- Pipelining means that different tokens are working on different layers at the same time. We always attend to each token’s highest available layer. That allows lower layers to be discarded.
- In our 100M comparison, PIPET maintains comparable performance to Transformer++ while using only about 16% of Transformer++’s KV-cache memory (a 6.27× reduction; see Table 11).
- This shows that we do not need to represent different types of information in different layers (cf. ELMo). With increased dimensionality and end-to-end training, higher layers can simply replace all lower ones, retaining their information through residual connections and also adding new information.
You seem particularly interested in this ability. When reducing KV cache size is important, e.g. for long context windows, PIPET could be combined with the methods you name. We are excited to explore such combinations in future work (§E, lines 2087–2103).
In particular, PIPET’s cache reduction approach seems to be orthogonal to the other cache reduction methods you name. The total KV cache size in a standard Transformer is :
- PIPET reduces the factor (# of layers) [as does YOCO - see below]
- Grouped-query and sliding-window attention reduce the factor (# of tokens)
- Latent attention reduces the factor (# of hidden dimensions)
Reviewer 37LU mentioned YOCO. We discussed YOCO in the original submission and have expanded that discussion in §E (“KV-cache reduction,” lines 2096–2103) to clarify its relationship to PIPET. Both YOCO and PIPET reuse a shared KV cache for the full history. YOCO builds this cache halfway through the network for reuse by the upper half, called the cross-decoder, whereas PIPET retains the terminal layer’s keys and values. YOCO can skip the cross-decoder during prefill; the paper reports roughly 30× lower prefill latency at 512K tokens.
Our expanded discussion also reviews additional methods that selectively retain cached tokens: H₂O reports comparable accuracy using 20% of the cache (5× smaller); PyramidKV reports comparable LongBench performance using 12% (about 8.3× smaller); and HeadKV-R2 retains about 1.5% (about 67× smaller) while preserving 97% of mean F1 on six LongBench question-answering tasks with Llama-3-8B-Instruct.
evaluation of generation quality
Held-out loss alone provides an incomplete evaluation of generation quality…A direct comparison of sampled text against standard Transformer++, measuring generative perplexity and entropy, would strengthen the evaluation
We agree that evaluation of generated text would complement our log-loss of observed text. We already acknowledged in §5 (lines 498–503) that teacher-forced evaluation does not capture how errors accumulate during free generation. We have expanded that discussion somewhat in the new version of §5.
However, we had imagined doing task-based evaluation. The trouble is that generation quality of all models is poor at 200M parameters. None of them can solve real tasks, so such a comparison at our current scale may be uninformative.
We see three possible ways to compare at this scale:
- We could mix the next-token probabilities of either PIPET or Transformer++ with those of the same larger model. Varying the weight given to each smaller model would let us compare its effect on generation quality, while the larger model helps maintain coherent output.
- We could evaluate PIPET as a draft model for speculative decoding, measuring how often a larger target model accepts its proposed tokens.
- We could abandon task-based evaluation and simply evaluate whether each model’s freely generated text goes off the rails, making it unlike real text. For example, we could evaluate its log-probability under a fixed open-weight 8B model.
Is 3. what you were suggesting when you wrote “generative perplexity and entropy”? We do not have any of these comparisons yet, but agree that we should add one to the paper.
cost analysis
Fair questions. We have answered them in detail in this post and in the newly added Appendix D.
As a highlight, §D.3 uses a timing model fitted to the measured FP32 timings to extrapolate end-to-end runtime and throughput on real-world benchmark traces. Table 4 estimates roughly 2–7× improvements in output throughput, including prefill and decoding time, on seven of eight benchmarks; long-context retrieval (MRCR) is the exception, with PIPET reaching 0.53× Transformer++’s speed at batch 128 (§D.3.3, lines 2029–2037).