acceptodds
Under review as a conference paper at ICLR 2027

Pipelined Transformer: Autoregressive Generation with Increased Parallelism

Abstract

Transformers are slow at generating text. Each token must be *fully* generated *before* work starts on the next token. Diffusion models break this serial decoding bottleneck, but they are non-autoregressive latent-variable models. We propose **PipeT**, a *pipelined* architecture that remains autoregressive and simple to train. Tokens are decoded in parallel while attending to one another bidirectionally, as in diffusion, but token is slightly delayed with respect to token , so it can depend autoregressively on token and all previous tokens. This architecture decodes faster than a standard Transformer, reduces KV cache size, and is formally more expressive. Although pretraining and prompt prefilling are slower than in a standard Transformer, this tradeoff may be worth it when generation is a bottleneck, as when generating code or long reasoning traces during RL training or after deployment. We evaluate models with 4M–200M parameters trained on the SlimPajama-6B subset using Chinchilla-style token budgets. At 100M parameters, a 3-layer **PipeT** matches the test loss of a 4-layer standard Transformer of the same size while decoding faster with a smaller KV cache. It nearly matches a 16-layer standard Transformer (the optimal number of layers that we evaluated at this parameter budget), with only nats/token higher test loss while decoding faster with a smaller KV cache.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Autoregressive language modeling, Transformer architectures, pipelined decoding, efficient inference, KV cache efficiency

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_tbibOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

PipeT is an autoregressive architecture that pipelines computation across token positions to accelerate decoding and reduce KV-cache memory. Mixed-layer attention enables interactions across layers, extending computation before the input layer and after the output layer. Within-layer recurrence increases formal expressivity, enabling state tracking on tasks where fixed-depth Transformers fail.

Critical strengths

PipeT is a creative architecture motivated by inference efficiency. The reported inference gains are impressive and appear to increase with model size. The authors conduct extensive experimental sweeps to determine compute optimal scaling. Beyond efficiency, the circuit-complexity analysis and supporting state-tracking experiments provide a compelling connection between theoretical expressivity and empirical capabilities. The architecture opens several promising directions for future work.

Critical weaknesses

The training cost remains insufficiently characterized. The authors acknowledge slower training and prefill, informally reporting a 3-10× training slowdown (lines 427–428), but this tradeoff is essential to interpreting Figure 2 and assessing the feasibility of the architecture. The appendix’s token counts and (6ND) compute estimates do not capture the wall-clock implications of sequential dependencies across tokens rather than layers. Even without an optimized implementation, performance modeling or basic roofline estimates would help quantify the overhead.

Held-out loss alone provides an incomplete evaluation of generation quality. The paper draws inspiration from diffusion models, where aggressively parallel sampling from predicted marginals can lead to poor, incoherent samples. In a similar way, PipeT aims for greater parallelism when sampling autoregressively. In addition to the current likelihood based evaluation, a more thorough investigation of the samples generated from PipeT is warranted. A direct comparison of sampled text against standard Transformer++, measuring generative perplexity and entropy, would strengthen the evaluation.

Questions for authors
  1. What are PipeT’s training and prefill costs relative to Transformer++, both empirically and under a reasonably optimized implementation or a theoretical performance model?

  2. How does inference performance vary with batch size, and how does its interactivity-throughput curve compare with Transformer++ when sweeping batch sizes?

  3. How competitive is PipeT against stronger inference baselines? Grouped-query attention, latent attention, and sliding-window attention offer alternative ways to reduce KV-cache pressure.

6 comments
AuthorsOpenReview ↗

cost analysis

informally reporting a 3-10× training slowdown

What are PipeT’s training and prefill costs relative to Transformer++, both empirically and under a reasonably optimized implementation or a theoretical performance model?

How does inference performance vary with batch size, and how does its interactivity-throughput curve compare with Transformer++ when sweeping batch sizes?

Fair questions. We have answered them in detail in this post and in the newly added Appendix D.

As a highlight, §D.3 uses a timing model fitted to the measured FP32 timings to extrapolate end-to-end runtime and throughput on real-world benchmark traces. Table 4 estimates roughly 2–7× improvements in output throughput, including prefill and decoding time, on seven of eight benchmarks; long-context retrieval (MRCR) is the exception, with PIPET reaching 0.53× Transformer++’s speed at batch 128 (§D.3.3, lines 2029–2037).

AuthorsOpenReview ↗

complementary methods for KV cache reduction

How competitive is PipeT against stronger inference baselines? Grouped-query attention, latent attention, and sliding-window attention offer alternative ways to reduce KV-cache pressure.

Recall that our primary goal in this paper -- and its primary novelty -- was to widen the serialization bottleneck of decoding. PIPET accomplishes this, with negligible increase in loss. The alternatives you mention don't solve our problem, so we didn’t compare to their loss increases.

The fact that PIPET reduces KV cache size is just a nice side benefit, which appears “naturally” as a consequence of our architecture design. To review:

  • Pipelining means that different tokens are working on different layers at the same time. We always attend to each token’s highest available layer. That allows lower layers to be discarded.
  • In our 100M comparison, PIPET maintains comparable performance to Transformer++ while using only about 16% of Transformer++’s KV-cache memory (a 6.27× reduction; see Table 11).
  • This shows that we do not need to represent different types of information in different layers (cf. ELMo). With increased dimensionality and end-to-end training, higher layers can simply replace all lower ones, retaining their information through residual connections and also adding new information.

You seem particularly interested in this ability. When reducing KV cache size is important, e.g. for long context windows, PIPET could be combined with the methods you name. We are excited to explore such combinations in future work (§E, lines 2087–2103).

In particular, PIPET’s cache reduction approach seems to be orthogonal to the other cache reduction methods you name. The total KV cache size in a standard Transformer is O(Lnd)O(Lnd):

  • PIPET reduces the LL factor (# of layers) [as does YOCO - see below]
  • Grouped-query and sliding-window attention reduce the nn factor (# of tokens)
  • Latent attention reduces the dd factor (# of hidden dimensions)

Reviewer 37LU mentioned YOCO. We discussed YOCO in the original submission and have expanded that discussion in §E (“KV-cache reduction,” lines 2096–2103) to clarify its relationship to PIPET. Both YOCO and PIPET reuse a shared KV cache for the full history. YOCO builds this cache halfway through the network for reuse by the upper half, called the cross-decoder, whereas PIPET retains the terminal layer’s keys and values. YOCO can skip the cross-decoder during prefill; the paper reports roughly 30× lower prefill latency at 512K tokens.

Our expanded discussion also reviews additional methods that selectively retain cached tokens: H₂O reports comparable accuracy using 20% of the cache (5× smaller); PyramidKV reports comparable LongBench performance using 12% (about 8.3× smaller); and HeadKV-R2 retains about 1.5% (about 67× smaller) while preserving 97% of mean F1 on six LongBench question-answering tasks with Llama-3-8B-Instruct.

AuthorsOpenReview ↗

evaluation of generation quality

Held-out loss alone provides an incomplete evaluation of generation quality…A direct comparison of sampled text against standard Transformer++, measuring generative perplexity and entropy, would strengthen the evaluation

We agree that evaluation of generated text would complement our log-loss of observed text. We already acknowledged in §5 (lines 498–503) that teacher-forced evaluation does not capture how errors accumulate during free generation. We have expanded that discussion somewhat in the new version of §5.

However, we had imagined doing task-based evaluation. The trouble is that generation quality of all models is poor at 200M parameters. None of them can solve real tasks, so such a comparison at our current scale may be uninformative.

We see three possible ways to compare at this scale:

  1. We could mix the next-token probabilities of either PIPET or Transformer++ with those of the same larger model. Varying the weight given to each smaller model would let us compare its effect on generation quality, while the larger model helps maintain coherent output.
  2. We could evaluate PIPET as a draft model for speculative decoding, measuring how often a larger target model accepts its proposed tokens.
  3. We could abandon task-based evaluation and simply evaluate whether each model’s freely generated text goes off the rails, making it unlike real text. For example, we could evaluate its log-probability under a fixed open-weight 8B model.

Is 3. what you were suggesting when you wrote “generative perplexity and entropy”? We do not have any of these comparisons yet, but agree that we should add one to the paper.

Reviewer_37LUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PIPET, a pipelined Transformer architecture designed to accelerate autoregressive decoding by overlapping computation across token positions. Each position has a residual stream initialized with a shared learned vector. Its token embedding is injected at a configurable layer L_in, and the next-token prediction is produced at layer L_out. When L_in > 0, the stream can perform context-dependent computation before receiving its token; when L_out < L, it continues refining its representation after predicting the next token.

Every attention update reads the latest available representation of each existing stream. These representations come from different depths: earlier positions generally provide more advanced states, with sufficiently old positions supplying their finalized layer-L states, while later positions provide less advanced states. When predicting the next token, later-position streams have not yet received their token embeddings, preserving autoregressive conditioning. After prediction, a stream can incorporate subsequently generated tokens during its remaining updates.

This schedule allows multiple streams at different depths to advance in parallel and share a KV history, retaining only one terminal KV pair per finalized position.

The paper evaluates trained models ranging from 4M to 200M parameters using held-out loss. It compares against transformer++ and shows faster decoding for an equivalent loss.

Critical strengths

The architecture is novel to the best of my knowledge, and is clearly presented. The paper explains how the architecture was optimized / selected, and mentions its advantages as well as its downsides.

Critical weaknesses

The main issue I have with this paper is the limited number of comparisons. This architecture has multiple potential comparison points:

  • parallel decoding: speculative decoding methods
  • state-tracking abilities: recurrent architectures (e.g. GDN2 or Mamba3) - for fairness, it may be good to compare hybrids that contain at least one full-attention layer (similar to the "finalized" latents pipeT can access throughout decoding)
  • smaller KV cache: full-attention / sliding window attention hybrids, you only cache once (YOCO)

While I understand that time and compute may be limited, presenting such an architecture without any comparison to such techniques does not allow the reader to appreciate what differentiates it from other architectures.

Questions for authors
  • why is PipeT using different (a) initialization strategies (as stated in E.1) (b) FFN activation and (c) normalization method (table 2) compared to the transformer++ baseline? An ablation of these choices would be helpful to understand their impact on model quality and/or training stability
5 comments
AuthorsOpenReview ↗

paper length

The main issue I have with this paper is the limited number of comparisons.

We’d gently push back on the word “limited.” This is already a 42-page paper (plus 7 pages of references and disclosures), submitted to a conference with a 9-page limit.

(Historically the purpose of CS conferences was to disseminate interesting ideas rapidly. The purpose of the page limit was to control the size of a publishable contribution, preventing an arms race and keeping papers manageable for authors, reviewers, and readers alike. Appendices were eventually allowed, but were originally intended only for proofs and replicability details.)

This architecture has multiple potential comparison points

That said, thanks for these thoughtful suggestions. They are well taken and we do like to be thorough. We can promise to at least discuss them in the final version, with the experiments below and possibly additional experiments. Please see the responses below.

AuthorsOpenReview ↗

state tracking

state-tracking abilities: recurrent architectures (e.g. GDN2 or Mamba3)

Sure. We now additionally compare PIPET to RNNs & Mamba on permutation state tracking. (We further show that RNNs & Mamba do poorly on other tasks such as Associative Recall, which benefits from softmax attention over a long context.). Please find discussion in this comment and in §§3.3-3.4 of the revised PDF.

for fairness, it may be good to compare hybrids that contain at least one full-attention layer (similar to the "finalized" latents pipeT can access throughout decoding)

Yes. We noted that PIPET has the abilities of both RNNs (state-tracking) and Transformers (full attention). So it is appropriate to compare to a hybrid that was explicitly designed to have both abilities.

We will add such a comparison to the final paper. We now have pilot results on a hybrid with 2 GRU layers and 1 full-attention Transformer layer. If added to Fig. 3, it would outperform all but RNNs on state-tracking (i.e., even better length generalization than PIPET), while achieving the same excellent performance of the non-RNN models (including PIPET) on associative recall.

Of course, this hybrid does not enjoy PIPET’s fast pipelined decoding. Further experiments will be needed to see if PIPET can be tweaked to catch up with the hybrid while retaining its fast decoding. We speculate that the RNN-only layers of the hybrid give it an inductive bias toward state tracking, which might be induced in PIPET through architectural adjustments or regularization.

AuthorsOpenReview ↗

speculative decoding

It’s correct that speculative decoding is another way to accelerate decoding and increase arithmetic intensity.

To be explicit, it uses prefill-like parallelism to verify the results of a separate model that has already predicted future words. In “easy” parts of the sentence, this can accelerate decoding to match the fast model. (In “hard” parts, it simply wastes computation. Note also that the fast model consumes some of the parameter budget, compute budget, and training budget.)

We opted to defer this comparison to a future paper on adaptive decoding. The reason is that we have already been discussing PIPET variants that similarly use adaptive computation, slowing down in “hard” parts of the sentence by inserting <pause> tokens and/or dynamically changing ℓin\ell_{in} and ℓout\ell_{out}. Speculative decoding can figure into the adaptive computation, so we think it will be more naturally discussed in the context of that future paper.

Reviewer_bZgbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PipeT, an autoregressive architecture that overlaps computation across token positions by moving token injection and next-token prediction to intermediate layers. Each stream can begin computation before receiving its token and continue refining its representation after predicting the next token. Mixed-layer attention reads the latest available state of each stream, while finalized streams contribute a single shared KV pair. Experiments with 4M–200M parameter models examine language-modeling quality, decoding latency, and KV-cache memory, alongside synthetic state-tracking tasks and architecture sweeps.

Critical strengths
  1. The architecture provides an interesting way to overlap computation while preserving autoregressive modeling. Separating token injection, prediction, and subsequent refinement allows successive streams to advance concurrently without speculative token guesses or a verification stage. The resulting model retains explicit next-token probabilities and ordinary teacher-forced likelihood training.

  2. The experiments explore meaningful quality–efficiency tradeoffs rather than relying on a single Transformer baseline. The authors sweep Transformer depth and width under matched parameter and training-token budgets and report both decoding latency and KV-cache usage. The partially stalled configurations are particularly encouraging: at 30M parameters, several configurations improve loss, latency, and cache usage simultaneously over the evaluated six-layer Transformer.

  3. The analysis clearly identifies the architectural benefits and their main costs. The paper distinguishes reduced sequential depth from total computational work and provides explicit accounting for retained KV histories and active-stream states. The appendices document the timing protocol and architecture search, while the discussion acknowledges reduced training and prefill parallelism and the limited evaluation scale.

Critical weaknesses
  1. The fixed-batch benchmark does not establish efficiency gains across serving loads. PipeT reduces the sequential dependency depth between consecutive outputs without substantially reducing the main computational workload. If part of the speedup comes from using otherwise idle compute resources, that benefit may diminish as larger batches improve the hardware utilization of the standard Transformer. The bandwidth benefit from shared KV states may persist. However, measurements at only batch size 8 in FP32 do not establish the latency–throughput tradeoff across different serving loads.

  2. Cross-position and cross-layer state dependencies may introduce distributed execution costs that the single-GPU experiments do not capture. Queries at different layers require access to shared historical KV states and the appropriate intermediate states of active streams. Distributing the model across devices may therefore require KV replication, state transfers, and synchronization. In MoE models, uneven expert loads and communication times may further delay computations that depend on those states. The speedups demonstrated on single-GPU dense models do not establish whether these benefits persist in multi-GPU or MoE deployments.

  3. The decoding gains come at the cost of reduced training and prefill parallelism, but an end-to-end cost evaluation is missing. PipeT’s recurrent state dependencies limit parallel computation across positions even when the entire sequence is known, so teacher forcing alone cannot eliminate this cost. The authors informally report a 3–10× training slowdown but do not systematically measure training and prefill costs. For the long-generation and RL workloads emphasized in the paper, decoding speed alone does not establish whether the savings outweigh additional costs in input processing, sequence evaluation, and parameter updates.

  4. The manuscript does not provide an implementation repository, limiting independent verification of the efficiency results. Although the appendix documents the architecture and measurement protocol, reproducing the runtime gains requires implementing the mixed-layer attention, stream-state updates, shared KV cache, and compiled decoding path. Without the implementation and benchmark scripts, it is difficult to verify that the reported speedups are reproducible and to assess how implementation choices affect the comparison.

Questions for authors
  1. Can the authors sweep batch sizes under a common BF16 setting and report both per-token latency and aggregate output tokens per second?

  2. Can the authors provide a representative multi-GPU experiment to assess whether communication and synchronization substantially reduce the pipelining gains?

  3. Can the authors report training throughput and end-to-end inference time including prefill, to identify workloads where PipeT provides an overall benefit?

3 comments
AuthorsOpenReview ↗

Animated diagram!

Presentation: 2: Generally understandable, but requires substantial effort in places.

The method is easier to explain with an animation of pipelined execution vs. standard execution.

We have placed an interactive visualization for you at https://anonymous.4open.science/w/pipet-website-8677/ (hover over a node to see its attention connections), along with a review of the main points. We’d link to the animation from the camera-ready version.

AuthorsOpenReview ↗

Empirical speedups and slowdowns at training, prefill, and decoding

Thanks for acknowledging the key advantages of our architecture and the work we did to explain and investigate them. With tens of trillions of output tokens already being generated daily, we are excited about the potential for real-world speedups and cost savings, if our results scale up to larger models.

Beyond the benefits you mentioned, our architecture also has more theoretical power than a standard Transformer (or state-space model), and we validated this with state tracking experiments (Fig. 3, §3.3; §B.3, lines 1320–1339).

The fixed-batch benchmark does not establish efficiency gains across serving loads.

If part of the speedup comes from using otherwise idle compute resources, that benefit may diminish as larger batches improve the hardware utilization of the standard Transformer.

The decoding gains come at the cost of reduced training and prefill parallelism, but an end-to-end cost evaluation is missing.

Can the authors report training throughput and end-to-end inference time including prefill, to identify workloads where PipeT provides an overall benefit?

Can the authors sweep batch sizes under a common BF16 setting and report both per-token latency and aggregate output tokens per second?

Regarding the set of questions above, we have answered them in detail here: Empirical speedups and slowdowns at training, prefill, and decoding.

While the above is likely to answer your concern, it was done in FP32. Your mention of BF16 made us realize that we had not studied how precision affects the comparison. We therefore extended the training, prefill, and decoding sweeps in §D.2 (lines 1854–1861) to TF32 and BF16, using the same 100M models, H100 NVL, and measurement procedure. The table below summarizes the results at n = 512, where B denotes batch size.

PrecisionPIPET’s training slowdownB=64B=64PIPET’s prefill slowdownB=1B=1B=8B=8B=128B=128PIPET’s decoding speedupB=1B=1B=8B=8B=128B=128
FP322.57×17.63×6.46×1.80×6.63×5.14×3.51×
TF324.97×21.64×13.92×3.22×6.63×5.55×3.55×
BF167.81×22.56×16.28×3.08×6.13×4.40×1.69×

Lower precision speeds up both models, but it benefits Transformer++ more on all tasks. In BF16, PIPET retains a 6.13× decoding speedup at batch 1, while its speedup at batch 128 falls to 1.69× and its training slowdown at batch 64 rises to 7.81×. Following the distinction in Appendix D, lower precision lets the GPU perform more arithmetic per second, easing the compute-capacity bottleneck and potentially making PIPET’s sequential dependencies across tokens more limiting during training and prefill.

Can PIPET catch up? Our profiling suggests that PIPET spends more time launching many small GPU operations, whereas Transformer++ spends more time in large matrix multiplications. Faster arithmetic reduces the cost of those multiplications but does little to reduce the overhead of launching each operation. We therefore suspect that the lack of specialized CUDA kernels in our current implementation limits PIPET’s gains from lower precision.

Thus, further engineering PIPET with customized CUDA kernels may improve our reported decoding speedups (§5, lines 476–477; see also §D.2). Beyond this, we are also thinking about other approaches that may also help speed up training and prefilling: an encoder–decoder variant could use a separate encoder to process user messages in parallel while PIPET generates assistant messages (§5). Training could potentially be accelerated by initializing from a pretrained model (warm-starting; §5 and §C.5) or learning from a teacher model (distillation; §5). Parallel methods for recurrent computation, such as ParaRNN, are another direction to investigate for training and prefill. Some of these directions were already mentioned in the original draft; we have expanded their discussion in §5 (lines 478–485).

AuthorsOpenReview ↗

Other responses

The manuscript does not provide an implementation repository

Here is a link to a snapshot of the repo: https://anonymous.4open.science/r/pipet-release-1CB6. We plan to release all code to GitHub together with the camera-ready version of the paper. If we speed up PIPET further via custom CUDA kernels, we will push the speedup to GitHub as well.

Can the authors provide a representative multi-GPU experiment to assess whether communication and synchronization substantially reduce the pipelining gains?

Thanks for this insight. We have also thought about this, but multi-GPU experiments are out of scope for this initial paper. We now discuss communication, synchronization, and uneven expert workloads as directions for future work in §5 (lines 495–497).

We hope this answers your concerns. Thanks for acknowledging the novelty. We hope you agree that these positive results are sound, thorough, and worth sharing with the community.

Reviewer_5aeeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PipeT moves token injection and next-token readout into intermediate layers, allowing token streams to overlap across depth. Mixed-layer attention reads snapshots of the latest representations, and fully pipelined decoding shares a terminal KV history. The architecture preserves autoregressive factorization and next-token likelihood training, at the cost of reduced training and prefill parallelism. Experiments on 4M–200M models trained on a SlimPajama subset measure held-out loss, decoding time, and KV-buffer size, alongside placement, stalling, and state-tracking studies.

Critical strengths
  1. The architecture provides an explicit autoregressive computation schedule that overlaps token streams without guessing future tokens or requiring speculative verification. Figure 1 and Algorithm 1 make the mechanism clear.

  2. The paper reports measured quality–efficiency tradeoffs under matched parameter and training-token budgets. At 100M, PipeT decodes 2.17× faster than the six-layer Transformer, with a 0.0164-nat/token loss increase. Depth–width, placement, and stalling studies provide useful supporting evidence.

Critical weaknesses
  1. Evaluation is limited to 4M–200M models with a 1,000-token vocabulary and length-512 sequences. Timing excludes prompt prefill and sampling. These results leave end-to-end gains for the proposed code/reasoning workloads untested, particularly given the slower prefill.

  2. Section 3.2 states that PipeT achieves no higher test loss than the six-layer Transformers, but Table 8 shows higher losses by 0.0164 and 0.0185 at 100M and 200M, respectively. This sentence needs correction; the reported speedup ratios remain consistent with the table.

Questions for authors

PipeT shares recurrent computation with Feedback Transformer and modern RNN/SSM approaches such as Mamba. What additional quality–efficiency value does cross-token pipelining provide, based on the existing results or analysis?

3 comments
AuthorsOpenReview ↗

Empirical speedups and slowdowns at training, prefill, and decoding

These results leave end-to-end gains for the proposed code/reasoning workloads untested, particularly given the slower prefill.

You and reviewers bZgb and tbib are right that the tradeoff is more nuanced than our original discussion suggested. We have expanded Appendix D with:

  • a discussion of how serial dependencies, data movement, and GPU compute limits affect speed (§D intro)

  • a new experimental setup (§D.1)

  • throughput measurements across batch sizes and sequence lengths (§D.2)

  • runtime estimates at operating points that match real-world benchmark traces (§D.3)

Specifically, in §D.2’s heatmap (Figure 13), we report both models’ throughput and their ratio over batch sizes 1–128 and token lengths 64–512 under each phase (training, prefill, and decoding). These measurements compare the lowest-loss 100M model in each family on one H100 NVL in FP32.

You can read §D for details, but here’s a summary of the findings:

  • On training and prefill, larger batches substantially reduce Transformer++’s advantage. At 512 tokens, PIPET’s training slowdown falls from about 23× at batch 1 to 2.1× at batch 128, while its prefill slowdown falls from about 18× to 1.8×. The upper-right corners of the training and prefill ratio heatmaps show these reduced penalties at large batches. Both large-batch slowdowns are far below the 32× slowdown from the idealized analysis, which assumes unlimited GPU parallelism. The finite compute and memory bandwidth of a real GPU help explain why the gap is smaller in practice (§D.2, lines 1828–1830 and 1836–1842).

  • On decoding, PIPET is indeed faster: it has lower latency and better throughput at all batch sizes and sequence lengths. Its throughput is 6.63× better at batch size 1 and 3.51× better at batch 128. (Larger batches give Transformer++ more independent sequences to process in parallel, narrowing PIPET’s relative gain while increasing total tokens per second.) At batch size 128, our 3.51× speedup comes in exchange for a 1.8× prefill slowdown, often a worthwhile tradeoff at inference time, and a 2.1× training slowdown (§D.2, lines 1841–1853).

  • Using a model to extrapolate to realistic workloads (benchmarks), we observe 2× to 7× improvements in end-to-end throughput (prefill+decoding). Most workloads are dominated by decoding time. Applying the measured phase timings to the input and output message lengths in benchmark traces gives estimated end-to-end speedups of roughly 2–7× on seven of eight benchmarks, including coding, multi-turn chat, and reasoning (Table 4, §D.3). PIPET is slower on long-context retrieval tasks that are dominated by prefill time: in MRCR we are only 0.53× the speed of Transformer++ at batch size 128, because its median example has 67,246 input tokens vs. 417 output tokens (§D.3.3, lines 2029–2037).

Note: Our implementation does not yet use CUDA kernels optimized for PIPET, so further engineering may improve the reported speedups (§5, lines 476–477).

AuthorsOpenReview ↗

Other responses

This sentence needs correction; the reported speedup ratios remain consistent with the table.

Thanks for spotting this! We have revised the wording in §3.2 (lines 357–360) to make this precise; apologies for rounding away the differences.

PipeT shares recurrent computation with Feedback Transformer and modern RNN/SSM approaches such as Mamba. What additional quality–efficiency value does cross-token pipelining provide, based on the existing results or analysis?

  • Our main goal was to achieve increased decoding speed with higher GPU utilization [§2.4, lines 259–265; Appendix D]. The models you mention remain slow!

  • The models you mention also don't have our smaller KV cache [§2.4, lines 267–271].

  • Our increased expressivity is a nice additional benefit [§2.4, lines 291–299]. Feedback Transformer (Fan et al., 2021) was motivated by increased expressive power, and we are in the same class because we have both attention and recurrent state [§2.4; Appendix E].

Our expressive power strictly beats that of bounded-depth circuits such as Mamba and Transformer++ [§2.4; Appendix B], which can’t solve (for example) the S5S_5 word task even in theory [§B.2, lines 1277–1283; §3.3; §B.3], let alone in practice [Fig. 3]. Since you asked about Mamba, we extended Fig. 3 to also test Mamba on the S5S_5 word task. See our discussion in the response below or paper sections §3.3 and §B.3 (lines 1327–1339) for the full details.

We hope these answers address the lingering concerns. Thanks for recognizing our contribution and clear exposition. We hope you can consider raising the rating! We have spent almost a full year on this project and are very excited to share these positive results with the community. Despite the expected training and prefill slowdown (which might also partly reflect our implementation not using custom CUDA kernels like the FlashAttention used by Transformer++), we think the several-fold decoding gains make this a promising direction to investigate at larger scales, where they could have huge commercial benefits.

AuthorsOpenReview ↗

Expressivity of PIPET

In response to questions from reviewers 37LU and 5aee, we expand our expressivity evaluations to include results from DenseGRU and Mamba. Qualitatively the results are the same as in the original submission, with DenseGRU acting like RNN and Mamba acting like Transformer++. Of all models tested, only PIPET does well on both our tasks (as would non-pipelined architectures that likewise combine recurrence and attention).

We have also added length generalization results: we now evaluate on S3S_{3} and S5S_{5} permutation composition [1] at up to 10×10\times (S3S_{3}) and 5×5\times (S5S_{5}) the training length.

We have also added an evaluation on Multi-Query Associative Recall (MQAR) [2], a task that recurrent models can only solve if their state grows with the number of key-value pairs to be stored [3].

All models (Transformer++, PIPET, Mamba [4] and a DenseGRU RNN [5]) are parameter-matched (about 50k parameters, 3 layers) and trained with the same recipe. For each model and task we select the learning rate on validation accuracy and report the maximum over 5 seeds, in keeping with prior work on the algorithmic expressivity of neural networks [6]. Please refer to Figure 3 and §§3.3–3.4 to find these updated results.

PIPET continues to outperform all Transformers on the state-tracking tasks (permutation composition). Mamba and Transformer++’s state-tracking (S5S_{5}) degrades to chance within 10 permutations, well inside the training regime (100 permutations). This is consistent with theory: these models lie in TC0\mathrm{TC}^0 [7][8], while the S5S_{5} word problem is NC1\mathrm{NC}^1-complete [9], so they cannot solve it at arbitrary lengths (unless TC0\mathrm{TC}^0 = NC1\mathrm{NC}^1). Mamba's parity tracking (the S2S_{2} subtask within S5S_{5}) maintains high accuracy up to roughly half the training regime (60 permutations) before falling to chance near the training length. In contrast, PIPET learns both parity and full S5S_{5} state tracking perfectly within the training regime, and demonstrates length generalization beyond the training sequence length (to about 170 permutations) before degrading to chance. On S3S_{3}, PIPET tracks the full state perfectly up to about 512 permutations (5×5\times training length). This ordering holds for every seed, not only the maximum: each of PIPET's 5 seeds stays above 99% accuracy past the training length on both tasks, while no Mamba or Transformer++ seed does.

The DenseGRU RNN does especially well on S3S_{3} and S5S_{5}. Training appears to recover the S5S_{5} finite-state automaton exactly: the trained model perfectly solves both tasks at lengths far exceeding the training regime (up to 1024 for S3S_{3} and 512 for S5S_{5}). PIPET might achieve this with more training, but the RNN architecture presumably has an inductive bias toward state-tracking.

PIPET matches all Transformers and strongly outperforms all RNNs on the recall task (MQAR). Multi-Query Associative Recall tests the memory storage capacity of models. This task involves using a sequence of key-value pairs that is provided in-context to answer a set of queries during evaluation.

Self-attention in Transformer++ enables this dynamic lookup to occur easily; transformers perfectly solve this task within our evaluation regime (up to k=64k=64 key-value pairs at sequence length 256). Recurrent models, including state-space models like Mamba, theoretically cannot solve this task at arbitrary dictionary sizes kk without growing their recurrent state: any recurrent model needs Ω(N)\Omega(N) bits of state to solve MQAR on inputs of length NN [3]. In practice, we do not hit this limit for Mamba at the synthetic scale we evaluate (k=64k=64). In contrast, the DenseGRU RNN, which must encode the entire dictionary into a much smaller vector state, performs poorly at this task. Even at small k=8k=8, the RNN falls to chance (1/k1/k).

In summary, these two tasks separate the models along the two axes we claim. On state tracking, Transformer++ and Mamba fail to fit S5S_{5} even at the training length, consistent with their known expressivity limits, while PIPET fits it in every seed and generalizes in length. On recall, PIPET matches attention, whereas the parameter-matched RNN, which tracks state at every length we test, fails from k=8k=8 on. We have updated Figure 3 and §§3.3–3.4 to reflect these new results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.