acceptodds
Under review as a conference paper at ICLR 2027

How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference

14 views

Abstract

Looping in transformer models reuses weights by repeatedly applying the same transformer, or parts of it, to token representations, effectively making the model deeper without increasing its parameter count. Whether looping is useful depends on which constraint dominates: model size, training compute, or autoregressive inference efficiency. We study these trade-offs jointly in dense decoder-only transformers trained from scratch, varying which parts of the model are looped, how many loops are applied, and how information is passed between loops. Our experiments span models with up to 1B parameters, trained on up to 155B tokens using up to training FLOPs, with up to loops. We find that looping the full transformer generally gives the strongest quality at a fixed parameter budget, whereas looping a middle block of layers gives a better return when training FLOPs are constrained. These trade-offs also depend on the number of loops: adding loops is not uniformly beneficial, and gains eventually saturate, reverse, or become unstable. This saturation motivates asking whether forcing every loop to use exactly the same weights limits the benefit of deeper looping. We address this by adding small loop-specific low-rank updates that improve quality, with larger gains as more loops are added. A separate practical concern for looped transformers is the sequential dependency during autoregressive generation, which causes a slowdown proportional to the number of loops. Parallel Loop Transformers address this limitation by passing information between loops across adjacent token positions via hidden states, allowing loop stages to run concurrently during decoding. However, in our experiments the original formulation does not consistently retain the quality gains of ordinary looping. We find that preserving the current-token embedding and separately normalizing it and the previous-token hidden state before combining them recovers strong performance across model scales and training budgets. Together, these results show that effective looping requires matching the design to the resource constraint and inference setting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Transformers, Parameter Efficiency, Loop Specialization, Parallel Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xtRjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper presents an empirical study of looped Transformers, examining how different looping strategies affect model performance under various parameter and training compute budgets. The authors systematically compare existing looping approaches, including full stack looping, middle stack looping, and layer wise looping, and investigate how model performance changes with the number of loops. They also explore loop specific low rank adapters to allow different loops to learn specialized transformations. Additionally, the paper studies Parallel Loop Transformers (PLTs) and proposes a modification that separately normalizes and combines the current token embedding and previous token hidden state to improve model performance while maintaining parallelizable inference. The experiments evaluate these approaches across different model sizes, training budgets, and language modeling benchmarks.

Critical strengths

The paper systematically investigates multiple looping strategies across different model sizes, training budgets, and loop counts. The experiments cover models ranging from 70M to 1B parameters and up to 32 loops.

The paper investigates different loop counts, looping locations, specialization methods, normalization strategies, and gating mechanisms.

Critical weaknesses

The paper provides an extensive empirical investigation of looped Transformers, but the methodological novelty appears limited. Several central components, including recurrent weight sharing, loop specific low rank adapters, and cross loop parallelism, have already been introduced in prior work.

The main technical modification to PLT consists of separately normalizing the current token embedding and recurrent hidden state, followed by gated mixing. While the empirical improvements are encouraging, these operations are well established, and the paper does not sufficiently establish why their combination constitutes a substantial methodological advance over existing recurrent architectures.

The systematic comparisons across looping strategies, model sizes, and resource budgets are potentially valuable. However, the authors should clarify which findings are genuinely new relative to existing scaling studies. Furthermore, the proposed PLT modifications should be evaluated using actual inference latency, throughput, and memory measurements to substantiate their practical efficiency claims.

Questions for authors

Could you provide actual inference latency, throughput, and memory measurements for the proposed modified PLT more specifically?

The proposed normalized PLT maintains separate KV caches for different loops, whereas the original PLT incorporates KV sharing. How does this design choice affect memory efficiency, particularly for long context lengths and larger loop counts?

Most evaluations focus on language modeling BPB and likelihood based downstream accuracy. Could the authors evaluate whether the proposed modifications improve actual autoregressive generation, particularly on mathematical reasoning, code generation, and longer generation tasks?

6 comments
AuthorsOpenReview ↗

Response to W1 - 1

We thank Reviewer xtRj for their careful review and constructive feedback, and for recognizing the breadth of our empirical investigation. Below we try to address some of the weaknesses and questions.

W1. The paper provides an extensive empirical investigation ... substantiate their practical efficiency claims. '

We agree that weight sharing, loop-specific adapters, and cross-loop parallelism are established mechanisms, and we acknowledge these precedents in Section 4 and Appendix A. Our contribution is the controlled empirical evidence about which looping designs to use, when specialization helps, and how parallel looping can remain useful beyond two loops. Establishing these findings required substantial experimental work across seven looping strategies, multiple loop depths, and different resource constraints. We compare all seven strategies at 70M–500M, with targeted experiments extending to 1B and training budgets up to 155B tokens. This breadth allows us to distinguish improvements in parameter efficiency from improvements in training-compute efficiency within a common model family. Our approach shares the motivation of Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2020) and Do Transformer Modifications Transfer Across Implementations and Applications? (Narang et al., 2021): systematic comparisons can establish which design choices work and under what conditions. We would like to note that the distinctions between our study and prior work are outlined in Section 4 and discussed in more detail in Appendix A. We summarize them again below to clarify how our experiments and findings differ from prior work.

Looping schedules and scaling. We compare alternative looping schedules within a common model family and training setup to identify architectural effects that prior scaling studies do not directly resolve. For example, our comparison of Full-Stack and Per-Layer Looping keeps the parameters and number of layer applications fixed while changing their execution order.

This complements Parcae: Scaling Laws for Stable Looped Language Models (Prairie et al., 2026), which studies the allocation of compute between recurrence and training data, and How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models (Schwethelm et al., 2026), which compares shared recurrence with independently parameterized depth. Hyperloop Transformers (Zeitoun et al., 2026) instead develops middle-stack recurrence with hyper-connections. These recently accepted COLM 2026 and NeurIPS 2026 papers are already discussed in Section 4 and Appendix A, and we view them as potentially concurrent work. Our experiments show why the architectural choices matter: Full-Stack Looping generally gives the strongest quality at fixed parameter and token budgets, while Middle-Stack Looping often offers a better return when training FLOPs are constrained. Thus, our comparisons establish how both the looping schedule and the resource constraint affect which design is preferable.

Loop-specific specialization. Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (Bae et al., 2025) uses depth-specific adapters to recover performance when converting a pretrained Transformer into a compact recursive model. Our work is different as we train the shared weights and adapters together from initialization and study how their benefit changes with loop depth. We find small gains at two loops and clearer gains at four and eight, with relatively few additional parameters (Section 3.3, Figure 5, Appendix F). We also compare against added physical depth, which can achieve better BPB at comparable training FLOPs when additional parameter storage is available. This establishes when specialization helps over exact sharing while distinguishing its storage benefits from compute efficiency.

[Continue Below]

AuthorsOpenReview ↗

Response to W1 - 2

Parallel looping. Parallel Loop Transformer for Efficient Test-Time Computation Scaling (Wu et al., 2025) introduces cross-loop parallelism, while LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling (Yang et al., 2026) reports declining performance beyond two loops. We investigate this limitation through ablations of the cross-token shift, normalization, state mixing, current-token access, and attention/cache mechanisms. We identify a substantial magnitude mismatch between the current-token embedding and shifted recurrent state. Separately normalizing the inputs, controlling their mixture, and preserving the current-token path recovers much of the lost quality. Our normalized shifted models generally improve or remain stable from two to four and eight loops (Section 3.4, Figure 7, Appendix G.1). Although the settings differ from LoopCoder-v2, this shows that useful parallel refinement need not end at two loops. Our 1B experiments also retain strong performance at two and four loops, with a remaining gap to the unshifted control at four loops (Figure 8, Appendix H).

We would like to note that we report measured decoding latency, throughput, and KV-cache memory in our response to the next question(s).

We therefore believe this study makes an important contribution by establishing when looping helps, which design choices matter, and how those choices depend on the available parameters, training data, and compute.

AuthorsOpenReview ↗

Response to Q1.and Q2.

Q1.and Q2. Could you provide actual inference latency, throughput, and memory measurements for the proposed modified PLT more specifically? The proposed normalized PLT maintains separate KV caches for different loops, whereas the original PLT incorporates KV sharing. How does this design choice affect memory efficiency, particularly for long context lengths and larger loop counts?

We address both questions in our response to Q3 by Reviewer HCDe, and summarize the main results here. Since the original PLT work already demonstrated inference benefits, our work focused on understanding which looping designs work and addressing limitations such as the degradation beyond two loops reported in LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling (Yang et al., 2026). Our normalization and input-mixing changes keep parallel looping useful at four and eight loops in the evaluated settings. Below we have benchmarked our modified PLT (20.0B-parameter model) in BF16 on a single NVIDIA GB300 GPU, using a 16k-token prompt.

Latency and Throughput: For a single sequence, standard four-loop execution takes 29.75 ms per generated token. Modified PLT reduces this to 10.86 ms with separate caches and 8.69 ms with a shared cache, close to the no-looping baseline of 8.24 ms. For a batch of eight sequences, total throughput increases from 226 tokens/s with standard looping to 489 with separate-cache PLT and 772 with shared-cache PLT, compared with 824 without looping. The shared-cache implementation also fuses attention across loops. Together, these changes provide approximately 3.4-fold improvements over standard looping while approaching the decoding performance of the no-looping baseline.

Cache memory savings. Standard looping and separate-cache PLT keep one KV cache for each loop. With four loops, this means storing four caches. Our modified PLT can also share a single cache across all loops, bringing cache memory back to the no-looping level. For the 20.0B BF16 model with one sequence containing 64,000 cached tokens, the calculated cache size is 5.86 GiB without looping. Four separate caches require 23.44 GiB, while four loops sharing one cache require only 5.86 GiB. Sharing therefore saves 75% of cache memory with four loops, or 50% with two. Note that the model weights occupy the same memory in all configurations.

We would also like to note that the speedups reported above apply to generating tokens after the input prompt has been processed. Processing the prefill, still takes longer with more loops. For a single 16k-token prompt, this takes 0.46 seconds without looping and 1.91 seconds with four-loop shared-cache PLT. Our response to Q3 provides the full timing and memory results.

Reviewer_5LNROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper conducts a very wide empirical study of how to construct a looped transformer. The paper includes experiments on which layers to loop (all, middle, etc.), how to add adapters to looping (lora, matrix mixing, etc), how to stabilize looping, which representations to use when looping, and many other setups. The authors study when each config is more efficient, such as which looping setups are more compute efficient and which are more parameter efficient.

Critical strengths

To the best of my knowledge, there have not been many large scale empirical studies of the (many) architectural knobs in looped models, so this paper is clearly useful in that regard. The paper is also well written and easy to follow and groups the design choices in a way that makes sense.

Critical weaknesses

This paper covers a lot of different methods and seems more like a large scale experiment dump or method survey than a single paper with a clear contribution beyond running lots of experiments. There is not much theoretical analysis on why certain interventions are needed (e.g. the RMSNorm for stability in PLT is added only after empirically observing that the magnitudes of the two components differ significantly, but no concrete analysis is given as to why this happens in the first place), and there is not much new methodology. The paper also does not include scaling laws (I didn't read the whole appendix since it was over 100 pages long, so I may have missed this), which seem particularly relevant since the paper is about how efficient different looping methods are at scale.

Questions for authors

Not particularly

2 comments
AuthorsOpenReview ↗

Response to W1

We thank Reviewer 5LNR for their thoughtful assessment and for recognizing the value of a large-scale, systematic study of looped Transformers, as well as the paper’s clarity and organization. Below we try to address the weakness raised.

W1. This paper covers ... methods are at scale.

On the central contribution and novelty. Our contribution is to compare three closely related design choices, where to loop, how to specialize shared weights, and how to parallelize looping, within a common experimental setting. These choices affect both quality and efficiency, and the preferred configuration depends on which resource is constrained. The experiments lead to concrete conclusions. Full-Stack Looping generally gives the strongest quality at a fixed parameter budget, whereas Middle-Stack Looping provides a better return under training-FLOP constraints. Loop-specific adapters become more beneficial relative to exact weight sharing as loop depth increases, and separate normalization with controlled state mixing recovers much of the quality lost in parallel looping (Sections 3.2–3.4). These findings help distinguish parameter efficiency from compute efficiency and show when additional looping is useful. This is also the motivation behind systematic studies such as Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2020) and Do Transformer Modifications Transfer Across Implementations and Applications (Narang et al., 2021): evaluating design choices in a shared setting can establish which benefits persist under consistent comparisons. We will make this central contribution more explicit in the revised paper.

On the motivation for normalization. Appendix G.2 examines the scale imbalance between the current-token embedding and the shifted hidden state across training budgets, loop depths, model sizes, and cache designs. Figure 116 shows that, for 100M models with eight loops, hidden-state RMS is consistently much larger than embedding RMS, and that the size of this difference varies across training budgets and loops. Adding these inputs with equal coefficients therefore does not give them comparable contributions: the larger hidden state can dominate the current-token embedding. Figure 117 shows that the mismatch persists at both 100M and 200M and is not removed by KV sharing or G-SWA. The ablations in Figure 7 and Figures 114–115 complement these measurements: separately normalizing the inputs and controlling their mixture recovers much of the lost quality. Together, these results support scale imbalance as a contributor to the degradation of raw shifted PLT and explain the motivation for the intervention. Normalization is applied immediately before the inputs are combined; it does not require the Transformer’s output RMS to match the embedding RMS. The evidence is empirical, and a theoretical explanation of why the magnitude differences arise remains an open direction. We will make this distinction clearer while giving the existing measurements and ablations greater prominence.

On scaling laws. Figure 1 presents data- and compute-efficiency estimates following MAI-Thinking-1: Building a Hill-Climbing Machine (The Microsoft AI Team, 2026), as cited in our paper. Appendix D.1 describe the power-law fits relating BPB to training tokens and total training FLOPs. Appendix H examines data and compute scaling for PLT: how BPB changes as training tokens and total training FLOPs increase, for different model sizes and loop counts. This shows when spending additional compute on repeated processing is more effective than spending it on more training data, and how that trade-off changes with model scale. We will highlight these analyses more clearly in the main text and explain their connection to the efficiency results in Figure 1.

Thank you once again for your time and thoughtful feedback. We remain at your disposal should you have any further questions or require additional clarification.

Thank you for your response. I will keep my score.

Reviewer_HCDeOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies looped Transformers along three dimensions: looping strategy, loop-specific specialization, and parallelized looping for autoregressive inference.

The experiments show that the preferred looping strategy depends on the resource constraint, that small loop-specific adaptations can improve exact weight sharing, and that separately normalizing and mixing the inputs to PLT substantially improves its quality. Overall, the paper provides a useful empirical characterization of the trade-offs in looped Transformers.

Critical strengths

The paper provides a broad empirical study of looping across model scales, training budgets, loop counts, and looping strategies. In particular, the distinction between parameter efficiency and training-compute efficiency is useful and well supported.

The PLT analysis identifies a practically important failure mode of shifted looping and shows that normalization and controlled mixing substantially recover model quality across a range of settings.

Critical weaknesses

The specialization experiments do not fully isolate why loop-specific parameters help. Loop-specific LoRA introduces additional capacity, so improvement over exact sharing alone does not establish that loop-dependent specialization is the relevant factor. A capacity-matched shared-adapter baseline would make this conclusion substantially stronger.

The mechanistic interpretation of the PLT results is not fully isolated. Some normalization/mixing comparisons also alter the overall magnitude of the combined input, making it difficult to distinguish effects from relative weighting, residual scale, and optimization stability. Moreover, while the modified PLT permits parallel loop execution, the paper does not measure its actual decoding latency or throughput.

Questions for authors
  1. How does loop-specific LoRA compare with a shared adapter under matched parameter, token, and compute budgets?

  2. Can the PLT experiments separate the effects of normalization, relative mixing, and overall input magnitude?

  3. What wall-clock latency, throughput, and KV-memory improvements does the modified PLT provide in practice?

  4. How sensitive are the reported compute-efficiency gains to the scaling-law fitting procedure, and do direct matched-FLOP comparisons support the same conclusion?

Willing to raise the score if fully addressed.

6 comments
AuthorsOpenReview ↗

Response to W1 and Q1

We thank Reviewer HCDe for their thoughtful feedback and for recognizing the value of our empirical study. Below, we address the weaknesses and questions raised.

W1. and Q1. How does loop-specific LoRA compare with a shared adapter under matched parameter, token, and compute budgets?

We would like to note that our use of loop-specific LoRA follows Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA, which we cite in the paper. To answer the question posed by the reviewer, we compare four versions of the same 10 transformer block backbone with 502.03M total parameters (337.9M excluding input embeddings), where the fourth version is one where parameters are increased by increasing the width of the MLP.

The four version are defined in the following sentences. Exact sharing uses identical weights at every loop without adding parameters. Loop-specific LoRA gives each loop its own rank-8 adapter, adding 3.604M, 7.209M, and 14.418M parameters for two, four, and eight loops, respectively. Shared LoRA reuses one larger adapter at every loop, with rank 16, 32, or 64, exactly matching the corresponding loop-specific model’s parameter count. The wider MLP adds no adapters; it enlarges the MLP and reuses it at every loop, adding 3.686M, 7.373M, and 14.746M parameters, respectively. Rounding the MLP width to a multiple of 32 leaves its total parameter count within 0.02–0.06% of the LoRA models. All models are trained from scratch on 13.51B tokens with three seeds, using the same data order and training settings. Relative to loop-specific LoRA, the other looping experiments use approximately 0.5% more FLOPs at two loops and 1.7% more at four loops, keeping compute within 2%. At eight loops, shared LoRA and the wider MLP use 4.2% and 4.3% more FLOPs, respectively; this comparison has identical tokens and matched or approximately matched parameters, but a larger compute difference. Results are mean BPB ± standard deviation across three seeds; lower is better. Note that the datasets in the table below are cited in our paper.

DatasetLoopsExact sharingLoop-specific LoRAShared LoRAWider MLP
DCLM21.0158 ± 0.00101.0147 ± 0.00101.0155 ± 0.00061.0156 ± 0.0006
DCLM41.0082 ± 0.00031.0067 ± 0.00031.0077 ± 0.00051.0061 ± 0.0005
DCLM80.9998 ± 0.00070.9982 ± 0.00070.9991 ± 0.00120.9957 ± 0.0010
C421.0195 ± 0.00121.0185 ± 0.00091.0193 ± 0.00061.0196 ± 0.0005
C441.0131 ± 0.00031.0115 ± 0.00021.0128 ± 0.00061.0110 ± 0.0005
C481.0053 ± 0.00061.0035 ± 0.00051.0043 ± 0.00131.0012 ± 0.0009

Loop-specific LoRA, shared LoRA, and the wider MLP give similar BPB on DCLM and C4 and are competitive options in our experiments. Compared with exact sharing without adapters, the reduction in BPB from loop-specific LoRA grows as the number of loops increases, although the absolute differences remain small. At eight loops, the wider MLP has slightly lower BPB, while using 4.3% more training FLOPs than loop-specific LoRA. These results suggest several useful ways to spend the additional parameters: give each loop its own adapter, reuse a larger shared adapter, or widen the shared MLP.

AuthorsOpenReview ↗

Response to Q2

Q2. Can the PLT experiments separate the effects of normalization, relative mixing, and overall input magnitude?

We thank the reviewer for raising this distinction. Figures 7 and 118 provide complementary ablations of normalization, overall input magnitude, and adaptive mixing, including comparisons at 200M parameters and TPP 40. Each comparison addresses a different aspect of how the current-token embedding and shifted hidden state are combined.

First, Figure 7 compares raw addition with separately normalized addition. Both variants add the current-token embedding and the shifted hidden state, but the latter normalizes each input before combining them. This places the two inputs on comparable scales and recovers part of the quality lost when replacing standard Full-Stack Looping with raw shifted CLP. This experiment shows that separate normalization improves quality even when the addition rule is retained.

Second, Figure 7 compares normalized addition with fixed equal averaging. Both variants use the same separately normalized inputs and assign them equal relative weight. Averaging simply halves the magnitude of their sum. At lower loop counts, fixed averaging improves over normalized addition. This experiment shows that overall input magnitude matters: reducing the scale of the combination improves quality without changing the relative weighting of its inputs.

Third, Figure 118 compares fixed equal averaging with learned mixing, retaining separate normalization in every variant. A learned gate determines how much the next loop receives from the current-token embedding versus the shifted hidden state. The scalar gate predicts one input-dependent mixing weight shared across all hidden dimensions. The rank-8 linear gate compresses the concatenated inputs into eight intermediate features and maps these to a separate mixing weight for each hidden dimension. The rank-8 MLP gate adds a ReLU between these projections, allowing a more expressive dependence on the inputs. All learned gates use a sigmoid to constrain their weights between zero and one. Their improvements over fixed averaging are small and vary across settings. This experiment shows that input-dependent relative weighting offers only a modest additional benefit once the inputs are separately normalized and averaged.

Together, these results show that separate normalization and fixed averaging account for much of the observed recovery, while more expressive learned mixing provides comparatively small additional gains.

AuthorsOpenReview ↗

Response to Q3

Q3. What wall-clock latency, throughput, and KV-memory improvements does the modified PLT provide in practice?

We benchmarked modified PLT with a 20.0B-parameter model using SGLang 0.5.23 on a single NVIDIA GB300 GPU. We compare sequential looping against PLT with separate KV caches and PLT with a shared KV cache. The shared-cache implementation also fuses attention computations across loops.

Latency and Throughput: All latency and throughput results below use a 16k-token input prefix. Each configuration was measured three times after warmup, including model execution, token sampling, and batch preparation. Throughput is the total across the batch. The BF16 results are:

ConfigurationLatency, batch 1 (ms/token)Throughput, batch 1 (tokens/s)Throughput, batch 8 (tokens/s)Throughput, batch 64 (tokens/s)
Single-pass baseline8.241218242,837
Sequential, 2 loops15.38654391,454
PLT, 2 loops, separate caches9.211086741,672
PLT, 2 loops, shared cache8.611168032,673
Sequential, 4 loops29.7534226Does not fit
PLT, 4 loops, separate caches10.8692489Does not fit
PLT, 4 loops, shared cache8.691157722,560

With four loops, shared-cache PLT reduces latency from 29.75 to 8.69 ms/token and increases batch-8 throughput from 226 to 772 tokens/s, approximately a 3.4-fold improvement in both. It also supports batch 64, where sequential looping and separate-cache PLT exceed GPU memory. The separate-cache results show that parallel execution alone provides substantial latency gains; sharing the cache and fusing attention further improves performance. We also evaluated shared-cache PLT in FP8 and NVFP4:

PrecisionLoopsLatency, batch 1 (ms/token)Throughput, batch 8 (tokens/s)Throughput, batch 64 (tokens/s)
FP826.721,0164,435
FP846.811,0483,967
NVFP425.131,3875,020
NVFP445.101,3124,596

Lower precision further improves decoding performance, with four-loop NVFP4 reaching 5.10 ms/token and 4,596 tokens/s at batch 64. For both precisions, latency remains nearly unchanged when increasing from two to four loops.

Cache Memmory Savings: KV-cache sharing allows all loops to use one cache instead of maintaining a separate cache for each loop. For the 20.0B model in BF16, a single sequence with 64,000 cached tokens requires 11.72 GiB of cache memory with two loops and 23.44 GiB with four loops. Sharing reduces both to 5.86 GiB, saving 50% and 75% of cache memory, respectively. FP8 and NVFP4 both use an 8-bit KV cache, so these memory figures are halved while the percentage savings remain unchanged. These savings apply specifically to the cache: BF16 model weights still occupy an additional 37.38 GiB. Processing the input prompt, or prefill, still requires approximately one model pass per loop. For a single sequence with a 16k-token prompt, BF16 prefill takes 0.46 seconds for the single-pass baseline, 0.94 seconds for two-loop shared-cache PLT, and 1.91 seconds for four-loop shared-cache PLT. Thus, PLT substantially accelerates decoding, while prefill cost continues to grow approximately in proportion to the number of loops.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.