acceptodds
Under review as a conference paper at ICLR 2027

Decode-Branch Transformers: Decoupling the Primary Prefill Path from Additional Decode Computation

Abstract

As large language models serve ever more requests, cumulative inference cost is growing relative to the one-time cost of training. In common serving regimes the two inference phases place different demands on hardware: prompt prefill runs in parallel and tends to be compute-bound, whereas autoregressive decode is sequential and memory-traffic-bound. Conventional width or depth scaling raises both costs together, since every added layer is evaluated in both phases and enlarges the weights read at every decode step. We instead ask whether additional learned computation can be allocated to continuation prediction while preserving the prompt-wide primary computation and a single persistent key–value (KV) cache. We realize this separation with the Decode-Branch Transformer. Its primary path is a complete causal language model that alone processes the prompt and writes the KV cache. At inference, the decode branch is omitted at earlier prompt positions and activated at the final prompt position and each generated position, where it adds continuation-prediction computation without writing persistent state or influencing the primary path. The paths share all major attention, MLP, and output matrices and use separate token embeddings with lightweight coupling. Grouped decode reuses each loaded weight tile and primary-cache region across both paths, so the added arithmetic does not proportionally increase the dominant memory traffic or decode latency. Across matched-training-token comparisons, Decode-Branch attains lower validation loss across architectures and data configurations. A training-compute control further favors interacting trajectories over additional tokens. In MoE models, its structural separation makes the primary and branch expert fan-outs independent knobs for trading prompt cost, continuation cost, and predictive quality. We study two allocation regimes: fixing prefill expert computation while increasing decode computation, and fixing decode expert computation while reallocating the expert budget between the two paths. Together, these experiments expose a prefill–decode–quality trade-off and establish a structural opportunity for phase-specific expert allocation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Prefilling, Decoding, Inference, Transformer, Computation Allocation, KV cache

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_q5FtOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces the Branch-Decode transformer, a dual-residual-stream architecture for autoregressive transformer language models. Both streams share the same transformer-layer weights while only the embedding matrices and lm-head matrices are doubled. During pre-fill, only the main stream is invoked to build up the KV cache, while during decode, both streams attend to the same KV cache separately while only the main branch writes new kv entries, so that (1) the KV cache does not grow compared to a one-stream architecture, and (2) pre-fill (which is memory-bound) is alleviated of the higher two-stream memory burden. The authors pretrain gpt-2-style and llama-style models in dense and Mixture-of-Experts settings on ~10B tokens in 120M-1B parameter regimes, reporting log-loss along training. Branch-Decode transformer attains lower log-loss than a one-stream transformer as well as one close architectural baseline.

Critical strengths
  1. The idea of giving the model more decode compute while leaving its memory burden during pre-fill unchanged and also using the same weights on two different streams with different embeddings is relevant and interesting.
Critical weaknesses
  1. The evaluation of the idea does not suffice for the claim that the architecture outperforms the prior state of the art and which effects its components have.

    1.1 You are adding capacity to the model, not only through a second embedding as params but you are essentially doubling decode compute. Hence important baselines are missing: (1) comparable compute baselines, e.g. a 2-step looped transformer (you could compare on memory, flops and accuracy - your architecture has lower memory requirements but for compute it would be comparable); (2) also hyper-connections, manifold-constrained hyper-connections which are very close to your idea, (3) comparable compute baselines with more params, e.g regular two-stream architectures where the weights are doubled (e.g. as an upper bound to performance gains). Your paper is not only missing any discussion on these prior related works, but also does not compare to these, which would be essential for judging the effect of your specific architectural decisions.

    1.2 Evaluation cannot only be loss/perplexity. Instead, you need downstream evals on a broad set of benchmarks too, e.g. GSM8K, PIQA, CSQA, MMLU, OpenBookQA, etc. For this, you do not train on enough training tokens, at least 50-100B tokens would be needed for above-chance performance on standard benchmarks at these scales (as shown in e.g. [1, 2, 3, 4]).

  2. The manuscript is incredibly hard to understand.

    2.1 It took me so much time and so many reads from start to finish, just to understand what your idea even is (although when I finally understood it I found it quite brilliant). Your primary contribution is a two-fold residual stream architecture, yet you never explicitly mention this nor the word "two streams" nor "residual stream" at all in the entire manuscript. Instead you introduce your method as: "We instantiate this objective with the Decode-Branch Transformer. The primary path never depends on the branch, while the branch reads primary states". The important sentence in between that is missing is exactly "[...], a two residual stream architecture", cause this way I was just wondering what the "primary path" and "the branch" are for a long time, also because the manuscript talks a lot about prefill and decode, so I thought of the two "paths/branches" as prefill and decode operations for a long time. The same applies to section 4, in which you talk about "primary trajectories" and "branch trajectories", but these terms do not reflect the nomenclature standard to the field of transformers and hence are difficult to understand.

    2.2 There are additional problems with the manuscript, for example overly short sections that seem out of place (e.g. section 3) or overcrowded sections whose structure leaves the reader confused (e.g. section 4).

    2.3 At times, symbols are left unintroduced (e.g. in Equation 9, I have no idea what "v" is).

    2.4 Section 4.5 on experiments first talks about "modded" nano-GPT experiments and then about llama-style experiments. A more scientific way of presenting this would be to present these as gpt-2-style (which nanoGPT essentially is) and llama-style architectures in the same section. This way, the manuscript reads more like an engineering report. Also, important details such as the total number of trainable nanoGPT-params are left out which makes it difficult for me to judge whether the size and number of training tokens is sufficient.

Questions for authors
  1. How different are the two embeddings? I could imagine that if the model does not easily find a solution where both streams are very different, it just collapses to nearly the same computation done twice. Perhaps an additional loss that constrains the two embeddings to be as different as possible would be even better. Please provide a quantitative and qualitative analysis on multiple trained checkpoints showing how different the learned residual streams are and whether they perform completely different operations or just differ slightly. E.g. one important way of showing this would be to plot the cosine distance between the two embedding vectors of the same tokens across an entire corpus, and to analyze when and how this distance is small or large, and whether it differs across differently initialized models.
  2. How do you initialize your two distinct embedding matrices, and what effect does initialization have on the way the two residual streams evolve and their optimization dynamics?
  3. Can you provide experiments where models are trained on 50-100B tokens and evaluated on a broad range of downstream benchmarks such as GSM8K, PIQA, CSQA, MMLU, etc.?
  4. Can you provide a comprehensive discussion of prior multi-stream as well as weight sharing architectures?
  5. Can you show how your method compares to prior methods, as well as what the effect of each component is (e.g. the number of streams, whether weights are shared or not, the way the two streams' likelihoods are combined, etc)
5 comments
AuthorsOpenReview ↗

Rebuttal, part 1

Thank you for the detailed suggestions on comparisons, stream behavior, and presentation. We clarify two factual points about Decode-Branch (DB) before addressing the individual comments.

Parameter sharing. The input embeddings are separate, but the LM-head/output matrix is shared, as stated immediately before Eq.8 (§4.2). There is no second independently parameterized LM head. The extra vocabulary-scale parameter table is the branch embedding.

Training budget and scope. The experiments are not limited to approximately 10B tokens: NanoGPT runs extend to D=20, approximately 39.85B tokens (§4.5.1), and the dense 1B-scale setting uses approximately 80B tokens (§4.5.3).


Critical Weaknesses

W1. Evaluation and attribution of the architectural contribution

“The evaluation of the idea does not suffice”

Response:

We add an mHC comparison, two downstream tasks, and a tied-embedding control below. These complement the submitted PHD-2 and component experiments and evaluate the complete phase-decoupled shared-KV design.

W1.1(1). Comparable-compute two-step looped Transformer

“a 2-step looped transformer”

Response:

A conventional two-step looped Transformer shares parameters but executes dependent full-backbone traversals at prompt and generated positions. DB instead organizes same-position streams within one layer stack, preserving the baseline prefill computation and persistent KV-cache size. Its execution targets the mismatch between prompt computation and low-batch decode.

The structural comparison concerns sequential looping; the following measurements quantify DB's additional serving cost relative to its single-stream backbone. Both models use the paper's dense 1B configuration, batch size 1, BF16, and FlashAttention:

Prefix lengthSingle-stream decodeDB decodeAdditional timeAdditional latency
1,0241.424 ms1.508 ms≈84 μs5.87%
2,0481.491 ms1.574 ms≈83 μs5.59%
4,0961.915 ms2.003 ms≈88 μs4.62%
8,1922.424 ms2.508 ms≈84 μs3.47%
16,3843.770 ms3.866 ms≈96 μs2.54%
32,7685.856 ms5.948 ms≈91 μs1.56%

These measurements on DB decode use the implementation that realizes the grouped, fused execution as mentioned in the paper. Because the primary and branch streams share weights, we group them and compute each shared projection as a single fused matmul, so each layer needs one kernel launch and one weight load. For attention, the branch query is packed into the same GQA FlashAttention call as the primary queries, so both streams read the KV cache in one pass.

This directly quantifies the execution benefit in the measured low-batch setting.

W1.1(2). HC and manifold-constrained HC

“manifold-constrained hyper-connections which are very close to your idea”

Response:

DB achieves 3.23806, compared with 3.25630 for the aligned standard mHC implementation, under the modded-NanoGPT D=1 setting of §4.5.1:

ModelValidation loss at 3,800 steps
Baseline3.27777
mHC3.25630
Decode-Branch3.23806

HC/mHC enrich residual-stream routing; DB evaluates an additional prediction stream with one-way primary-to-branch dependence and a shared persistent KV history. The table compares observed quality at the same training-step budget.

W1.1(3). Higher-parameter two-stream reference

“regular two-stream architectures where the weights are doubled”

Response:

The partial-unsharing experiment in Appendix D.2/Fig.12 gives the branch independent Q/O/MLP weights while retaining primary KV, and achieves better predictive quality. It also increases parameter and optimizer-state memory during training and enlarges the decode weight footprint. This experiment shows the capacity benefit and the resource trade-off of relaxing DB's sharing constraint.

W1.1, concluding concern. Missing discussion of these related methods

Response:

The original §2 covers PHD, PLT, and ParScale. The comparisons above add HC/mHC and distinguish residual routing, serial looping, and branch-specific capacity. Q4 summarizes their different execution and state-sharing properties.

W1.2. Downstream evaluation and training budget

“Evaluation cannot only be loss/perplexity.”

Response:

Our dense 1B-scale experiment already uses approximately 80B training tokens (§4.5.3). Under this dense LLaMA 1B setting, the added downstream results are:

TaskBaselineDBImprovement
PIQA71.672.8+1.2
OpenBookQA33.434.2+0.8

Both measured tasks improve, supplementing the language-modeling quality results.

AuthorsOpenReview ↗

Rebuttal, part 2

W2. Overall clarity

“The manuscript is incredibly hard to understand.”

Response:

We will clarify the two-stream definition, section organization, notation, and configuration reporting as follows.


W2.1. Two residual streams versus the two inference phases

Response:

We will place the following definition near the beginning of the introduction:

Decode-Branch maintains two residual streams at each token position. The primary independently constructs causal states and KV; the branch reads primary information to improve next-token prediction, without feeding back into the primary or storing cross-position branch KV.

We will distinguish primary/branch, which name model-internal streams, from prefill/decode, which name inference phases. In the common regime motivating the paper, prefill tends to be compute-bound and low-batch decode memory-bound; the manuscript's phase definitions and §4.4 will be made easier to locate.


W2.2. Short background and crowded method section

Response:

We propose separating architecture/objective, inference execution/cost, and experiments, and consolidating the short background material. This gives the model definition and execution analysis distinct entry points.


W2.3. Undefined symbol v in Eq.9

Response:

In Eq.9, V is the vocabulary and v ∈ V indexes a vocabulary token; the sum is over the primary next-token probabilities. This definition will be included alongside the equation.


W2.4. NanoGPT naming, configuration, and parameter reporting

Response:

We use the GPT-style modded-NanoGPT configuration described in §4.5.1: 12 layers, width 768, head dimension 128, vocabulary size 50,257, sequence length 1,024, and a global batch of 512 sequences. The second embedding contains 50,257 × 768 = 38.6M parameters; major attention/MLP matrices and the output matrix are shared. Each data multiplier D corresponds to 3,800D training steps.


Questions for Authors

Q1. Embedding/stream differences, collapse, and checkpoint analysis

“How different are the two embeddings?”

Response:

We examine the underlying concern—whether the streams provide different predictive contributions—through readout and training-objective diagnostics. At the same standard DB checkpoint, the combined output improves validation loss by 0.31926 over primary-only readout:

ReadoutValidation loss
Primary stream alone3.55732
Combined output3.23806

We also retrained the model with the 0.5 lower bound removed throughout training. Its primary-only loss increases to 5.36957:

Training configurationPrimary-only validation loss
Standard DB, lower bound 0.53.55732
DB retrained without the lower bound5.36957

This training-objective ablation shows that the lower bound helps maintain primary predictive capability. The combined-versus-primary result establishes a difference in predictive behavior at the evaluated checkpoint.

In a new NanoGPT D=1 control, tying the primary and branch input embeddings increases validation loss by only 0.006 relative to standard DB. The validation loss increases from 3.2380 to 3.2441 while the loss for baseline is 3.27777.. The gain therefore persists when the additional embedding table is removed: it is not explained entirely by the extra embedding capacity.


Q2. Embedding initialization and optimization dynamics

“How do you initialize your two distinct embedding matrices”

Response:

The two input embedding tables are randomly initialized using torch.nn.Embedding. We will include the initialization details in the implementation description.


Q3. 50–100B-token training and downstream tasks

Response:

The dense LLaMA 1B setting uses approximately 80B training tokens (§4.5.3); the added results improve PIQA from 71.6 to 72.8 and OpenBookQA from 33.4 to 34.2. W1.2 gives the task table. NanoGPT also extends to approximately 39.85B tokens (§4.5.1).

AuthorsOpenReview ↗

Rebuttal, part 3

Q4. Multi-stream and weight-sharing related work

Response:

The comparison concerns both residual computation and persistent state:

ArchitectureRelevant strengthDistinction from DB
PHD-2Extra prediction computation with original-token long-range KVDB uses same-position query-only branch attention, ordinary causal kernels, and likelihood combination. DB does not require additional attention kernels or incur extra latency during training to accommodate more fragmented attention masks.
PLTParallel decode across loops/tokens and shared first-loop global KVIts training graph has dependent full-backbone loops; DB uses same-position streams in one layer stack
ParScaleShared-backbone parallel streams and prediction aggregationIts streams process prompts and maintain separate KV states; DB preserves primary prefill and one KV history
HC/mHCResidual-stream routing with limited extra major layer-function arithmeticDB adds a separately evaluated predictive stream with asymmetric dependencies and joint readout

The original §2 provides the PHD/PLT/ParScale discussion; W1.1 supplies the added mHC result and the concrete execution comparisons.


Q5. Prior methods, number of streams, weight sharing, and likelihood combination

“the number of streams, whether weights are shared or not, the way the two streams’ likelihoods are combined”

Response:

The submitted experiments separate the following design choices:

  • Prior method and readout: minimal branch-only DB approximately matches PHD-2 at smaller budgets and improves as data grows; coupling gives a small consistent increment and the probability mixture performs best (§4.5.2/Fig.3; C.1/Fig.8).
  • Number of branches: two decode branches improve loss over one branch at every reported D=1–5 budget (D.1/Table 3/Fig.11).
  • Weight sharing: branch-specific Q/O/MLP weights improve quality while increasing the stored weight footprint (D.2/Fig.12).
  • Input embeddings: the new D=1 tied-embedding control increases loss by only 0.006, so the quality gain persists without the second input table.
  • Mixture lower bound: retraining without the 0.5 bound increases primary-only loss from 3.55732 to 5.36957 (Q1).

The branch-count and partial-unsharing experiments vary capacity and computation; the tied-embedding control isolates the effect of the second input table.

Reviewer_HP3JOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the Decode-Branch Transformer. A standard primary path processes the prompt and writes the KV cache. A second branch path, with its own token embedding and lightweight coupling vectors, shares all major weights and reads the primary KV cache but writes no persistent state. The branch runs only at the final prompt position and at generated positions, and the prediction is a mixture of the two paths’ next-token distributions. The authors argue this adds decode-time computation without increasing prefill cost, KV size, or memory traffic. For MoE models, separate expert counts for the two paths allow prefill and decode compute to be set independently. Experiments on NanoGPT and LLaMA-style dense and MoE models up to 1B show lower validation loss than baselines at matched training tokens.

Critical strengths
  • The idea of adding computation only at continuation steps, while keeping the prompt path, KV cache, and weights unchanged, is clean. Training and inference stay consistent because the branch never writes state, and the asymmetric coupling makes the primary path independent of the branch.

  • The gains in validation loss are consistent across data budgets, model sizes, and dense and MoE backbones, and the ablations against PHD-2 and the component variants are reasonably controlled. The MoE experiments expose an interesting trade-off between prefill and decode expert budgets that standard architectures cannot express.

Critical weaknesses
  • The central efficiency claim is never measured. The paper argues that grouped decode adds little latency because decode is memory-bound, but it reports no decode latency, throughput, or prefill time, and no kernel implementation. The claim rests only on a roofline argument.

  • The efficiency argument holds only in the low-batch, memory-bound regime. In high-throughput serving with large batches, decode approaches the compute roof, and doubling the backbone arithmetic would directly increase latency. The paper acknowledges this but does not quantify where the crossover occurs.

  • The main comparisons are matched in training tokens, not in training compute or inference cost. Decode-Branch uses about twice the training FLOPs and an extra embedding table. The only compute-matched comparison is a single NanoGPT point, and there is no comparison against a wider or deeper model with matched decode latency, which is the alternative the paper claims to improve on.

  • The extra embedding table is a significant parameter increase at small scales, for example about 38M parameters for a 50k vocabulary at width 768. No ablation gives the baseline comparable extra parameters, so part of the gain may come from added capacity rather than from the branch design.

  • The ablation suggests much of the gain comes from the probability mixture readout, which behaves like ensembling two predictors. The branch-only variant roughly matches PHD-2, and coupling adds only a small improvement. This weakens the claim that the gain comes from additional interacting computation.

  • The reduced-prefill claims are weak under strict accounting. In the strictly compute-matched MoE control, halving prefill routed-expert compute only matches the baseline, and quarter-prefill is worse. This control is run only at one small scale. The default fixed-decode slices are not matched in attention and shared-expert compute.

  • The evaluation uses only validation loss, with no downstream benchmarks or generation quality. The experiments use models up to 1B with 1,024-token sequences, while the motivation emphasizes long-context inference and agents.

  • Baselines are limited. The paper compares only against PHD-2 and does not include Parallel Scaling, Parallel Loop Transformer, or other methods that add latent computation per token.

Questions for authors

please see questions outlined in weakness

2 comments
AuthorsOpenReview ↗

Rebuttal, Part 1

Thank you for recognizing the asymmetric design, training–inference consistency, and quality improvements. We provide additional measurements and attribution controls for Decode-Branch (DB) below.


W1. Missing system measurements and implementation evidence

“The central efficiency claim is never measured.”

Response:

For the paper's dense 1B model at batch size 1 in BF16, with FlashAttention enabled for both models, the measured decode latencies are:

Prefix lengthSingle-stream decodeDB decodeAdditional timeAdditional latency
1,0241.424 ms1.508 ms≈84 μs5.87%
2,0481.491 ms1.574 ms≈83 μs5.59%
4,0961.915 ms2.003 ms≈88 μs4.62%
8,1922.424 ms2.508 ms≈84 μs3.47%
16,3843.770 ms3.866 ms≈96 μs2.54%
32,7685.856 ms5.948 ms≈91 μs1.56%

These measurements on DB decode use the implementation that realizes the grouped, fused execution as mentioned in the paper. Because the primary and branch streams share weights, we group them and compute each shared projection as a single fused matmul, so each layer needs one kernel launch and one weight load. For attention, the branch query is packed into the same GQA FlashAttention call as the primary queries, so both streams read the KV cache in one pass.

This directly quantifies the execution benefit in the measured low-batch setting.

For prefill, DB executes exactly the same model and computation graph as the baseline (§4.4), so its prefill time is exactly 1× the baseline. Its persistent KV-cache size and structure are unchanged.


W2. Low-batch regime and unmeasured crossover

“The paper acknowledges this but does not quantify where the crossover occurs.”

Response:

Our measurements evaluate batch size 1, a setting relevant to local single-device deployment and low-volume or sporadic inference services, where forming large batches can introduce queueing delay. In this setting, the table in W1 shows that additional prediction computation incurs a small decode increment. This is the low-batch deployment regime targeted by the architecture.


W3. Matched tokens versus matched training/inference cost

“The main comparisons are matched in training tokens, not in training compute or inference cost.”

Response:

At the same training-token budget, DB uses approximately 1.90× the matmul FLOPs and 1.926× the training time of the baseline. Fig.2 compares DB D=10 with baseline D=20, giving the baseline twice the training tokens. DB is the cheaper member of this pair: approximately 5.0% fewer matmul FLOPs and 4.0% less training time, while attaining lower validation loss. The resource allocation therefore favors the baseline, making this a conservative test of DB's quality advantage.

ConfigurationStepsTraining tokensTraining matmul FLOPsTraining time
Baseline, D=1038,00019.923B17.027 × 10¹⁸1.76 h
DB, D=1038,00019.923B32.362 × 10¹⁸3.39 h
Baseline, D=2076,00039.846B34.054 × 10¹⁸3.53 h

The table reports the training FLOPs and wall-clock times recorded for the compared runs. At the same training-token budget, the recorded DB-to-baseline time ratio is 1.926×.

The main quality comparisons in §4.5 use matched training tokens; Fig.2 provides the separate approximate training-compute comparison above. Together with W1, these results quantify both training cost and low-batch decode cost for the corresponding primary/base architecture.


W4. Additional embedding capacity

“No ablation gives the baseline comparable extra parameters, so part of the gain may come from added capacity rather than from the branch design.”

Response:

In a new NanoGPT D=1 control, tying the primary and branch input embeddings increases validation loss by only 0.006 relative to standard DB. The validation loss increases from 3.2380 to 3.2441 while the loss for baseline is 3.27777. The gain therefore persists when the additional embedding table is removed: it is not explained entirely by the extra embedding capacity.

The standard second table adds 38.6M parameters at the NanoGPT vocabulary and width. This token-lookup capacity has a different execution cost from adding backbone width/depth: the branch embedding is omitted at earlier prompt positions, and the backbone matrices remain shared.

AuthorsOpenReview ↗

Rebuttal, Part 2

W5. Mixture contributes more than coupling

“The branch-only variant roughly matches PHD-2, and coupling adds only a small improvement.”

Response:

Likelihood combination is a core component of the method (§4.2): the introduction explicitly gives both trajectories a direct predictive training signal. Fig.3 and §4.5.2 already separate the effects: coupling provides a small consistent gain, and the probability-mixture version performs best. The empirical contribution is this complete architecture/objective design.

Beyond the likelihood-combination objective, DB uses ordinary causal attention over the primary keys and values without an interleaved or method-specific mask (§4.1). It supports standard FlashAttention directly, avoiding the specialized attention-kernel handling needed for efficient PHD-style structured attention.


W6. Interpretation of the stricter MoE control

“This control is run only at one small scale.”

Response:

At 0.25B, K=4, the half allocation achieves validation loss close to baseline while reducing prompt routed-expert applications by 50% (C.3/Fig.10). This is the intended quality–prompt-cost trade-off. The quarter allocation has worse loss.

The default fixed-decode slices match routed-expert applications; C.3 additionally matches one attention evaluation and one shared-expert evaluation per layer and decode position. The stricter result is reported at this scale, with the saving defined specifically in prefilling routed experts..


W7. Missing downstream evaluation and application extrapolation

“The evaluation uses only validation loss, with no downstream benchmarks or generation quality.”

Response:

Under our dense LLaMA 1B setting trained with approximately 80B tokens, DB improves both evaluated downstream tasks:

TaskBaselineDBImprovement
PIQA71.672.8+1.2
OpenBookQA33.434.2+0.8

These results extend the quality evidence beyond validation loss.


W8. Limited architectural baselines

“The paper compares only against PHD-2 and does not include Parallel Scaling, Parallel Loop Transformer, or other methods that add latent computation per token.”

Response:

In a new comparison under the modded-NanoGPT D=1 setting of §4.5.1, baseline/mHC/DB validation losses are 3.27777/3.25630/3.23806, respectively. DB improves over the aligned standard mHC baseline by 0.01824 at the same training-step budget.

The original §2 discusses PHD, PLT, and ParScale, with a controlled PHD-2 experiment in §4.5.2.

ParScale shares backbone weights and aggregates parallel predictions; its streams also process the prompt and maintain separate KV states. DB preserves the baseline prompt computation and one persistent KV history. PLT supports parallel decode across loops/tokens, while its two-loop training graph contains dependent full-backbone traversals. DB organizes same-position streams within one layer stack, supporting parallel training without a second full-depth traversal.

We would also like to emphasize that, beyond achieving a lower loss, our DB architecture offers another advantage over the PHD architecture: it does not require additional attention kernels or incur extra latency during training to accommodate more fragmented attention masks.

Reviewer_byujOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces the Decode-Branch Transformer, which decouples prompt prefill computation from additional continuation-side computation. A complete primary causal path alone processes the prompt and maintains the persistent KV cache, while a second branch is activated only at the final prompt position and subsequent decode positions. The branch shares the main attention, MLP, and output matrices, reads the primary KV cache without writing its own persistent state, and receives lightweight asymmetric coupling from the primary trajectory. The two paths are trained using an adaptive mixture next-token objective. The paper reports consistent validation-loss improvements over matched-token Transformer baselines across NanoGPT data scaling, dense and sparse LLaMA-style models, component ablations, and MoE expert-allocation experiments; an approximate training-compute comparison also favors the interacting trajectories over training the baseline on additional tokens.

Critical strengths

The main strength is a well-motivated architectural separation between prompt computation and continuation computation. In particular, the primary path remains a complete causal model and is independent of the branch, while the branch reuses the primary KV cache and the major model matrices without creating persistent branch state. This directly realizes the paper’s intended prefill/decode asymmetry rather than merely imposing a runtime approximation. The method improves validation loss across matched-token data scaling, dense and sparse model scales, and MoE allocation settings, while the component ablations indicate that both primary-to-branch coupling and mixture readout contribute to the final result.

Critical weaknesses

The weakness is that the central systems-efficiency claim is supported primarily by a roofline argument rather than by measurements of an implemented inference system. Section 4.4 argues that grouped execution can reuse loaded weight tiles and primary KV regions, so the nearly doubled backbone arithmetic need not produce a proportional increase in decode latency in a memory-bound regime. However, the experiments mainly establish predictive-quality and expert-computation trade-offs; they do not report end-to-end inter-token latency, throughput, memory traffic, or hardware utilization across batch sizes and context lengths. Consequently, the paper convincingly establishes that Decode-Branch preserves a single persistent KV cache and does not introduce a second unique backbone, but it remains uncertain how much of the proposed latency advantage is realized in practice and over what serving regimes. Since this practical benefit is central to the motivation, the missing systems validation limits the strength of the conclusion.

Questions for authors
  1. Can the authors provide end-to-end decode latency or throughput measurements for an implementation that realizes the proposed grouped or fused execution? Comparisons against the base Transformer and an appropriate width- or depth-scaled baseline across representative batch sizes and context lengths would clarify where Decode-Branch remains memory-bound and whether its additional arithmetic produces substantially less than a proportional latency increase.

  2. For the approximate training-compute comparison in Fig. 2, what are the actual FLOP estimates, parameter counts, and measured training times of the compared runs? The paper notes that the D=20D=20 Transformer is slightly more expensive than the D=10D=10 Decode-Branch because the branch does not construct an additional set of keys and values. Quantifying this difference, and indicating whether the conclusion remains unchanged when matching measured training cost, would make the claim that interacting trajectories outperform additional training tokens more precise.

1 comment

Thank you for recognizing the architectural separation and consistent quality improvements. We provide additional execution and training-cost evidence for Decode-Branch (DB) below.


Critical Weakness: Systems claims rely on roofline reasoning rather than execution measurements

“supported primarily by a roofline argument rather than by measurements of an implemented inference system”

Response:

We now provide direct decode measurements across six prefix lengths, together with a quantitative training-cost comparison. The dense 1B, batch-1, BF16 results in Q1 show a small decode increment with FlashAttention enabled for both models.


Question 1: Implemented execution, baseline comparisons, and batch/context regimes

“Can the authors provide end-to-end decode latency or throughput measurements for an implementation that realizes the proposed grouped or fused execution?”

Response:

For the paper's dense 1B model at batch size 1 in BF16, with FlashAttention enabled for both models, the measured decode latencies are:

Prefix lengthBaselineDB decodeAdditional timeAdditional latency
1,0241.424 ms1.508 ms≈84 μs5.87%
2,0481.491 ms1.574 ms≈83 μs5.59%
4,0961.915 ms2.003 ms≈88 μs4.62%
8,1922.424 ms2.508 ms≈84 μs3.47%
16,3843.770 ms3.866 ms≈96 μs2.54%
32,7685.856 ms5.948 ms≈91 μs1.56%

These measurements on DB decode use the implementation that realizes the grouped, fused execution as mentioned in the paper. Because the primary and branch streams share weights, we group them and compute each shared projection as a single fused matmul, so each layer needs one kernel launch and one weight load. For attention, the branch query is packed into the same GQA FlashAttention call as the primary queries, so both streams read the KV cache in one pass.

This table directly quantifies the execution benefit in the measured low-batch setting.

For prefill, DB executes exactly the same model and computation graph as the baseline (§4.4), so its prefill time is exactly 1× the baseline. Its persistent KV-cache size and structure are also unchanged. Width/depth scaling increases prefill&decode time adn cache size; our comparison isolates the gain available while preserving the baseline's computation time and cache.


Question 2: Fig.2 FLOPs, parameters, and training time

“what are the actual FLOP estimates, parameter counts, and measured training times of the compared runs?”

Response:

At the same training-token budget, DB uses approximately 1.90× the matmul FLOPs and 1.926× the training time of the baseline. Fig.2 compares DB D=10 with baseline D=20, giving the baseline twice the training tokens. DB is the cheaper member of this pair: approximately 5.0% fewer matmul FLOPs and 4.0% less training time, while attaining lower validation loss. The resource allocation therefore favors the baseline, making this a conservative test of DB's quality advantage.

ConfigurationStepsTraining tokensTraining matmul FLOPsTraining time
Baseline, D=1038,00019.923B17.027 × 10¹⁸1.76 h
DB, D=1038,00019.923B32.362 × 10¹⁸3.39 h
Baseline, D=2076,00039.846B34.054 × 10¹⁸3.53 h

The table reports the training FLOPs and wall-clock times recorded for the compared runs. At the same training-token budget, the recorded DB-to-baseline time ratio is 1.926×.

The additional embedding contains 50,257 × 768 ≈ 38.6 M parameters. These are token-lookup parameters, so adding this table does not increase dense-matrix computation as width/depth scaling would. Attention/MLP matrices and the output matrix remain shared; coupling vectors contribute less than 0.1% of non-embedding parameters (§4.4).

KV cache and computation: the persistent cache is exactly the same size and structure as the baseline. Only the primary computes and writes K/V; the branch reads the same cache. K/V projections are evaluated once per layer and token, exactly as in the baseline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.