- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces the Branch-Decode transformer, a dual-residual-stream architecture for autoregressive transformer language models. Both streams share the same transformer-layer weights while only the embedding matrices and lm-head matrices are doubled. During pre-fill, only the main stream is invoked to build up the KV cache, while during decode, both streams attend to the same KV cache separately while only the main branch writes new kv entries, so that (1) the KV cache does not grow compared to a one-stream architecture, and (2) pre-fill (which is memory-bound) is alleviated of the higher two-stream memory burden. The authors pretrain gpt-2-style and llama-style models in dense and Mixture-of-Experts settings on ~10B tokens in 120M-1B parameter regimes, reporting log-loss along training. Branch-Decode transformer attains lower log-loss than a one-stream transformer as well as one close architectural baseline.
Critical strengths
- The idea of giving the model more decode compute while leaving its memory burden during pre-fill unchanged and also using the same weights on two different streams with different embeddings is relevant and interesting.
Critical weaknesses
-
The evaluation of the idea does not suffice for the claim that the architecture outperforms the prior state of the art and which effects its components have.
1.1 You are adding capacity to the model, not only through a second embedding as params but you are essentially doubling decode compute. Hence important baselines are missing: (1) comparable compute baselines, e.g. a 2-step looped transformer (you could compare on memory, flops and accuracy - your architecture has lower memory requirements but for compute it would be comparable); (2) also hyper-connections, manifold-constrained hyper-connections which are very close to your idea, (3) comparable compute baselines with more params, e.g regular two-stream architectures where the weights are doubled (e.g. as an upper bound to performance gains). Your paper is not only missing any discussion on these prior related works, but also does not compare to these, which would be essential for judging the effect of your specific architectural decisions.
1.2 Evaluation cannot only be loss/perplexity. Instead, you need downstream evals on a broad set of benchmarks too, e.g. GSM8K, PIQA, CSQA, MMLU, OpenBookQA, etc. For this, you do not train on enough training tokens, at least 50-100B tokens would be needed for above-chance performance on standard benchmarks at these scales (as shown in e.g. [1, 2, 3, 4]).
-
The manuscript is incredibly hard to understand.
2.1 It took me so much time and so many reads from start to finish, just to understand what your idea even is (although when I finally understood it I found it quite brilliant). Your primary contribution is a two-fold residual stream architecture, yet you never explicitly mention this nor the word "two streams" nor "residual stream" at all in the entire manuscript. Instead you introduce your method as: "We instantiate this objective with the Decode-Branch Transformer. The primary path never depends on the branch, while the branch reads primary states". The important sentence in between that is missing is exactly "[...], a two residual stream architecture", cause this way I was just wondering what the "primary path" and "the branch" are for a long time, also because the manuscript talks a lot about prefill and decode, so I thought of the two "paths/branches" as prefill and decode operations for a long time. The same applies to section 4, in which you talk about "primary trajectories" and "branch trajectories", but these terms do not reflect the nomenclature standard to the field of transformers and hence are difficult to understand.
2.2 There are additional problems with the manuscript, for example overly short sections that seem out of place (e.g. section 3) or overcrowded sections whose structure leaves the reader confused (e.g. section 4).
2.3 At times, symbols are left unintroduced (e.g. in Equation 9, I have no idea what "v" is).
2.4 Section 4.5 on experiments first talks about "modded" nano-GPT experiments and then about llama-style experiments. A more scientific way of presenting this would be to present these as gpt-2-style (which nanoGPT essentially is) and llama-style architectures in the same section. This way, the manuscript reads more like an engineering report. Also, important details such as the total number of trainable nanoGPT-params are left out which makes it difficult for me to judge whether the size and number of training tokens is sufficient.
Questions for authors
- How different are the two embeddings? I could imagine that if the model does not easily find a solution where both streams are very different, it just collapses to nearly the same computation done twice. Perhaps an additional loss that constrains the two embeddings to be as different as possible would be even better. Please provide a quantitative and qualitative analysis on multiple trained checkpoints showing how different the learned residual streams are and whether they perform completely different operations or just differ slightly. E.g. one important way of showing this would be to plot the cosine distance between the two embedding vectors of the same tokens across an entire corpus, and to analyze when and how this distance is small or large, and whether it differs across differently initialized models.
- How do you initialize your two distinct embedding matrices, and what effect does initialization have on the way the two residual streams evolve and their optimization dynamics?
- Can you provide experiments where models are trained on 50-100B tokens and evaluated on a broad range of downstream benchmarks such as GSM8K, PIQA, CSQA, MMLU, etc.?
- Can you provide a comprehensive discussion of prior multi-stream as well as weight sharing architectures?
- Can you show how your method compares to prior methods, as well as what the effect of each component is (e.g. the number of streams, whether weights are shared or not, the way the two streams' likelihoods are combined, etc)
5 comments
Rebuttal, part 2
W2. Overall clarity
“The manuscript is incredibly hard to understand.”
Response:
We will clarify the two-stream definition, section organization, notation, and configuration reporting as follows.
W2.1. Two residual streams versus the two inference phases
Response:
We will place the following definition near the beginning of the introduction:
Decode-Branch maintains two residual streams at each token position. The primary independently constructs causal states and KV; the branch reads primary information to improve next-token prediction, without feeding back into the primary or storing cross-position branch KV.
We will distinguish primary/branch, which name model-internal streams, from prefill/decode, which name inference phases. In the common regime motivating the paper, prefill tends to be compute-bound and low-batch decode memory-bound; the manuscript's phase definitions and §4.4 will be made easier to locate.
W2.2. Short background and crowded method section
Response:
We propose separating architecture/objective, inference execution/cost, and experiments, and consolidating the short background material. This gives the model definition and execution analysis distinct entry points.
W2.3. Undefined symbol v in Eq.9
Response:
In Eq.9, V is the vocabulary and v ∈ V indexes a vocabulary token; the sum is over the primary next-token probabilities. This definition will be included alongside the equation.
W2.4. NanoGPT naming, configuration, and parameter reporting
Response:
We use the GPT-style modded-NanoGPT configuration described in §4.5.1: 12 layers, width 768, head dimension 128, vocabulary size 50,257, sequence length 1,024, and a global batch of 512 sequences. The second embedding contains 50,257 × 768 = 38.6M parameters; major attention/MLP matrices and the output matrix are shared. Each data multiplier D corresponds to 3,800D training steps.
Questions for Authors
Q1. Embedding/stream differences, collapse, and checkpoint analysis
“How different are the two embeddings?”
Response:
We examine the underlying concern—whether the streams provide different predictive contributions—through readout and training-objective diagnostics. At the same standard DB checkpoint, the combined output improves validation loss by 0.31926 over primary-only readout:
| Readout | Validation loss |
|---|---|
| Primary stream alone | 3.55732 |
| Combined output | 3.23806 |
We also retrained the model with the 0.5 lower bound removed throughout training. Its primary-only loss increases to 5.36957:
| Training configuration | Primary-only validation loss |
|---|---|
| Standard DB, lower bound 0.5 | 3.55732 |
| DB retrained without the lower bound | 5.36957 |
This training-objective ablation shows that the lower bound helps maintain primary predictive capability. The combined-versus-primary result establishes a difference in predictive behavior at the evaluated checkpoint.
In a new NanoGPT D=1 control, tying the primary and branch input embeddings increases validation loss by only 0.006 relative to standard DB. The validation loss increases from 3.2380 to 3.2441 while the loss for baseline is 3.27777.. The gain therefore persists when the additional embedding table is removed: it is not explained entirely by the extra embedding capacity.
Q2. Embedding initialization and optimization dynamics
“How do you initialize your two distinct embedding matrices”
Response:
The two input embedding tables are randomly initialized using torch.nn.Embedding. We will include the initialization details in the implementation description.
Q3. 50–100B-token training and downstream tasks
Response:
The dense LLaMA 1B setting uses approximately 80B training tokens (§4.5.3); the added results improve PIQA from 71.6 to 72.8 and OpenBookQA from 33.4 to 34.2. W1.2 gives the task table. NanoGPT also extends to approximately 39.85B tokens (§4.5.1).
Rebuttal, part 3
Q4. Multi-stream and weight-sharing related work
Response:
The comparison concerns both residual computation and persistent state:
| Architecture | Relevant strength | Distinction from DB |
|---|---|---|
| PHD-2 | Extra prediction computation with original-token long-range KV | DB uses same-position query-only branch attention, ordinary causal kernels, and likelihood combination. DB does not require additional attention kernels or incur extra latency during training to accommodate more fragmented attention masks. |
| PLT | Parallel decode across loops/tokens and shared first-loop global KV | Its training graph has dependent full-backbone loops; DB uses same-position streams in one layer stack |
| ParScale | Shared-backbone parallel streams and prediction aggregation | Its streams process prompts and maintain separate KV states; DB preserves primary prefill and one KV history |
| HC/mHC | Residual-stream routing with limited extra major layer-function arithmetic | DB adds a separately evaluated predictive stream with asymmetric dependencies and joint readout |
The original §2 provides the PHD/PLT/ParScale discussion; W1.1 supplies the added mHC result and the concrete execution comparisons.
Q5. Prior methods, number of streams, weight sharing, and likelihood combination
“the number of streams, whether weights are shared or not, the way the two streams’ likelihoods are combined”
Response:
The submitted experiments separate the following design choices:
- Prior method and readout: minimal branch-only DB approximately matches PHD-2 at smaller budgets and improves as data grows; coupling gives a small consistent increment and the probability mixture performs best (§4.5.2/Fig.3; C.1/Fig.8).
- Number of branches: two decode branches improve loss over one branch at every reported D=1–5 budget (D.1/Table 3/Fig.11).
- Weight sharing: branch-specific Q/O/MLP weights improve quality while increasing the stored weight footprint (D.2/Fig.12).
- Input embeddings: the new D=1 tied-embedding control increases loss by only 0.006, so the quality gain persists without the second input table.
- Mixture lower bound: retraining without the 0.5 bound increases primary-only loss from 3.55732 to 5.36957 (Q1).
The branch-count and partial-unsharing experiments vary capacity and computation; the tied-embedding control isolates the effect of the second input table.
Rebuttal, part 1
Thank you for the detailed suggestions on comparisons, stream behavior, and presentation. We clarify two factual points about Decode-Branch (DB) before addressing the individual comments.
Parameter sharing. The input embeddings are separate, but the LM-head/output matrix is shared, as stated immediately before Eq.8 (§4.2). There is no second independently parameterized LM head. The extra vocabulary-scale parameter table is the branch embedding.
Training budget and scope. The experiments are not limited to approximately 10B tokens: NanoGPT runs extend to D=20, approximately 39.85B tokens (§4.5.1), and the dense 1B-scale setting uses approximately 80B tokens (§4.5.3).
Critical Weaknesses
W1. Evaluation and attribution of the architectural contribution
Response:
We add an mHC comparison, two downstream tasks, and a tied-embedding control below. These complement the submitted PHD-2 and component experiments and evaluate the complete phase-decoupled shared-KV design.
W1.1(1). Comparable-compute two-step looped Transformer
Response:
A conventional two-step looped Transformer shares parameters but executes dependent full-backbone traversals at prompt and generated positions. DB instead organizes same-position streams within one layer stack, preserving the baseline prefill computation and persistent KV-cache size. Its execution targets the mismatch between prompt computation and low-batch decode.
The structural comparison concerns sequential looping; the following measurements quantify DB's additional serving cost relative to its single-stream backbone. Both models use the paper's dense 1B configuration, batch size 1, BF16, and FlashAttention:
These measurements on DB decode use the implementation that realizes the grouped, fused execution as mentioned in the paper. Because the primary and branch streams share weights, we group them and compute each shared projection as a single fused matmul, so each layer needs one kernel launch and one weight load. For attention, the branch query is packed into the same GQA FlashAttention call as the primary queries, so both streams read the KV cache in one pass.
This directly quantifies the execution benefit in the measured low-batch setting.
W1.1(2). HC and manifold-constrained HC
Response:
DB achieves 3.23806, compared with 3.25630 for the aligned standard mHC implementation, under the modded-NanoGPT D=1 setting of §4.5.1:
HC/mHC enrich residual-stream routing; DB evaluates an additional prediction stream with one-way primary-to-branch dependence and a shared persistent KV history. The table compares observed quality at the same training-step budget.
W1.1(3). Higher-parameter two-stream reference
Response:
The partial-unsharing experiment in Appendix D.2/Fig.12 gives the branch independent Q/O/MLP weights while retaining primary KV, and achieves better predictive quality. It also increases parameter and optimizer-state memory during training and enlarges the decode weight footprint. This experiment shows the capacity benefit and the resource trade-off of relaxing DB's sharing constraint.
W1.1, concluding concern. Missing discussion of these related methods
Response:
The original §2 covers PHD, PLT, and ParScale. The comparisons above add HC/mHC and distinguish residual routing, serial looping, and branch-specific capacity. Q4 summarizes their different execution and state-sharing properties.
W1.2. Downstream evaluation and training budget
Response:
Our dense 1B-scale experiment already uses approximately 80B training tokens (§4.5.3). Under this dense LLaMA 1B setting, the added downstream results are:
Both measured tasks improve, supplementing the language-modeling quality results.