acceptodds
Under review as a conference paper at ICLR 2027

Progressive Branch-Merge:Trainable Sequence Processing Across Frozen Transformer Depth

Abstract

Equally sized additions to a frozen language model can learn different computations: some widen updates to existing states, while others transform token context throughout decoder depth.We introduce Progressive Branch-Merge (PBM), which pairs each frozen decoder block with a trainable branch that attends to earlier tokens and writes the fused backbone–branch output into a shared residual state that both paths read at the next stage.After one pass through the training text in mathematics, physics, chemistry, and biology, PBM has the lowest mean next-token loss among the tested adaptations of Pythia-1B.On separately fixed fresh evaluation arrays after two passes, it also lowers next-token negative log likelihood relative to a nearly parameter-matched branch-free updater in mathematics, physics, and biology, with four-domain means of 2.0781 and 2.0847; the direct updater has lower loss in chemistry.PBM's relative gain rises with sequence difficulty estimated from thirteen other adaptations, and bypassing its trained branch blocks raises loss more on harder sequences.These results establish stagewise trainable causal processing as a useful allocation of adaptation parameters and show where that allocation yields the larger loss reduction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Frozen language models, domain adaptation, residual branches, capacity placement, parameter-matched evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xwnzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies a frozen language model adapted to a new domain with a fixed number of trainable parameters. It asks whether those parameters are better spent on adding causal token processing at every layer or on wider per-token updates to hidden states. The proposed method, Progressive Branch-Merge (PBM), places a narrow trainable causal Transformer block (width 360) beside each of the 16 frozen blocks of Pythia-1B. One gated MLP, shared across layers, merges the frozen and branch outputs back into the residual stream; in total PBM has 78.9M trainable parameters. Each domain (math, physics, chemistry, biology) is trained for one pass over 18.7M–32.0M tokens.

On held-out NLL, PBM has the lowest four-domain mean of all tested one-pass methods: 2.0441, compared with 2.0597 for the strongest parameter-matched baseline (a feed-forward parallel adapter) and 2.0715 for rank-152 LoRA. After a second pass over the same data, it also beats a parameter-matched branch-free updater in three of four domains on newly sampled sequences. The paper further reports that PBM's gain grows with sequence difficulty.

Critical strengths

S1: The paper is transparent about its limitations, and its numbers are internally consistent.

The paper openly acknowledges four limitations:

  • the main held-out arrays were accessed during exploration;
  • each configuration was trained only once;
  • matching parameters does not match FLOPs;
  • a same-stage serial variant beats PBM.

The parameter counts can be reproduced from the stated dimensions: exactly for the fusion module and the baselines, and approximately for the branch blocks, assuming a standard 4× feed-forward width that the paper does not state. This makes the paper easy to audit. However, these limitations are acknowledged rather than addressed, so this does not change my recommendation.

S2: The control experiments target the right question.

To ask whether causal token processing is worth its parameters, the right controls are a parameter-matched branch-free updater plus tokenwise and same-stage serial variants, and the paper includes all three. As discussed in W1, however, the effects these controls isolate are small (0.0007–0.0067 nats), and the same-stage serial control outperforms PBM itself.

Critical weaknesses

W1: The motivation does not hold up.

The framing. The abstract frames the question as a choice between additions that "widen updates to existing states" and additions that "transform token context throughout decoder depth". The paper's main example of the first kind is the branch-free updater, and I read adapters and LoRA as falling on the same side. This distinction is much less sharp than the framing suggests:

  • Appendix B.2 states that rank-152 LoRA is applied to four linear maps in each of the 16 blocks. A Pythia (GPT-NeoX) block has exactly four linear maps: the fused QKV projection, the attention output projection and the two MLP projections. Rank 152 on these four gives exactly the reported 79.69M parameters, so the LoRA baseline also adapts QKV and already changes how tokens attend to each other.
  • Any adapter that writes to the residual stream also changes the input of every later frozen attention layer.

The evidence. The only direct evidence that trainable token mixing matters is the tokenwise control. Replacing the branch's attention with per-token feed-forward blocks costs 0.0007 nats (Table 7), which does not suggest that missing token mixing is a bottleneck for domain adaptation. The abstract also wraps this simple question in opaque wording.

The other controls. They also suggest that the specific mechanism contributes little:

  • Letting the branch read the frozen block's output instead of its input does better than PBM (2.0992 vs. 2.1015, Table 7).
  • Two-thirds of the trainable parameters (52.5M of 78.9M) sit in the shared fusion MLP rather than in the causal branch.
  • The branch-free updater puts its entire budget into a wider fusion MLP. It is worse than PBM, but only by 0.0067 nats on average (Table 5).

W2: The comparison is unfair even against the most basic LoRA baseline, and the added parameter budget is too large.

Budget size. PBM is matched to LoRA at about 79M trainable parameters. That is roughly 8% of the 1B backbone and far beyond typical PEFT budgets.

Compute. The two are not matched in compute:

  • The 5,115-wide shared fusion MLP runs after all 16 blocks. I estimate it alone at about 0.84 GMAC per token, from (4096 × 10230 + 5115 × 2048) × 16. That is comparable to the backbone's own linear layers.
  • With the branch blocks, PBM adds about 0.89 GMAC per token, roughly 11 times the added compute of unmerged rank-152 LoRA (about 0.08 GMAC).
  • Measured training throughput is 7,210 tokens/s for PBM versus 16,257 for LoRA (Table 8).
  • Unlike LoRA, PBM cannot be merged into the weights.
  • The branch-free updater's fusion MLP costs even more, about 1.26 GMAC. Compute therefore does not explain PBM's edge over that control, but it matters for the comparison with LoRA and the adapters.

Tuning. Tuning effort is also unequal. As the paper acknowledges (Appendix A), PBM went through an optimizer-setting selection (Table 6 caption), while the baselines use fixed configurations whose search effort was not matched to PBM's. No learning rate, warmup or weight decay value is reported for any method.

Size of the gain. Against this, the gain over LoRA is 0.027 nats of mean NLL (2.0441 vs. 2.0715, Table 1), from a single training run per configuration.

W3: The training budget is too small, with fewer training tokens than trainable parameters.

Each domain has only 18.7M–32.0M training tokens, or 285–488 updates at batch size 64 (§5.1, Appendix B.4). That is fewer than PBM's 78.9M trainable parameters, or 0.24–0.41 tokens per parameter.

This is far from realistic domain-adaptive pretraining. Gururangan et al. (2020), which the paper cites for its setting, train for 12.5K steps at an effective batch size of 2048. With so few updates and unequally tuned, unreported optimizer settings, the comparison mostly measures how quickly each method improves in its first few hundred steps. The two-pass experiment only repeats the same data.

W4: The evaluation is cursory and only measures in-domain NLL.

The only quality metric is next-token NLL on held-out slices of the training corpora. There is no downstream task, no evaluation on general text, and no variance across seeds. The evaluation sets also leave open questions:

  • The main held-out arrays were accessed during exploration (Table 1 caption).
  • Table 6 reports the design path on held-out mathematics, while §5.1 says the design was selected on validation data. It is therefore unclear which split actually guided the design choices.
  • Appendix B.5 indicates that PBM, the branch-free updater and all thirteen baselines, including LoRA and the adapters, were scored on the fresh arrays. Only the PBM-versus-branch-free numbers are reported, so the paper never shows these methods side by side on one evaluation set.
Questions for authors

Q1: What were the optimization settings for each method, and on which data was PBM's design selected?

This is a clarification request. Please report the learning rate, warmup, weight decay and gradient-clipping threshold for every method, together with the search space of PBM's optimizer-setting selection. Please also clarify whether the design path was selected on validation data, as §5.1 states, given that Table 6 reports it on held-out mathematics. If tuning was unequal, the main comparison is weaker still.

Q2: What are the training and inference FLOPs per token of PBM, the branch-free updater, LoRA and the adapters?

This is a clarification request. My own estimate puts PBM's added compute at roughly 11 times that of rank-152 LoRA. If that is correct, parameter count is not a meaningful budget for this comparison.

Q3: Does PBM still win under a compute-matched and tuning-matched comparison with multiple seeds?

This is a requested experiment that could change my assessment. Please compare PBM, at the same FLOPs or wall-clock time, against:

  • LoRA at standard ranks (e.g., 16–64) and at rank 152, each with a tuned learning rate;
  • full-parameter continued pretraining.

Please use at least 3 seeds per method.

Q4: Does the advantage persist with a realistic training budget?

This is a requested experiment. Please repeat the main comparison with at least 10 times more tokens per domain, so that the number of training tokens clearly exceeds the number of trainable parameters.

Q5: How does PBM perform beyond in-domain NLL?

This is a requested experiment. Please report at least one downstream task per domain and NLL on general text. Please also report PBM, the branch-free updater, LoRA and the adapters side by side on the same evaluation set; the fresh-array scores of Appendix B.5 should already allow this.

Reviewer_NKobOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to allocate additional trainable parameters for domain-adaptive continued pretraining with a frozen language model. The proposed Progressive Branch-Merge (PBM) places a trainable causal branch alongside each frozen Transformer block, combines their outputs through a shared depth-conditioned gated module, and feeds the merged state into the next stage. Across four scientific-text domains with Pythia-1B, PBM achieves the lowest mean NLL in the one-pass comparison and subsequently outperforms a nearly parameter-matched branch-free updater in three domains. Structural controls and difficulty-stratified analyses examine the contribution of additional sequence processing.

Critical strengths
  1. The paper addresses a meaningful adaptation-design question: how different computations use the same trainable-parameter budget. The method is clearly described, including the relationships among the branches, fusion module, and shared residual state.
  2. The experiments include targeted controls, such as a parameter-matched branch-free updater, tokenwise branches, and serial causal branches. These comparisons help distinguish the contribution of additional sequence processing from the effects of specific architectural connections.
Critical weaknesses

The substantive contribution beyond prior work remains insufficiently clear. Freezing the backbone, adding trainable branches or Transformer blocks, and incorporating intermediate representations have all been explored previously. While this paper presents a specific combination of these ideas, it does not sufficiently identify a key limitation of existing approaches or establish how PBM addresses it. The current results primarily show that a particular parameter allocation can improve language-modeling loss under specific settings, without establishing a broadly applicable design principle or demonstrating a distinct capability beyond existing methods.

Questions for authors
  1. What do the authors consider the central contribution: an adaptation architecture with distinctive advantages, or an empirical finding about trainable-parameter allocation? Compared with existing adapters, side networks, and Transformer block expansion methods, what specific unresolved problem does this work address?
  2. Based on the available evidence, under which scenarios or resource constraints should PBM be preferred over existing alternatives? To what extent do the observed gains support conclusions beyond the particular backbone and training settings studied here?
Reviewer_DKPuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Progressive Branch-Merge (PBM), which places a trainable causal Transformer branch alongside each decoder block of a frozen language model. The backbone and branch receive the same residual state as input, and a shared, depth-conditioned gated fusion module combines their outputs.

Critical strengths

1. The paper presents a clear research question about parameter allocation and useful structural controls.

The paper uses a nearly parameter-matched branch-free updater and a branch that processes tokens independently to distinguish the effects of parameter allocation and trainable causal processing. The comparison with a separately trained updater is particularly valuable. A performance drop caused by removing a jointly trained component at inference time is not, by itself, sufficient to establish the superiority of that architecture. These controls also reveal cases in which PBM is not the best-performing configuration.

2. The paper interprets mixed results carefully and reports them transparently.

The authors explicitly acknowledge limitations, including the reversal in chemistry, the stronger performance of the same-stage serial causal branch, and the additional training cost. The difficulty analysis ranks sequences using losses from other models and distinguishes the extent to which the trained model relies on its branch from the advantage of a branch-based architecture over other trained architectures.

Critical weaknesses

The contribution of PBM’s distinctive shared-state architecture has not been sufficiently isolated and validated.

The mathematics comparison in Table 6 demonstrates the usefulness of refreshing the branch input from the merged state. However, the paper does not present replicated comparisons across multiple domains against an architecture that carries an independent branch state forward while keeping other conditions fixed.

In Table 7, the improvement from causal token mixing is small, and the same-stage serial causal branch, which takes the backbone output as input, achieves a lower mean NLL than PBM. The current results provide stronger support for the broad conclusion that distributing trainable causal processing across depth is useful than for the value of PBM’s particular computation graph.

Questions for authors

1. Request for additional evidence: Do the differences persist across independent training runs?

Are the differences among PBM, the branch-free updater, and the tokenwise branch consistent across different training seeds? If possible, please provide replicated results that also include the same-stage serial causal branch.

2. Clarification: How independent are the evaluation data at the document level?

How were source documents assigned to the training data and the fresh evaluation arrays? The paper confirms the absence of complete-sequence overlap but does not establish document-level separation.

Can document identifiers or data provenance information be used to determine the extent to which content from the same documents appears in both sets? If such overlap exists, a document-disjoint evaluation would strengthen the experimental evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.