- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies a frozen language model adapted to a new domain with a fixed number of trainable parameters. It asks whether those parameters are better spent on adding causal token processing at every layer or on wider per-token updates to hidden states. The proposed method, Progressive Branch-Merge (PBM), places a narrow trainable causal Transformer block (width 360) beside each of the 16 frozen blocks of Pythia-1B. One gated MLP, shared across layers, merges the frozen and branch outputs back into the residual stream; in total PBM has 78.9M trainable parameters. Each domain (math, physics, chemistry, biology) is trained for one pass over 18.7M–32.0M tokens.
On held-out NLL, PBM has the lowest four-domain mean of all tested one-pass methods: 2.0441, compared with 2.0597 for the strongest parameter-matched baseline (a feed-forward parallel adapter) and 2.0715 for rank-152 LoRA. After a second pass over the same data, it also beats a parameter-matched branch-free updater in three of four domains on newly sampled sequences. The paper further reports that PBM's gain grows with sequence difficulty.
Critical strengths
S1: The paper is transparent about its limitations, and its numbers are internally consistent.
The paper openly acknowledges four limitations:
- the main held-out arrays were accessed during exploration;
- each configuration was trained only once;
- matching parameters does not match FLOPs;
- a same-stage serial variant beats PBM.
The parameter counts can be reproduced from the stated dimensions: exactly for the fusion module and the baselines, and approximately for the branch blocks, assuming a standard 4× feed-forward width that the paper does not state. This makes the paper easy to audit. However, these limitations are acknowledged rather than addressed, so this does not change my recommendation.
S2: The control experiments target the right question.
To ask whether causal token processing is worth its parameters, the right controls are a parameter-matched branch-free updater plus tokenwise and same-stage serial variants, and the paper includes all three. As discussed in W1, however, the effects these controls isolate are small (0.0007–0.0067 nats), and the same-stage serial control outperforms PBM itself.
Critical weaknesses
W1: The motivation does not hold up.
The framing. The abstract frames the question as a choice between additions that "widen updates to existing states" and additions that "transform token context throughout decoder depth". The paper's main example of the first kind is the branch-free updater, and I read adapters and LoRA as falling on the same side. This distinction is much less sharp than the framing suggests:
- Appendix B.2 states that rank-152 LoRA is applied to four linear maps in each of the 16 blocks. A Pythia (GPT-NeoX) block has exactly four linear maps: the fused QKV projection, the attention output projection and the two MLP projections. Rank 152 on these four gives exactly the reported 79.69M parameters, so the LoRA baseline also adapts QKV and already changes how tokens attend to each other.
- Any adapter that writes to the residual stream also changes the input of every later frozen attention layer.
The evidence. The only direct evidence that trainable token mixing matters is the tokenwise control. Replacing the branch's attention with per-token feed-forward blocks costs 0.0007 nats (Table 7), which does not suggest that missing token mixing is a bottleneck for domain adaptation. The abstract also wraps this simple question in opaque wording.
The other controls. They also suggest that the specific mechanism contributes little:
- Letting the branch read the frozen block's output instead of its input does better than PBM (2.0992 vs. 2.1015, Table 7).
- Two-thirds of the trainable parameters (52.5M of 78.9M) sit in the shared fusion MLP rather than in the causal branch.
- The branch-free updater puts its entire budget into a wider fusion MLP. It is worse than PBM, but only by 0.0067 nats on average (Table 5).
W2: The comparison is unfair even against the most basic LoRA baseline, and the added parameter budget is too large.
Budget size. PBM is matched to LoRA at about 79M trainable parameters. That is roughly 8% of the 1B backbone and far beyond typical PEFT budgets.
Compute. The two are not matched in compute:
- The 5,115-wide shared fusion MLP runs after all 16 blocks. I estimate it alone at about 0.84 GMAC per token, from (4096 × 10230 + 5115 × 2048) × 16. That is comparable to the backbone's own linear layers.
- With the branch blocks, PBM adds about 0.89 GMAC per token, roughly 11 times the added compute of unmerged rank-152 LoRA (about 0.08 GMAC).
- Measured training throughput is 7,210 tokens/s for PBM versus 16,257 for LoRA (Table 8).
- Unlike LoRA, PBM cannot be merged into the weights.
- The branch-free updater's fusion MLP costs even more, about 1.26 GMAC. Compute therefore does not explain PBM's edge over that control, but it matters for the comparison with LoRA and the adapters.
Tuning. Tuning effort is also unequal. As the paper acknowledges (Appendix A), PBM went through an optimizer-setting selection (Table 6 caption), while the baselines use fixed configurations whose search effort was not matched to PBM's. No learning rate, warmup or weight decay value is reported for any method.
Size of the gain. Against this, the gain over LoRA is 0.027 nats of mean NLL (2.0441 vs. 2.0715, Table 1), from a single training run per configuration.
W3: The training budget is too small, with fewer training tokens than trainable parameters.
Each domain has only 18.7M–32.0M training tokens, or 285–488 updates at batch size 64 (§5.1, Appendix B.4). That is fewer than PBM's 78.9M trainable parameters, or 0.24–0.41 tokens per parameter.
This is far from realistic domain-adaptive pretraining. Gururangan et al. (2020), which the paper cites for its setting, train for 12.5K steps at an effective batch size of 2048. With so few updates and unequally tuned, unreported optimizer settings, the comparison mostly measures how quickly each method improves in its first few hundred steps. The two-pass experiment only repeats the same data.
W4: The evaluation is cursory and only measures in-domain NLL.
The only quality metric is next-token NLL on held-out slices of the training corpora. There is no downstream task, no evaluation on general text, and no variance across seeds. The evaluation sets also leave open questions:
- The main held-out arrays were accessed during exploration (Table 1 caption).
- Table 6 reports the design path on held-out mathematics, while §5.1 says the design was selected on validation data. It is therefore unclear which split actually guided the design choices.
- Appendix B.5 indicates that PBM, the branch-free updater and all thirteen baselines, including LoRA and the adapters, were scored on the fresh arrays. Only the PBM-versus-branch-free numbers are reported, so the paper never shows these methods side by side on one evaluation set.
Questions for authors
Q1: What were the optimization settings for each method, and on which data was PBM's design selected?
This is a clarification request. Please report the learning rate, warmup, weight decay and gradient-clipping threshold for every method, together with the search space of PBM's optimizer-setting selection. Please also clarify whether the design path was selected on validation data, as §5.1 states, given that Table 6 reports it on held-out mathematics. If tuning was unequal, the main comparison is weaker still.
Q2: What are the training and inference FLOPs per token of PBM, the branch-free updater, LoRA and the adapters?
This is a clarification request. My own estimate puts PBM's added compute at roughly 11 times that of rank-152 LoRA. If that is correct, parameter count is not a meaningful budget for this comparison.
Q3: Does PBM still win under a compute-matched and tuning-matched comparison with multiple seeds?
This is a requested experiment that could change my assessment. Please compare PBM, at the same FLOPs or wall-clock time, against:
- LoRA at standard ranks (e.g., 16–64) and at rank 152, each with a tuned learning rate;
- full-parameter continued pretraining.
Please use at least 3 seeds per method.
Q4: Does the advantage persist with a realistic training budget?
This is a requested experiment. Please repeat the main comparison with at least 10 times more tokens per domain, so that the number of training tokens clearly exceeds the number of trainable parameters.
Q5: How does PBM perform beyond in-domain NLL?
This is a requested experiment. Please report at least one downstream task per domain and NLL on general text. Please also report PBM, the branch-free updater, LoRA and the adapters side by side on the same evaluation set; the fresh-array scores of Appendix B.5 should already allow this.