- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes ReHiGate, a way to add capacity to a partially trained Transformer without widening or deepening it. For each block, a low-rank controller reads a stride-4 set of preceding residual-stream states and predicts token- and channel-wise multiplicative gates for the attention and MLP branches. The gates are initialised at exactly one, so the inherited function is preserved at insertion while the controller still receives gradients. On 124M and 1.5B GPT-style models, inserting ReHiGate during pre-training gives a lower final validation loss than plain continuation (by up to 0.044 and 0.017 nats, respectively) and than parameter-matched width and depth growth. Validation consists of single-run validation-loss comparisons, ablations at 124M, and insertion-time alignment diagnostics.
Critical strengths
- Clear paper and simple method. The paper is easy to read, and the method can be understood in one pass. This makes the claims easy to verify and the method easy to reimplement.
Critical weaknesses
- Experiments do not support the claims. There is no model trained from scratch at the target scale, with or without the ReHiGate. We see the gain being largest at the first insertion point (table 1), so the benefit from the insertion cannot be separated with that of a larger architecture with more training. The role of history is also not isolated, there is no gate conditioned on x_l only, all history patterns in table 4 are very close to each other and the cross layer methods discussed in the related work (DenseFormer, MUDDFormer, Hyper-connections, LAuReL) are not evaluated. The central claim, that using history as a control signal is preferable to mixing it into the representation, is therefore not tested. Addressing this requires more experiments than the paper currently contains, so I do not see it as a minor fix.
- The reported margins may be withing noise. No seeds are being reported and there is no description of the error bars in Fig. 2 and Fig. 3(b). In the rank sweep (Table 3), the loss varies by 0.023 between neighbouring settings (3.047 at r=256, 3.070 at r=384, 3.049 for the full projection), which is larger than the margin over the growth baselines (0.003–0.020). In addition, validation loss is the only metric and is used both to select the reference configuration among about 20 variants (Tables 3 and 4) and to report the result, without a separate test split.
Questions for authors
Clarifications:
- How many seeds were run, and what do the error bars in Fig. 2(a) and Fig. 3(b) and the shaded regions in Fig. 2(b–d) represent?
- How is the validation split constructed, how large is it, and is it disjoint from the training subset?
- Were the learning rate and warmup of the newly added parameters tuned for the growth baselines as they were for the controller (Table 6)?
- At 1.5B scale, Eq. 7 with stride-4 history, rank 256 and 52 layers gives roughly 327M added parameters by my count (the U factors alone are about 286M), whereas 145.6M is reported. Is the history truncated at this scale, and if so how?
- Which activation, MLP expansion ratio and normalisation does the backbone use? The paper only says "GPT-style". If the MLP is not gated (e.g. GELU), part of the gain may come from adding input-dependent multiplicative gating in general, which gated MLPs such as SwiGLU already provide. Does the gain persist on a backbone with a gated MLP?
Additional results that would affect my assessment:
- How does ReHiGate trained from scratch compare with a 14-layer or d=816 model trained from scratch for 10B tokens, and with the inserted variants in Table 1?
- What is the loss when the gate is conditioned on x_l only, and when it is a static learned per-channel scale?
- How does ReHiGate compare with at least the closest cross-layer methods from the related work under the same budget?
Suggestions for future work (not needed for my assessment): statistics of the learned gate values, which are bounded in (0.5, 1.5); a downstream metric in addition to validation loss; a code release.