acceptodds
Under review as a conference paper at ICLR 2027

ReHiGate: Function-Preserving Capacity Augmentation via History-Conditioned Residual Control

Abstract

Large-scale pre-training is expensive, motivating approaches that reuse partially trained models rather than train every target model from scratch. Function-preserving model growth offers a natural solution by expanding the backbone while keeping its function unchanged at insertion. However, forward preservation alone does not guarantee that the added capacity provides useful optimization directions. We propose **ReHiGate** (*Residual History Gating*), which keeps the backbone intact and instead introduces a history-conditioned residual controller. ReHiGate reads a sparse set of preceding-layer states and predicts token- and channel-wise gates for the attention and MLP residual branches, using cross-layer history as control rather than directly mixing it into the representation stream. Its identity-centered initialization exactly recovers the inherited computation while retaining active first-order controller directions, and insertion-time functional diagnostics show that these directions are less aligned with the trained backbone than those introduced by parameter-matched width and depth growth. We evaluate ReHiGate on 124M- and 1.5B-parameter Transformers with continuation endpoints at 10B and 100B tokens, respectively. Across both scales, ReHiGate outperforms plain continuation and parameter-matched width and depth growth at every evaluated insertion point, reducing validation loss by up to 0.044 and 0.017 nats.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model growth, continued pre-training, function-preserving augmentation, residual gating, cross-layer representations, optimization dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_k2rWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes ReHiGate, a way to add capacity to a partially trained Transformer without widening or deepening it. For each block, a low-rank controller reads a stride-4 set of preceding residual-stream states and predicts token- and channel-wise multiplicative gates for the attention and MLP branches. The gates are initialised at exactly one, so the inherited function is preserved at insertion while the controller still receives gradients. On 124M and 1.5B GPT-style models, inserting ReHiGate during pre-training gives a lower final validation loss than plain continuation (by up to 0.044 and 0.017 nats, respectively) and than parameter-matched width and depth growth. Validation consists of single-run validation-loss comparisons, ablations at 124M, and insertion-time alignment diagnostics.

Critical strengths
  • Clear paper and simple method. The paper is easy to read, and the method can be understood in one pass. This makes the claims easy to verify and the method easy to reimplement.
Critical weaknesses
  • Experiments do not support the claims. There is no model trained from scratch at the target scale, with or without the ReHiGate. We see the gain being largest at the first insertion point (table 1), so the benefit from the insertion cannot be separated with that of a larger architecture with more training. The role of history is also not isolated, there is no gate conditioned on x_l only, all history patterns in table 4 are very close to each other and the cross layer methods discussed in the related work (DenseFormer, MUDDFormer, Hyper-connections, LAuReL) are not evaluated. The central claim, that using history as a control signal is preferable to mixing it into the representation, is therefore not tested. Addressing this requires more experiments than the paper currently contains, so I do not see it as a minor fix.
  • The reported margins may be withing noise. No seeds are being reported and there is no description of the error bars in Fig. 2 and Fig. 3(b). In the rank sweep (Table 3), the loss varies by 0.023 between neighbouring settings (3.047 at r=256, 3.070 at r=384, 3.049 for the full projection), which is larger than the margin over the growth baselines (0.003–0.020). In addition, validation loss is the only metric and is used both to select the reference configuration among about 20 variants (Tables 3 and 4) and to report the result, without a separate test split.
Questions for authors

Clarifications:

  1. How many seeds were run, and what do the error bars in Fig. 2(a) and Fig. 3(b) and the shaded regions in Fig. 2(b–d) represent?
  2. How is the validation split constructed, how large is it, and is it disjoint from the training subset?
  3. Were the learning rate and warmup of the newly added parameters tuned for the growth baselines as they were for the controller (Table 6)?
  4. At 1.5B scale, Eq. 7 with stride-4 history, rank 256 and 52 layers gives roughly 327M added parameters by my count (the U factors alone are about 286M), whereas 145.6M is reported. Is the history truncated at this scale, and if so how?
  5. Which activation, MLP expansion ratio and normalisation does the backbone use? The paper only says "GPT-style". If the MLP is not gated (e.g. GELU), part of the gain may come from adding input-dependent multiplicative gating in general, which gated MLPs such as SwiGLU already provide. Does the gain persist on a backbone with a gated MLP?

Additional results that would affect my assessment:

  1. How does ReHiGate trained from scratch compare with a 14-layer or d=816 model trained from scratch for 10B tokens, and with the inserted variants in Table 1?
  2. What is the loss when the gate is conditioned on x_l only, and when it is a static learned per-channel scale?
  3. How does ReHiGate compare with at least the closest cross-layer methods from the related work under the same budget?

Suggestions for future work (not needed for my assessment): statistics of the learned gate values, which are bounded in (0.5, 1.5); a downstream metric in addition to validation loss; a code release.

Reviewer_JeFAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReHiGate augments a partially trained Transformer with low-rank controllers that read the current state and selected earlier-layer states. These controllers produce token- and channel-wise gates for attention and MLP contributions. Zero-initializing the output factors makes insertion function-preserving while allowing immediate controller updates. Continuation experiments at 124M and 1.5B scales compare the method with plain continuation and width/depth growth, supplemented by architectural ablations and functional tangent diagnostics.

Critical strengths
  1. The continuation results are encouraging across a substantial range of insertion times. At 124M, ReHiGate beats both growth baselines at all eight insertion points; the larger-model results show the same ordering. This is more convincing than a gain obtained at one carefully chosen checkpoint and suggests that the intervention can be useful at different training stages.

  2. The insertion procedure is clean and preserves the inherited optimizer state and schedule. The controller starts at the identity, but its output factor can receive gradients immediately. This provides a sensible way to introduce additional capacity without disrupting the existing computation or restarting backbone optimization.

Critical weaknesses
  1. I could reproduce the reported controller parameter count at 124M, but not at 1.5B. With independent attention and MLP controllers, Eq. (7) gives 2rd*sum_l(k_l+1) parameters, plus biases. For 52 layers, d=1536, r=256, and the stride-4 history in Eq. (5), sum_l(k_l)=364, giving approximately 327.3M additional parameters rather than 145.6M. Tables 5–6 do not mention a history cap or factor sharing that would explain this. This matters because the larger-model comparison is presented as parameter-matched; an undocumented configuration or an incorrect count changes how I interpret the central result.

  2. The experiments do not directly establish that cross-layer history is responsible for the improvement. Every history ablation still reads preceding states: even window-1 includes the previous layer. A controller using only the current state, with comparable capacity and the same initialization and optimization, seems the essential comparison here. The layer-swap and spectral analyses describe redundancy, but do not show that historical states provide information needed by the gate. Likewise, lower tangent alignment at insertion does not establish that history causes the better continuation performance. Without this control, I am unsure how much the paper adds beyond introducing an identity-initialized residual controller.

Questions for authors
  1. Could you reconcile the 1.5B controller count with Eqs. (5) and (7)? Please specify the actual history sources, factor shapes, rank, and any parameter sharing used at this scale. If the implemented parameter budget differs from the reported one, how does the comparison with the width/depth baselines change?

  2. How does a current-state-only controller perform under the same continuation protocol? I would find both a same-rank comparison and a comparison matching the added-parameter budget useful. A clear advantage over this control would substantially strengthen the case for history-conditioned augmentation.

Reviewer_Kr3jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReHiGate, a method for expanding the capacity of partially pretrained Transformer models without modifying the existing backbone. It introduces lightweight controllers that use hidden states from previous layers to generate token-wise and channel-wise gates for attention and MLP residual outputs.

The controllers are initialized to exactly preserve the original model's function while allowing immediate optimization. Empirical experiments on LLMs with up to 124M and 1.5B parameters show consistent improvements over continued pretraining and parameter-matched width/depth expansion methods.

Critical strengths
  1. Empirical results show clear improvements: The method outperforms the evaluated baselines across multiple insertion points and two model scales.
  2. The authors have conducted relatively many ablation studies to verify the various design choices of the proposed method.
Critical weaknesses
  1. Limited novelty: Residual gating, cross-layer connections, and function-preserving initialization are established techniques. The main contribution is primarily their combination.
  2. Unclear computational efficiency: The method seems to introduce cross-layer dependencies, but does not report training throughput, FLOPs, or inference overhead, which I think is very important. The authors should also discuss whether this method is compatible with pipeline parallelism training strategy for large-scale LLM training.
  3. Experiments use a training context length of 1024, which is way to short compared to widely-used configurations.
  4. The authors do not report downstream task results, which is arguably more important than language modeling loss.
  5. Missing key ablation: The paper does not directly compare history-conditioned gating against gating based only on the current hidden state, making it difficult to establish the benefit of cross-layer history.
Questions for authors
  • Some equations are not consistent. E.g., Equation 1 uses "Attn" while Equation 3 uses "AttnMix".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.