acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width

Abstract

Residual connections underpin Transformers, but they implicitly tie the residual width carried through depth to the compute width used inside each layer. We introduce Hierarchical Residuals (HiRes), a minimalist architectural change that decouples these dimensions, widening the residual state while keeping per-layer compute width fixed. HiRes groups layers into blocks: standard -dimensional PreNorm layers operate inside each block, while a wider -dimensional PostNorm outer residual connects blocks. Learned read/write projections bridge the two spaces at block boundaries, leaving individual Transformer layers unchanged. Across different model sizes and depths, HiRes consistently outperforms vanilla residual addition, with fitted scaling laws indicating a improvement in FLOPs efficiency. On OLMoE-1B-7B pretraining for B tokens, HiRes achieves lower training loss and -point higher downstream accuracy. HiRes is also competitive with other residual expansion methods such as VWN, mHC and AttnRes with less activation memory and I/O costs. Built from only dense projections and normalizations, it requires neither specialized kernels nor infrastructure-level optimizations. HiRes thus exposes residual width as a practical scaling variable using only standard operators.

Primary area:
foundation or frontier models, including LLMs
Keywords:
residual connections, residual expansion, Transformers architecture, large language models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_9aVdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Typically, in Transformers, the residual width carried through depth is tied to the compute width used in each layer. This paper introduces Hierarchical Residuals (HiRes), which decouples residual width from compute width by widening the residual state from d to D dimensions while the layers themselves still compute at width d. Embed Proj maps the token embeddings into this wider D-dimensional state, and Head Proj maps it back to d before the LM head. Instead of layer-wise residuals only, the authors group layers into blocks. Instead of a single type of normalization, they use PreNorm inside each block and PostNorm across blocks, applied after each block’s output is added to the outer residual. To move between the two widths, they use a Read projection that brings the state from D down to d at the start of each block, and a Write projection that brings the block’s output from d back up to D at the end. Finally, instead of a single residual connection, the paper uses two: an inner d-dimensional residual inside each block and an outer D-dimensional residual across blocks. They show that residual regulation is provided by the boundary normalization whereas residual expansion is provided by the projection mechanism, both of which together help to improve the validation loss.

Critical strengths
  1. Easily understandable idea. Decoupling residual width from compute width using dense projections, norms and additional residual connection is easy to understand and implement.
  2. Using HiNorm helps to understand the contribution of regulation apart from residual expansion.
  3. The gains are consistent for all three models as shown in Fig. 2a and Fig. 3. The gap between baseline and HiNorm as well as gap between HiNorm and HiRes (n=2) is noticeable. It is interesting to see that HiRes (n=1) performs as close as HiNorm indicating that just hierarchy does not add much benefit.
  4. HiRes has significant improvement in activation I/O and peak memory throughout even if the accuracies are close to existing competitive methods.
Critical weaknesses
  1. Due to the use of widened projections, the number of FLOPs and the number of parameters is highest in HiRes.
  2. Authors claim that HiRes achieves competitive performance with other methods. HiRes may be competitive with techniques such as HC, mHC, VWN and better than them with respect to IO cost, HiRes still has higher FLOPs than others.
  3. While acknowledging the significant I/O gains, the average accuracies of competitive methods are reproduced by the authors. Since they are not averaged over seeds, the numbers can still fluctuate I believe. Since the numbers are very close, it is not possible to order them. However the reviewer is not questioning the improvement with respect to baseline.
Questions for authors
  1. The inclusion of figure 2 and figure 5 is very helpful to understand how the validation loss changes with training FLOPs. Can the competitors be also included in the plots to understand where the competitors stand? To be fair, better to include them in Figure 2/5.
  2. Either add seed variance or just highlight the advantage in terms of activation I/O.
  3. For ease of reading, please show the intermediate states from eq (3) to (5) in Fig. 1, e.g., h0h_0 after Embed Proj, hb−1h_{b−1} and hbh_b before and after each block, xb,0x_{b,0} after Read Proj, xb,ix_{b,i} inside the block, and xfinalx_{final} after Head Proj.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 9aVd (1/2)

We thank the reviewer for recognizing HiRes' clear formulation, HiNorm's role in separating regulation from expansion, and the consistent gains with significantly lower activation I/O and memory than competitive methods. Below we address the concerns.

W1: The number of FLOPs and the number of parameters of HiRes are higher than baseline.

A: We agree that, at a fixed backbone size, the dense projections at HiRes block boundaries add 8–9% more parameters and about 5% more training FLOPs. With this difference in mind, we account for the extra compute in our scaling experiments and include a wider-FFN baseline for a direct comparison.

(1) The scaling comparison has already included the overhead. A direct comparison at a fixed backbone size would give HiRes a larger parameter and FLOP budget. To account for this difference, we fit scaling laws across backbone sizes using total training FLOPs, including the cost of HiRes’s read and write projections (Section 3.2, Figure 2). We then compare the two fitted curves at the same validation loss. At a loss of 3.0, the vanilla baseline requires 1.30× as many training FLOPs as HiRes.

(2) We added a wider-FFN baseline that closely matches HiRes in both parameters and FLOPs in Appendix A.3 (Table 9). At (d,L)=(768,12)(d,L)=(768,12), we widened the vanilla baseline’s FFN width from 3072 to 3968, forming a wider-FFN baseline that matches HiRes in both parameters and FLOPs.

Table: Comparison with a wider-FFN baseline at comparable parameters and FLOPs

ModelResidual width DDCompute width ddFFN width dffnd_{\mathrm{ffn}}Params (M)Training FLOPs (×10¹⁹)Relative FLOPsTrain loss ↓Eval loss ↓
Vanilla baseline7687683072138.436.41861.000×3.21183.1911
Wider-FFN baseline7687683968150.826.79021.058×3.18133.1626
HiRes (n=2n=2)15367683072150.246.77271.055×3.16103.1384

With nearly identical parameters and FLOPs, HiRes still achieves 0.0203 lower train loss and 0.0242 lower validation loss than the wider-FFN baseline, suggesting that the gain goes beyond simply increasing the parameter count.

W2: HiRes has higher FLOPs than HC/mHC/VWN, although competitive in performance and better in I/O cost.

A: HiRes uses more FLOPs, but runs faster than mHC-4 and VWN-4 by reducing activation I/O. At the same backbone size and training budget, it requires fewer projected GPU-hours across all three implementations.

Table: Validation loss and resource costs at (d,L)=(768,12)(d,L)=(768,12) with 50B tokens

MethodVal. loss ↓Activation I/O ↓Peak memory (GiB) ↓FLOPs (×10¹⁹) ↓Naive PyTorch GPU-hours ↓Optimized PyTorch GPU-hours ↓Optimized Triton GPU-hours ↓
Vanilla3.19413d3d16.316.419163.0163.0163.0
mHC-43.145917d17d19.716.474454.9280.5188.3
VWN-43.122817d17d20.896.564363.2238.6199.7
Block AttnRes-43.16495.25d5.25d17.186.420292.8202.9174.0
HiRes3.13004.58d4.58d16.866.773184.0177.8177.8

I/O is per token per sublayer. Peak memory is per GPU with optimized Triton. GPU-hours are projected from end-to-end training-step throughput described in Appendix E.1–E.3 (Table 16 for optimized implementations), using the projection protocol in Appendix C.1.

Residual mixing is typically memory-bound because it repeatedly reads and writes the expanded state at every sublayer while performing relatively little computation. HiRes accesses this state only at block boundaries and applies dense projections, trading extra projection FLOPs for lower I/O and shifting most residual overhead toward compute-bound matrix multiplications. The kernel-time decomposition in Appendix E.4 (Figure 12b) supports this explanation.

W3: Accuracy results are reproduced by the authors, not averaged over seeds and can fluctuate.

A: Accuracy can indeed fluctuate. To reduce evaluation noise, we average accuracy across 12 tasks and the final three evaluations, spaced 1,000 steps apart (Section 3.1). We also report the more stable training and validation losses to complement the accuracy comparison (Tables 1 and 2). Together, these results suggest that HiRes is competitive with other residual-expansion methods across all three metrics, while offering a simpler design and lower activation I/O. We regard these methods as competitive and do not claim a reliable accuracy ranking among them. To assess variation across seeds, we additionally conducted a three-seed comparison between vanilla and HiRes, reported in Q2 below.

AuthorsOpenReview ↗

Response to Reviewer 9aVd (2/2)

Q1: Can mHC/VWN/AttnRes be included in Figure 2 (scaling over model sizes) and Figure 5 (ablations on expansion ratio and number of blocks).

A: (1) Scaling comparison (Figure 2). We extended the scaling analysis to include mHC-4, VWN-4, and Block AttnRes-4 at three shared scales, (d,L)=(768,12),(1024,16),(1280,20)(d,L)=(768,12), (1024,16), (1280,20). Figure 10 in Appendix C.1 shows the fitted curves. The table below summarizes the estimated costs of reaching a validation loss of 3.0 (Table 14), with GPU-hours projected from the throughput measurements described in Appendix E.1–E.3 (Table 16 for optimized implementations), using the projection protocol in Appendix C.1.

Table: FLOPs and projected GPU-hours to reach a matched validation loss of 3.0

MethodFLOPs (×10²⁰) ↓Naive PyTorch GPU-hours ↓Optimized PyTorch GPU-hours ↓Optimized Triton GPU-hours ↓
Vanilla1.954461461461
mHC-41.441963602395
VWN-41.349730465388
Block AttnRes-41.660761506420
HiRes1.465375363363

HiRes uses more FLOPs than mHC-4 and VWN-4 to reach the same loss, yet requires fewer projected GPU-hours across all three implementations. This is because residual mixing is typically memory-bound. HiRes trades extra projection FLOPs for lower activation I/O, decreasing the actual runtime.

(2) Hyperparameter ablations (Figure 5). Figure 5 focuses on expansion-ratio and block-count sweeps for HiRes. Table 1 already includes a limited set of ablations of competing methods, comparing 2× and 4× expansion for HC/mHC/VWN, as well as Full AttnRes and Block AttnRes-4. Repeating comparable sweeps for every competing method on our backbones would exceed our available compute budget. These hyperparameters have been studied in the original papers.

  • HC/mHC. HC studies expansion ratios in Figure 5 and Table 1. The 4× setting is also used by mHC, as specified in Table 5.
  • VWN. Figure 7 compares expansion ratios of 2×, 4×, and 8×.
  • AttnRes. Figure 6 studies the effect of block size on validation loss.

Guided by these studies, we used fixed configurations mHC-4, VWN-4, and Block AttnRes-4 for comparison with HiRes across three scales in Table 2.

Q2: Either add seed variance or just highlight the advantage in terms of activation I/O.

A: (1) Seed variance. Our study includes over 80 pretraining experiments, making multi-seed repeats of every setting costly. We therefore added a paired comparison between vanilla and HiRes across three seeds in Appendix A.4 (Table 10), to verify that seed variance is small relative to the observed loss improvements.

Table: Results across three seeds at (d,L)=(768,12)(d,L)=(768,12) with 50B tokens

SeedMethodTrain loss ↓Validation loss ↓Average accuracy (%) ↑
6198Vanilla3.21183.191146.17
HiRes3.16103.138446.92
6199Vanilla3.21213.191545.87
HiRes3.15823.130846.77
6200Vanilla3.21143.190346.65
HiRes3.15803.133346.69
Mean ± stdVanilla3.2118 ± 0.00043.1910 ± 0.000646.23 ± 0.40
HiRes3.1591 ± 0.00173.1342 ± 0.003846.79 ± 0.11

From the table, the training and validation loss improvements are substantially larger than the across-seed standard deviations. HiRes also achieves higher average accuracy in all three paired runs.

(2) Activation I/O. Table 15 compares activation I/O across optimized implementations, with detailed derivations in Appendix D. The discussions in Section 3.4 and Appendix C.1 emphasize HiRes’s main advantage over other residual-expansion methods: competitive performance with lower activation I/O and memory usage.

Q3: Show intermediate-state notation in Figure 1 (the HiRes architecture diagram).

A: Thank you for this helpful suggestion. We agree that adding intermediate-state notation would make the data flow clearer and the method easier to follow. To avoid overcrowding Figure 1, we have added this notation in Appendix Figure 6 and referenced it in the Figure 1 caption and Section 2.3.

  1. The reviewer appreciates the experiment to compare with Wider-FFN.
  2. In the table above I can see that HiRes has lesser resource cost than other benchmarks while havining similar range of val loss. The reviewer appreciates the results.
  3. I am not completely confident about averaging training loss over the final 1000 optimization steps, since I believe that the loss over final optimization steps may vary depending on the type of seed chosen. For example, just among 3 seeds, one of the seeds (6200) has similar accuracy between vanilla and HiRes.
Reviewer_ecpGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper propose HiRes to separate residual width and compute width in Transformer. It use wider outer residual and smaller inner computation, with read/write projection between blocks. Experiments show better loss and scaling. On OLMoE 500B tokens, it improve validation loss by 0.018 and downstream accuracy by 1.14 points.

Critical strengths
  1. Simple and practical architecture. HiRes separate residual width and compute width by using two-level residual structure. It only need dense projections and normalization. Original Transformer layers does not need change. This make method easy to implement and potentially useful for existing LLM training.

  2. Good experimental validation. Authors test different model sizes, depths, and MoE pretraining. HiRes consistently reduce validation loss compared with vanilla baseline. The 500B-token experiment is strong evidence for scalability. Also comparison with mHC, VWN and AttnRes show competitive performance with smaller activation memory and I/O cost.

Critical weaknesses

HiRes use extra parameters compared with vanilla and other residual methods. For example, at (768,12), HiRes have 150.2M parameters while VWN-4 only 143.3M. But VWN still achieve better validation loss. The paper have some normalization and width ablations, which is good. However, it is still not clear how much improvement really from wider residual, instead of extra parameters or better normalization. A parameter-matched wider Transformer or stronger HiNorm baseline will make comparison more convincing. Also there is no deeper explanation why wide residual can preserve more useful information.

Efficiency and result reliability still limited. The 1.30× FLOPs efficiency is based on fitted scaling law, not direct real training time. Actually HiRes only keep around 92% baseline throughput. It also not always outperform mHC or VWN in validation loss, so advantage more about memory rather than best model quality. Each configuration only run once, no variance or confidence interval. Some downstream improvements are small.

Questions for authors

Can authors compare HiRes with vanilla Transformer having same parameter count and training FLOPs? For example, increase FFN width or model width in baseline. This can help understand whether improvement is really from hierarchical residual, or just additional model capacity.

Can authors analyze what information is stored in extra residual dimensions? For example, measure representation rank, similarity, or remove some expanded channels during evaluation. If those dimensions really preserve useful information, should see clear performance drop. This may help explain actual mechanism.

The paper claim 1.30× FLOPs improvement but actual throughput is lower. Can authors report wall-clock GPU hours needed to reach same validation loss? Also compare with VWN and mHC under same compute budget. This is important for practical efficiency claim.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ecpG (1/2)

We thank the reviewer for recognizing HiRes' simplicity and practical, consistent gains across scales, and scalability to 500B-token 1B-7B MoE training, together with its low overhead. Below we address the concerns.

W1: Do the gains come from wider residual states, extra parameters, or better normalization?

A: Thank you for this insightful question. We examine these factors through the scaling comparison, component ablations, and a new parameter-matched control.

(1) The scaling comparison includes the overhead. A direct comparison at a fixed backbone size would give HiRes a larger parameter and FLOP budget. To account for this difference, we fit scaling laws across backbone sizes using total training FLOPs, including the cost of HiRes’s read and write projections (Section 3.2, Figure 2). We then compare the two fitted curves at the same validation loss. At a loss of 3.0, the vanilla baseline requires 1.30× as many training FLOPs as HiRes.

(2) The component ablations separate normalization from the wider-state architecture. Table 3 reports the following results at (d,L)=(768,12)(d,L)=(768,12) with 50B tokens:

VariantTrain loss ↓Validation loss ↓
Vanilla3.22283.1941
HiNorm3.19563.1714
HiRes (n=1n=1)3.18693.1573
HiRes (n=2n=2) without boundary RMSNorm3.18743.1552
HiRes (n=2n=2)3.16453.1300

HiNorm improves loss without widening the residual state, while HiRes still improves over vanilla without boundary RMSNorm. Increasing the residual width from n=1n=1 to n=2n=2 further improves loss. These comparisons support contributions from both normalization (residual regulation) and the wider-state architecture (residual expansion).

(3) We added a wider-FFN baseline that closely matches HiRes in both parameters and FLOPs. See Q1 below for the full comparison.

W2.1: The 1.30× FLOPs efficiency is based on fitted scaling law, not direct real training time.

A: HiRes retains near-baseline MFU. Its FLOPs advantage therefore largely translates into GPU-hour savings. At a validation loss of 3.0, the three-scale fits give 363 projected GPU-hours for HiRes versus 461 for vanilla (1.27× speedup) with optimized Triton, based on measured end-to-end training-step throughput (see Q3 for details).

W2.2: Each configuration only run once, no variance or confidence interval.

A: Our study includes over 80 pretraining experiments, making multi-seed repeats of every setting costly. We therefore added a paired comparison between vanilla and HiRes across three seeds in Appendix A.4 (Table 10), to verify that seed variance is small relative to the observed loss improvements.

Table: Results across three seeds at (d,L)=(768,12)(d,L)=(768,12) with 50B tokens

SeedMethodTrain loss ↓Validation loss ↓Average accuracy (%) ↑
6198Vanilla3.21183.191146.17
HiRes3.16103.138446.92
6199Vanilla3.21213.191545.87
HiRes3.15823.130846.77
6200Vanilla3.21143.190346.65
HiRes3.15803.133346.69
Mean ± stdVanilla3.2118 ± 0.00043.1910 ± 0.000646.23 ± 0.40
HiRes3.1591 ± 0.00173.1342 ± 0.003846.79 ± 0.11

From the table, the training and validation loss improvements are substantially larger than the across-seed standard deviations. HiRes also achieves higher average accuracy in all three paired runs.

W2.3: Some downstream improvements are small.

A: HiRes consistently improves average downstream accuracy across model scales (Table 13) and shows a sustained advantage during long-horizon pretraining (Figure 4c). On OLMoE-1B-7B trained for 500B tokens, average accuracy increases from 64.24% to 65.38%, with improvements on 9 of 12 tasks. Some individual gains are substantial: ARC-Challenge accuracy rises from 44.70% to 49.61%, an 11.0% relative improvement (+4.91 percentage points; Table 7b).

AuthorsOpenReview ↗

Response to Reviewer ecpG (2/2)

Q1: Compare HiRes with vanilla Transformer having same parameter count and training FLOPs.

A: We added a wider-FFN baseline that closely matches HiRes in both parameters and FLOPs in Appendix A.3 (Table 9). At (d,L)=(768,12)(d,L)=(768,12), we widened the vanilla baseline's FFN width from 3072 to 3968 while leaving its residual architecture unchanged.

Table: Comparison with a wider-FFN baseline at comparable parameters and FLOPs

ModelResidual width DDCompute width ddFFN width dffnd_{\mathrm{ffn}}Params (M)Training FLOPs (×10¹⁹)Relative FLOPsTrain loss ↓Eval loss ↓
Vanilla baseline7687683072138.436.41861.000×3.21183.1911
Wider-FFN baseline7687683968150.826.79021.058×3.18133.1626
HiRes (n=2n=2)15367683072150.246.77271.055×3.16103.1384

HiRes achieves 0.0203 lower train loss and 0.0242 lower validation loss than the wider-FFN baseline, suggesting that the gain goes beyond simply increasing the parameter count.

Q2: What information is stored in the additional residual dimensions?

A: Thank you for this helpful suggestion. We added an analysis of the dimensionality and predictive relevance of the expanded residual state in Appendix B, Figure 9. We examine the final checkpoints of vanilla and HiRes with n=1,2,3,4n=1,2,3,4, trained at (d,L)=(768,12)(d,L)=(768,12) for 50B tokens.

(1) PCA dimensionality. We count the principal directions retaining 99% of the centered activation variance.

Table 1: PCA dimensions retaining 99% of variance

ModelAfter layer 3After layer 6After layer 9After layer 12
Vanilla682650701713
HiRes n=1n=1690710720723
HiRes n=2n=293810568531087
HiRes n=3n=3999132412111609
HiRes n=4n=41205142516221862

Expanded models require more than 768 PCA directions at every sampled layer boundary, showing that their representations extend beyond the compute width.

(2) Loss-preserving dimension. We estimate how many leading PCA directions keep the loss increase within 0.001. Bases are fitted on separate calibration data; each intervention changes one position with fixed model weights.

Table 2: Estimated loss-preserving dimensions at ε=0.001\varepsilon=0.001

ModelAfter layer 3After layer 6After layer 9After layer 12
Vanilla760762757768
HiRes n=1n=1744750767766
HiRes n=2n=21085124313041365
HiRes n=3n=31295162318231952
HiRes n=4n=41348181422852334

For n=2n=2, preserving loss requires 1365 dimensions versus 1087 for 99% variance, showing that low-variance directions matter. The final retained fraction falls from 88.9% to 76.0% as nn increases from 2 to 4, consistent with diminishing returns.

(3) Compression to the compute width. We retain only the leading 768 PCA directions at one position at a time.

Table 3: Loss increase after compression to 768 PCA dimensions

ModelAfter layer 3After layer 6After layer 9After layer 12
HiRes n=2n=2+0.0102+0.0479+0.1332+0.7357
HiRes n=3n=3+0.0192+0.1004+0.3230+1.4396
HiRes n=4n=4+0.0252+0.1379+0.5137+1.0822

Compression becomes more harmful at later layers for all three expanded models.

These results show that the additional residual dimensions contribute to prediction.

Q3: Report wall-clock GPU hours needed to reach same validation loss. Also compare with VWN and mHC under same compute budget.

A: We added scaling fits for mHC-4, VWN-4, and Block AttnRes-4 at (d,L)=(768,12),(1024,16),(1280,20)(d,L)=(768,12), (1024,16), (1280,20) (Appendix C.1, Figure 10). The table reports estimated costs to reach validation loss 3.0 (Table 14). GPU-hours are projected from measured end-to-end training-step throughput (Appendix E.1–E.3; Table 16 for optimized implementations), following the protocol in Appendix C.1.

Table: FLOPs and projected GPU-hours to reach a matched validation loss of 3.0

MethodFLOPs (×10²⁰) ↓Naive PyTorch GPU-hours ↓Optimized PyTorch GPU-hours ↓Optimized Triton GPU-hours ↓
Vanilla1.954461461461
mHC-41.441963602395
VWN-41.349730465388
Block AttnRes-41.660761506420
HiRes1.465375363363

HiRes uses more FLOPs than mHC-4 and VWN-4 to reach the same loss, yet requires fewer projected GPU-hours across all three implementations.

This is because residual mixing is typically memory-bound, so reducing activation I/O can outweigh the cost of additional FLOPs. mHC-4 and VWN-4 read, mix, and write back the expanded state at every sublayer while performing relatively little computation along the stream dimension. HiRes addresses this by (i) using dense projections and (ii) accessing the expanded state only at block boundaries. It thus trades extra projection FLOPs for lower activation I/O, shifting most of the residual overhead toward compute-bound matrix multiplications and reducing memory traffic. The kernel-time decomposition in Appendix E.4 (Figure 12b) supports this explanation.

THank you the author for running more experiments. your new added results (espesically where does the gain come from, you have done many experiments for that and it help addressing my concerns.) and the wider-FFN baseline that closely matches HiRes is also a good fair comparison.

I have raised my small scores based on your reply. good luck my friends and wish you guys great success!

AuthorsOpenReview ↗

Thanks for your positive feedback

Thank you for reviewing the additional experiments, raising the Soundness and Informativeness ratings, and sharing such encouraging feedback! We’re glad that the new results helped address your concerns. Your suggestions, including a fairer baseline comparison and a closer analysis of the expanded residual state, helped us strengthen the paper a lot.

We believe HiRes’s clean conceptual design and simple yet effective architecture can encourage broader adoption of residual width as an independent scaling variable and provide a solid foundation for further research on residual expansion.

If there are any remaining concerns that might affect your overall assessment, we’d be happy to discuss them.

We really appreciate the time and care you’ve put into reviewing our work. All the best to you as well!

Reviewer_onEdOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The topic of the paper is hierarchical residuals (HiRes, as the paper calls). The idea is to modify the transformer architecture in a way to decouple the residual stream width (across the depth) called D from the width d of each attention/FFN sublayer. One has to be careful doing this since naively done, this will blow up the number of parameters. To mitigate this, the paper proposes to group the transformer layers into B contiguous blocks and apply the residual idea per block. At block boundaries, (learned) projection matrices help go between D and d dimensions. The paper also has additional ideas to stabilize training under this architectural modification.

The main empirical finding is that HiRes is able to obtain a 1.3x FLOP efficiency gain over vanilla baselines and can match the quality of specialized multi-stream residual methods while requiring lower activation memory and being simple to realize in practice.

Critical strengths
  • Well written paper
  • Decoupling residual and attention widths and working with blocks - are good practical deployable recipes
  • 91+% baseline throughput achieved with ~5% memory overhead; the gains look real
  • Good experimental methodology and coverage
Critical weaknesses
  • More parameters and more training FLOPs: Since HiRes uses dxD dense matrices at block boundaries, it incurs 8-9% more parameters and ~5% more training FLOPs than baseline.
Questions for authors
  • If you plot mHC-4/VWN-4/Block AttnRes-4 on Fig 2a across three scales, how would HiRes compute multiplier compare to mHC-4 and VWN-4 (at the same FLOPs)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer onEd

We thank the reviewer for recognizing HiRes’ practical design, broad experimental coverage, and performance gains achieved with near-baseline throughput and modest memory overhead. Below we address the concerns.

W1: HiRes introduces more parameters and training FLOPs than baseline.

A: We agree that, at a fixed backbone size, the dense projections at HiRes block boundaries add 8–9% more parameters and about 5% more training FLOPs. With this difference in mind, we account for the extra compute in our scaling experiments and include a wider-FFN baseline for a direct comparison.

(1) The scaling comparison accounts for the additional compute. A direct comparison at a fixed backbone size would give HiRes a larger parameter and FLOP budget. To account for this difference, we fit scaling laws across backbone sizes using total training FLOPs, including the cost of HiRes’s read and write projections (Section 3.2, Figure 2). We then compare the two fitted curves at the same validation loss. At a loss of 3.0, the vanilla baseline requires 1.30× as many training FLOPs as HiRes.

(2) We added a wider-FFN baseline that closely matches HiRes in both parameters and FLOPs in Appendix A.3 (Table 9). At (d,L)=(768,12)(d,L)=(768,12), we widened the vanilla baseline’s FFN width from 3072 to 3968, forming a wider-FFN baseline that matches HiRes in both parameters and FLOPs.

Table: Comparison with a wider-FFN baseline at comparable parameters and FLOPs

ModelResidual width DDCompute width ddFFN width dffnd_{\mathrm{ffn}}Params (M)Training FLOPs (×10¹⁹)Relative FLOPsTrain loss ↓Eval loss ↓
Vanilla baseline7687683072138.436.41861.000×3.21183.1911
Wider-FFN baseline7687683968150.826.79021.058×3.18133.1626
HiRes (n=2n=2)15367683072150.246.77271.055×3.16103.1384

With nearly identical parameters and FLOPs, HiRes still achieves 0.0203 lower train loss and 0.0242 lower validation loss than the wider-FFN baseline, suggesting that the gain goes beyond simply increasing the parameter count.

Q1: How does HiRes’s compute multiplier compare with those of mHC-4 and VWN-4?

A: Thank you for suggesting this comparison. We extended the scaling analysis in Figure 2 to compare all residual expansion methods at three shared scales: (d,L)=(768,12),(1024,16),(1280,20)(d,L)=(768, 12), (1024, 16), (1280, 20). Figure 10 in Appendix C.1 shows the fitted scaling curves. The following table reports compute multipliers at a matched validation loss of 3.0, defined as vanilla’s cost divided by each method’s cost. We report multipliers in both training FLOPs and projected GPU-hours, with GPU-hours estimated from the throughput measurements described in Appendix E.1–E.3 (Table 16 for optimized implementations), using the projection protocol in Appendix C.1.

Table: FLOPs and projected GPU-hour speedup at matched loss of 3.0

MethodFLOPs multiplier ↑Naive PyTorch GPU-hour multiplier ↑Optimized PyTorch GPU-hour multiplier ↑Optimized Triton GPU-hour multiplier ↑
Vanilla1.000×1.000×1.000×1.000×
mHC-41.356×0.479×0.765×1.165×
VWN-41.448×0.631×0.992×1.187×
Block AttnRes-41.177×0.606×0.911×1.098×
HiRes1.334×1.230×1.269×1.270×

As shown in the table, HiRes uses more FLOPs than mHC-4 and VWN-4 to reach the same loss, yet requires fewer GPU-hours across all three implementations, including both naive and optimized versions.

This is because residual mixing is typically memory-bound, so reducing activation I/O can outweigh the cost of additional FLOPs. mHC-4 and VWN-4 read, mix, and write back the expanded state at every sublayer while performing relatively little computation along the stream dimension. HiRes addresses this by (i) using dense projections and (ii) accessing the expanded state only at block boundaries. It thus trades extra projection FLOPs for lower activation I/O, shifting most of the residual overhead toward compute-bound matrix multiplications and reducing memory traffic. The kernel-time decomposition in Appendix E.4 (Figure 12b) supports this explanation.

Reviewer_onEdOpenReview ↗

Response

Thank you for the additional experiments and numbers for the additional comparison. I will keep my score.

AuthorsOpenReview ↗

Thank you for your feedback and support

Thank you for reviewing the additional results, and for your continued support! Your feedback helped us clarify the benefits of HiRes.

We believe HiRes’s clean conceptual design and simple yet effective architecture can encourage broader adoption of residual width as an independent scaling variable and provide a solid foundation for further research on residual expansion.

We really appreciate the time and care you’ve put into reviewing our work. Best wishes!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the Area Chair and all three reviewers for their time and engagement throughout the review process. All three reviewers recommended acceptance in their initial reviews, with overall ratings of 4 (onEd), 3 (ecpG), and 3 (9aVd). The scores remained unchanged during the discussion, while ecpG raised both Soundness and Informativeness from 2 to 3.

Strengths highlighted by the reviewers

  1. Simple, practical design (onEd, ecpG, 9aVd). onEd praised the clear presentation and practical block design. ecpG and 9aVd highlighted the ease of implementation, with ecpG noting that HiRes leaves the original Transformer layers unchanged.

  2. Experimental methodology and consistent gains (onEd, ecpG, 9aVd). onEd praised the experimental methodology and coverage. ecpG and 9aVd noted consistent gains across model scales, and ecpG highlighted the 500B-token 7B-parameter MoE pretraining as evidence of scalability.

  3. Training efficiency in practice (onEd, ecpG, 9aVd). onEd noted that HiRes retains at least 91% of baseline throughput with only about 5% memory overhead. Compared with other residual-expansion methods, ecpG and 9aVd highlighted HiRes’s competitive performance, lower activation I/O, and lower memory use.

  4. Insight from the HiNorm ablation (9aVd). 9aVd highlighted HiNorm as a useful control for distinguishing the contribution of residual regulation from that of residual expansion.

Main questions and revisions

  1. Do the gains come from extra parameters or FLOPs? (onEd, ecpG, 9aVd) The original scaling comparison includes the cost of HiRes’s projections and compares total training FLOPs at matched validation loss (Figure 2). For a direct comparison, we also widened the FFN of a vanilla Transformer to closely match HiRes in both parameters and FLOPs. After 50B tokens, HiRes achieves lower validation loss than this wider-FFN baseline (3.1384 vs. 3.1626, Table 9).

  2. How does HiRes compare with other residual-expansion methods in practice? (onEd, ecpG, 9aVd) We extended the scaling comparison to mHC-4, VWN-4, and Block AttnRes-4 at three shared model scales (Figure 10). HiRes uses more FLOPs than mHC-4 and VWN-4 at matched loss, but reduces activation I/O by accessing the expanded state only at block boundaries. Based on measured end-to-end training-step throughput, we project that HiRes needs the fewest GPU-hours to reach a validation loss of 3.0 across naive Python and optimized Triton implementations (Table 14).

  3. Do the gains hold across different seeds? (ecpG, 9aVd) We added a paired comparison of HiRes and vanilla using three seeds (Table 10). Across seeds, HiRes reduced training loss by 0.0527 ± 0.0017 and validation loss by 0.0568 ± 0.0040, and improved average downstream accuracy by 0.56 ± 0.46 percentage points. All three metrics favored HiRes in every pair.

  4. Does HiRes use the additional residual dimensions? (ecpG) We added PCA and compression experiments on the trained models (Figure 9). For HiRes with n=2n=2, using PCA to compress the final residual state from 1536 dimensions to the compute width of 768 increases loss by 0.7357, indicating that the discarded directions contribute to prediction.

We sincerely appreciate the reviewers’ suggestions. We believe HiRes’s clean conceptual design and simple yet effective architecture can encourage broader adoption of residual width as an independent scaling variable and provide a solid foundation for further research on residual expansion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.