- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Typically, in Transformers, the residual width carried through depth is tied to the compute width used in each layer. This paper introduces Hierarchical Residuals (HiRes), which decouples residual width from compute width by widening the residual state from d to D dimensions while the layers themselves still compute at width d. Embed Proj maps the token embeddings into this wider D-dimensional state, and Head Proj maps it back to d before the LM head. Instead of layer-wise residuals only, the authors group layers into blocks. Instead of a single type of normalization, they use PreNorm inside each block and PostNorm across blocks, applied after each block’s output is added to the outer residual. To move between the two widths, they use a Read projection that brings the state from D down to d at the start of each block, and a Write projection that brings the block’s output from d back up to D at the end. Finally, instead of a single residual connection, the paper uses two: an inner d-dimensional residual inside each block and an outer D-dimensional residual across blocks. They show that residual regulation is provided by the boundary normalization whereas residual expansion is provided by the projection mechanism, both of which together help to improve the validation loss.
Critical strengths
- Easily understandable idea. Decoupling residual width from compute width using dense projections, norms and additional residual connection is easy to understand and implement.
- Using HiNorm helps to understand the contribution of regulation apart from residual expansion.
- The gains are consistent for all three models as shown in Fig. 2a and Fig. 3. The gap between baseline and HiNorm as well as gap between HiNorm and HiRes (n=2) is noticeable. It is interesting to see that HiRes (n=1) performs as close as HiNorm indicating that just hierarchy does not add much benefit.
- HiRes has significant improvement in activation I/O and peak memory throughout even if the accuracies are close to existing competitive methods.
Critical weaknesses
- Due to the use of widened projections, the number of FLOPs and the number of parameters is highest in HiRes.
- Authors claim that HiRes achieves competitive performance with other methods. HiRes may be competitive with techniques such as HC, mHC, VWN and better than them with respect to IO cost, HiRes still has higher FLOPs than others.
- While acknowledging the significant I/O gains, the average accuracies of competitive methods are reproduced by the authors. Since they are not averaged over seeds, the numbers can still fluctuate I believe. Since the numbers are very close, it is not possible to order them. However the reviewer is not questioning the improvement with respect to baseline.
Questions for authors
- The inclusion of figure 2 and figure 5 is very helpful to understand how the validation loss changes with training FLOPs. Can the competitors be also included in the plots to understand where the competitors stand? To be fair, better to include them in Figure 2/5.
- Either add seed variance or just highlight the advantage in terms of activation I/O.
- For ease of reading, please show the intermediate states from eq (3) to (5) in Fig. 1, e.g., after Embed Proj, and before and after each block, after Read Proj, inside the block, and after Head Proj.
6 comments
Response to Reviewer 9aVd (2/2)
Q1: Can mHC/VWN/AttnRes be included in Figure 2 (scaling over model sizes) and Figure 5 (ablations on expansion ratio and number of blocks).
A: (1) Scaling comparison (Figure 2). We extended the scaling analysis to include mHC-4, VWN-4, and Block AttnRes-4 at three shared scales, . Figure 10 in Appendix C.1 shows the fitted curves. The table below summarizes the estimated costs of reaching a validation loss of 3.0 (Table 14), with GPU-hours projected from the throughput measurements described in Appendix E.1–E.3 (Table 16 for optimized implementations), using the projection protocol in Appendix C.1.
Table: FLOPs and projected GPU-hours to reach a matched validation loss of 3.0
| Method | FLOPs (×10²⁰) ↓ | Naive PyTorch GPU-hours ↓ | Optimized PyTorch GPU-hours ↓ | Optimized Triton GPU-hours ↓ |
|---|---|---|---|---|
| Vanilla | 1.954 | 461 | 461 | 461 |
| mHC-4 | 1.441 | 963 | 602 | 395 |
| VWN-4 | 1.349 | 730 | 465 | 388 |
| Block AttnRes-4 | 1.660 | 761 | 506 | 420 |
| HiRes | 1.465 | 375 | 363 | 363 |
HiRes uses more FLOPs than mHC-4 and VWN-4 to reach the same loss, yet requires fewer projected GPU-hours across all three implementations. This is because residual mixing is typically memory-bound. HiRes trades extra projection FLOPs for lower activation I/O, decreasing the actual runtime.
(2) Hyperparameter ablations (Figure 5). Figure 5 focuses on expansion-ratio and block-count sweeps for HiRes. Table 1 already includes a limited set of ablations of competing methods, comparing 2× and 4× expansion for HC/mHC/VWN, as well as Full AttnRes and Block AttnRes-4. Repeating comparable sweeps for every competing method on our backbones would exceed our available compute budget. These hyperparameters have been studied in the original papers.
- HC/mHC. HC studies expansion ratios in Figure 5 and Table 1. The 4× setting is also used by mHC, as specified in Table 5.
- VWN. Figure 7 compares expansion ratios of 2×, 4×, and 8×.
- AttnRes. Figure 6 studies the effect of block size on validation loss.
Guided by these studies, we used fixed configurations mHC-4, VWN-4, and Block AttnRes-4 for comparison with HiRes across three scales in Table 2.
Q2: Either add seed variance or just highlight the advantage in terms of activation I/O.
A: (1) Seed variance. Our study includes over 80 pretraining experiments, making multi-seed repeats of every setting costly. We therefore added a paired comparison between vanilla and HiRes across three seeds in Appendix A.4 (Table 10), to verify that seed variance is small relative to the observed loss improvements.
Table: Results across three seeds at with 50B tokens
| Seed | Method | Train loss ↓ | Validation loss ↓ | Average accuracy (%) ↑ |
|---|---|---|---|---|
| 6198 | Vanilla | 3.2118 | 3.1911 | 46.17 |
| HiRes | 3.1610 | 3.1384 | 46.92 | |
| 6199 | Vanilla | 3.2121 | 3.1915 | 45.87 |
| HiRes | 3.1582 | 3.1308 | 46.77 | |
| 6200 | Vanilla | 3.2114 | 3.1903 | 46.65 |
| HiRes | 3.1580 | 3.1333 | 46.69 | |
| Mean ± std | Vanilla | 3.2118 ± 0.0004 | 3.1910 ± 0.0006 | 46.23 ± 0.40 |
| HiRes | 3.1591 ± 0.0017 | 3.1342 ± 0.0038 | 46.79 ± 0.11 |
From the table, the training and validation loss improvements are substantially larger than the across-seed standard deviations. HiRes also achieves higher average accuracy in all three paired runs.
(2) Activation I/O. Table 15 compares activation I/O across optimized implementations, with detailed derivations in Appendix D. The discussions in Section 3.4 and Appendix C.1 emphasize HiRes’s main advantage over other residual-expansion methods: competitive performance with lower activation I/O and memory usage.
Q3: Show intermediate-state notation in Figure 1 (the HiRes architecture diagram).
A: Thank you for this helpful suggestion. We agree that adding intermediate-state notation would make the data flow clearer and the method easier to follow. To avoid overcrowding Figure 1, we have added this notation in Appendix Figure 6 and referenced it in the Figure 1 caption and Section 2.3.
- The reviewer appreciates the experiment to compare with Wider-FFN.
- In the table above I can see that HiRes has lesser resource cost than other benchmarks while havining similar range of val loss. The reviewer appreciates the results.
- I am not completely confident about averaging training loss over the final 1000 optimization steps, since I believe that the loss over final optimization steps may vary depending on the type of seed chosen. For example, just among 3 seeds, one of the seeds (6200) has similar accuracy between vanilla and HiRes.
Response to Reviewer 9aVd (1/2)
We thank the reviewer for recognizing HiRes' clear formulation, HiNorm's role in separating regulation from expansion, and the consistent gains with significantly lower activation I/O and memory than competitive methods. Below we address the concerns.
W1: The number of FLOPs and the number of parameters of HiRes are higher than baseline.
A: We agree that, at a fixed backbone size, the dense projections at HiRes block boundaries add 8–9% more parameters and about 5% more training FLOPs. With this difference in mind, we account for the extra compute in our scaling experiments and include a wider-FFN baseline for a direct comparison.
(1) The scaling comparison has already included the overhead. A direct comparison at a fixed backbone size would give HiRes a larger parameter and FLOP budget. To account for this difference, we fit scaling laws across backbone sizes using total training FLOPs, including the cost of HiRes’s read and write projections (Section 3.2, Figure 2). We then compare the two fitted curves at the same validation loss. At a loss of 3.0, the vanilla baseline requires 1.30× as many training FLOPs as HiRes.
(2) We added a wider-FFN baseline that closely matches HiRes in both parameters and FLOPs in Appendix A.3 (Table 9). At (d,L)=(768,12), we widened the vanilla baseline’s FFN width from 3072 to 3968, forming a wider-FFN baseline that matches HiRes in both parameters and FLOPs.
Table: Comparison with a wider-FFN baseline at comparable parameters and FLOPs
With nearly identical parameters and FLOPs, HiRes still achieves 0.0203 lower train loss and 0.0242 lower validation loss than the wider-FFN baseline, suggesting that the gain goes beyond simply increasing the parameter count.
W2: HiRes has higher FLOPs than HC/mHC/VWN, although competitive in performance and better in I/O cost.
A: HiRes uses more FLOPs, but runs faster than mHC-4 and VWN-4 by reducing activation I/O. At the same backbone size and training budget, it requires fewer projected GPU-hours across all three implementations.
Table: Validation loss and resource costs at (d,L)=(768,12) with 50B tokens
I/O is per token per sublayer. Peak memory is per GPU with optimized Triton. GPU-hours are projected from end-to-end training-step throughput described in Appendix E.1–E.3 (Table 16 for optimized implementations), using the projection protocol in Appendix C.1.
Residual mixing is typically memory-bound because it repeatedly reads and writes the expanded state at every sublayer while performing relatively little computation. HiRes accesses this state only at block boundaries and applies dense projections, trading extra projection FLOPs for lower I/O and shifting most residual overhead toward compute-bound matrix multiplications. The kernel-time decomposition in Appendix E.4 (Figure 12b) supports this explanation.
W3: Accuracy results are reproduced by the authors, not averaged over seeds and can fluctuate.
A: Accuracy can indeed fluctuate. To reduce evaluation noise, we average accuracy across 12 tasks and the final three evaluations, spaced 1,000 steps apart (Section 3.1). We also report the more stable training and validation losses to complement the accuracy comparison (Tables 1 and 2). Together, these results suggest that HiRes is competitive with other residual-expansion methods across all three metrics, while offering a simpler design and lower activation I/O. We regard these methods as competitive and do not claim a reliable accuracy ranking among them. To assess variation across seeds, we additionally conducted a three-seed comparison between vanilla and HiRes, reported in Q2 below.