- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that there are two distinct scale axes, compute and capacity, in the Looped Transformer. Inspired by this observation, the authors introduce dual-path block and expose these two axies, deep and wide, in parallel with per-token gate to combine them. They show that the best dual-path configuration beats both single-axis with using fewer parameters than the width-scaled control. This paper also shows interpretable results related with the function of word through the gate values per token.
Critical strengths
The perspective of separating compute and capacity provides an interesting way to understand recent studies on looped Transformers and language model scaling. The proposed dual path architecture also demonstrates promising performance at smaller model scales. Also, token-wise mechanistic interpretation through the gates provides additional interesting insights into model behavior.
Critical weaknesses
It is quite unclear whether iso-FLOP comparisons are really fair. The authors should provide a clearer flop computation and generation wall-clock times to establish the fair comparison.
The stability and scalability of the proposed architecture remain concerning. Although the gate soft cap addresses the instability at greater depths, Dual path still underperforms wide path on several benchmarks at L=32. It is unclear whether the proposed architecture remains effective as model depth increases.
Questions for authors
-
In Equation 9, do the attention FLOPs only account for the query, key, value, and output projections? Are query-key scoring and weighted value aggregation operations, which are dependent on sequence length, excluded? If so, the flops of the looped and dual-path settings may be underestimated, meaning they could use more FLOPs than reported. Additionally, reporting generation wall-clock times would help readers compare these methods from practical perspective.
-
In Table 3, dual path still underperforms pure wide path on language modeling and commonsense tasks at L=32, even with the gate soft cap. Could the authors explain why its advantage decreases at greater depths and whether this issue persists at larger scales, w,r,t scalability?
-
Could the authors include an ablation comparing the learned token-wise gates with static gates in the same dual path architecture? It will help the reader to understand whether the effectiveness is from "token-wise" gates or just gating itself.
3 comments
We want to thank the reviewer again and hope our initial comment has addressed their main concern. Here are the follow up results which have now finished training:
- The third seed for Dual α=25 at F80M (first table). With all 3 seeds the row is:
| Params | C4 | Wiki | CS acc | GSM8K | Agg. BPB | |
|---|---|---|---|---|---|---|
| F80M PureWide | 927M | 1.029 | 0.933 | 0.528 | 0.091 | 0.865 |
| F80M Dual α=25 | 606M | 1.032 | 0.931 | 0.528 | 0.104 | 0.865 |
The conclusion is unchanged: level on aggregate, PureWide slightly better on C4, Dual better on math, with 35% fewer parameters.
- L=32 with F80M, α=50, K=4, Dual with gate cap (due to congestion on the cluster and longer training times we only have one seed so far):
| L=32 | C4 | Wiki | CS acc | GSM8K | Math BPB |
|---|---|---|---|---|---|
| PureWide | 1.008 | 0.903 | 0.544 | 0.126 | 0.493 |
| Dual | 1.014 | 0.905 | 0.541 | 0.175 | 0.485 |
PureWide is ahead on language modelling and commonsense at this depth and Dual keeps a lead on GSM8K and Math BPB. We think these results point to the capacity gap described above (dual model has 472M fewer parameters) and will state this limitation clearly in the final version.
- Uncapped token-wise gates, same code and seeds as the other gate variants (3 seeds):
| Gates | C4 | Wiki | CS acc | GSM8K |
|---|---|---|---|---|
| token-wise (paper, no cap) | 1.038 | 0.940 | 0.523 | 0.098 |
| token-wise (paper, capped) | 1.037 | 0.935 | 0.523 | 0.118 |
| token-wise on updates | 1.033 | 0.930 | 0.525 | 0.122 |
Without the cap, token-wise gates on the full output are the weakest variant, because they also rescale the residual x by g_d + g_w. Bounding this (cap) or removing it (gates on updates only) fixes it, and the update-only variant remains the best. Since Table 1 used the uncapped variant, its gains are already conservative.
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns. The new total FLOP experiments largely address my FLOP matching concern and provide encouraging results for the dual-path architecture. However, the single-GPU prefill overhead and missing generation latency still raise questions about practical efficiency.
For Q3, the new ablations show that static and per-layer gates perform comparably to token-wise gating, suggesting that most of the gains comes from the dual-path architecture itself. While this does not undermine the architectural contribution, it makes the benefit of token-wise allocation less clear (Please clarify this, if i'm misunderstanding some contributions). Also, the proposed update only gating changes the original formulation, so it is unclear whether the main results and gate analyses would still hold. +) the L=32 results are still based on a single seed, and the reduced gains on language modeling and commonsense tasks leave my scalability concern unresolved.
Overall, I appreciate the additional evidence, but I will maintain my original rating.
Thank you for your detailed review, we appreciate the input and thank the reviewer for highlighting the strengths of our novel architecture. We ran new experiments for all three points raised by the reviewer, see the results below.
W1/Q1:
W2/Q2:
Q3: We have now included experiments where we train the model from scratch with the following gate ablations:
We note that most of the gain comes from the dual-architecture itself, as even static mixing gets close. The best variant is token-wise on updates only which we will adapt as our final version, so we want to thank the reviewer for this suggestion! Note that a) uses the gate cap of Table 2, an uncapped run is still training but we will post it here when its done.