- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- This paper introduces a new approach to spectrally reparameterize continuous fraction networks, which avoids singularities and enables more stable training.
- The authors replace feedforward layers in Transformers with this new class of layer, which avoids the need for incremental training, and run pertaining up to billions of parameters.
- They demonstrate competitive performance in both language modelling and image generation.
Critical strengths
- I like the fact that this paper arrives to the results through a theoretically motivated approach (similar to stability techniques used in digital signal processing), whose main advantages hold up empirically.
- I appreciate the amount of experiments carried out by the authors, which gives value to their approach. The fact that the authors scaled their models to billions of parameters makes them particularly convincing.
- I also like that the performance achieved stemmed from a significantly more parameter efficient approach, while simultaneously circumventing the incremental training approach used in previous work.
Critical weaknesses
- I problem I see with the paper is the lack of an intuitive explanation of why continued fraction networks work well, or why they would be particularly suited to replace FFN layers in Transformers intuitively. I think this could be strengthened.
- Although the authors have discussed the use of spectral constraints to prevent singularities, these do not necessarily ensure well behaved gradients. Transformer blocks rely on carefully designed normalization schemes and residual connections to effectively propagate information and avoid gradients issues in depth. It is not clear how the use of continued fraction networks would affect these properties.
Questions for authors
- Is there an intuitive explanation for the performance of this type of network?
- Have the authors investigated issues of gradient propagation with increased depth?
- Is the network very sensitive the depth of the continued fraction used and the spectral constraints?
5 comments
Response to Reviewer 3ihA (Part II)
Is the network very sensitive the depth of the continued fraction used and the spectral constraints?
As mentioned above, the network is not sensitive to depth with our proposed reparametrization. In fact, performance overall improved slightly with depth.
Note that the spectral constraint is for the reparameterized partial denominators to lie outside the Gershgorin radius, which is exactly defined and computed. This guarantees stable results.
However, note that the above condition can be violated by setting to a smaller value such as . In this scenario, gets rounded to due to bf16 precision, and this will lead to a singular tridiagonal matrix that makes training unstable. This however, reflects a data format limitation, rather than a methodological failure.
Finally, we also train a DDT-XL/2 model with a higher as shown in Table A6. We observe that the results for the higher are only slightly worse in FID as expected, where training was stable. This further supports the robustness of our method and asserts the lack of need for tuning, which is highly beneficial given how expensive pre-training can be.
We have also now added some example language and image generations of Spec-CoFrGeNet in Appendix C.
Further questions?
Dear Reviewer 3ihA,
Since, the author-reviewer discussions are soon to close please let us know if there are any additional questions/concerns you might have. Thank you for your time and effort.
Response to Reviewer 3ihA (Part I)
We thank you for your constructive comments and appreciate the several positive remarks. Below, we address your major concerns. Please note that we have updated the paper where our edits are in pink and we point you to the relevant ones in our response below.
CoFrNets are a compact representation of a rich function class which enables them to express highly complex functional relationships and subsequently replace FFNs. Intuitively, a CoFrNet ladder of depth d gives us a d+1 degree rational function, while the number of parameters just scales linearly with depth. Thus we get a multiplicative representability benefit relative to the number of parameters required. In contrast, a FFN adds quadratic number of parameters per depth.
Moreover, having multiple ladders (i.e. width > 1) produces multiple rational functions which when combined into a single rational function would imply multiplying the denominator polynomials. This leads to multiplication of their degrees and hence a highly rich function. This idea was formally leveraged in the universal approximation proof for CoFrNets in Puri et. al. 2021 [1]. This is the reason for their favorable performance, while being parsimonious.
First, we would like to mention that Spec-CoFrGeNet is robust to depth of a continued fraction ladder when replacing FFNs in language and vision models. Previous work [2] introduced the idea of incremental training i.e. freezing parameters at higher depths and then gradually training them in later iterations since, ladders of depth > 1 were unstable to train. However, with the reparametrization introduced in our work, we do not need such a stabilization scheme and are able to optimize deep ladders training everything together. This is evident from the experiments in our paper which have a (high) depth (and width) of 5.
Moreover, we provide additional results in the appendix for models using ladders with depth (and width) of 1. For language modeling, these include performance of pretrained models of GPT2-XL (Table A1), Granite 4.1 (Table A2) and Llama-MoE (Table A3). As observed from these results, the performance of models with the higher depth reported in the main paper were either similar or better. For image generation, we are also training DDT-XL/2 models using Spec-CoFrGeNet with depth, width of 1, 1 and 10, 10. As shown in Table A5, we observe that not only is there no stability issue with increasing depth, in fact some training losses are slightly lower indicating potential performance improvement with increased depth. Thus, there are no stability issues with increasing ladder depth with our reparametrization.
Second, we address the concern of gradient propagation through the network. We note that original layer normalizations and skip connections of the Transformer architectures are maintained, with only the FFNs being replaced with CoFrNets. This preserves the benefits of carefully established Transformer design choices, with our approach being a drop-in replacement and thus easy to integrate and evaluate.
Additionally, we now have performed a gradient analysis comparing CoFrGeNets and Spec-CoFrGeNets for vision where CoFrGeNets failed. The failure happens because the loss hits NaN values. We compute gradient norms for the available checkpoints just before this instability. Specifically, we compute ∣∣∂L/∂xl∣∣2, where L denotes the training loss and xl denotes the activation after layer l i.e. after residual addition. As observed in Fig. 5, CoFrGeNet has a gradient explosion of the order of 105 through the backward pass which explains the observed instability. In contrast, the gradients of our proposed method stay stable.
We thank the reviewer for this thoughtful question. We have added the above analysis in Section 5.2 of our paper.
[1] Puri, Isha, et al. "CoFrNets: interpretable neural architecture inspired by continued fractions." Advances in neural information processing systems 34 (2021): 21668-21680.
[2] Dhurandhar, Amit, et al. "CoFrGeNet: Continued Fraction Architectures for Language Generation." ICML (2026).